Mobile robot autonomous grasping and placing method and system

Abstract

This invention discloses a method for autonomous grasping and placing of a mobile robot, comprising: identifying a target object using an image processing method based on acquired RGB images, and estimating the pose of the target object relative to the camera coordinate system; calculating the real-time position and attitude information of the mobile robot in the map coordinate system based on a two-dimensional LiDAR SLAM system, and according to the continuous motion estimation information of the mobile robot acquired by the wheel speed sensor and the depth information acquired by the RGBD camera; generating a global path from the real-time position of the mobile robot to the target position of the target object based on an environmental cost map, optimizing the local path according to real-time sensor data, and sending chassis speed commands to the mobile robot according to the global path and the local path to control the mobile robot to move to the target object; and controlling the mobile robot chassis and robotic arm to complete the grasping or placing operation of the target object according to the relative pose relationship between the target object and the mobile robot.

Description

Technical Field

[0001] This invention relates to the field of robotics, specifically to a method and system for autonomous grasping and placing of objects by a mobile robot. Background Technology

[0002] With the rapid development of mobile robot technology, mobile robots that integrate visual recognition, autonomous navigation, and robotic arm operation have been widely used in fields such as intelligent manufacturing, warehousing and logistics, automated sorting, and robotics competitions. These robots typically need to perform complex tasks such as target recognition, environmental mapping, path planning, obstacle avoidance control, and object grasping and placement.

[0003] In practical applications, robot systems typically consist of multiple functional modules, including a vision perception module, a localization and mapping module, a navigation planning module, a robotic arm control module, and a top-level task decision-making module. The vision module detects target markers in the environment and estimates their pose information; the localization and mapping module constructs an environmental map and estimates the robot's own position in real time; the navigation module plans motion paths based on the environmental map and controls the chassis movement; the robotic arm module performs precise grasping and placement actions; and the top-level decision-making module coordinates the logical relationships between the various functional modules to complete task flow control.

[0004] Currently, in both simulation and real-world environments, mobile robot systems typically employ LiDAR SLAM or RGBD SLAM for localization and mapping, combined with a hierarchical path planning framework for autonomous navigation. Simultaneously, visual recognition algorithms are used for target detection and pose estimation, and robotic arm motion control enables grasping and placement operations.

[0005] However, due to factors such as changes in lighting, obstruction, sensor noise, control errors, and complex task processes in real-world environments, the collaborative operation of multi-module systems often faces significant technical challenges.

[0006] Therefore, existing mobile robot autonomous grasping and placement technologies still have significant shortcomings in multi-sensor fusion, robust navigation, grasping success determination, and unified decision-making and control. There is an urgent need to propose a mobile robot autonomous grasping and placement method and system that can achieve multi-module collaboration, has anomaly recovery capability, and has a high task success rate. Summary of the Invention

[0007] Based on the above situation, the main objective of this invention is to provide a multi-sensor collaborative method for mobile robots to autonomously grasp and place objects, which integrates visual recognition, two-dimensional LiDAR SLAM positioning, and hierarchical path navigation planning. This method significantly improves the stability of mobile robots in complex environments and increases the success rate of task completion.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for autonomous grasping and placing of objects by a mobile robot, comprising the following steps: S100: Based on the acquired RGB image, an image processing method is used to identify the target object and estimate the pose of the target object relative to the camera coordinate system. S200 is based on a two-dimensional lidar SLAM system and calculates the real-time position and attitude information of the mobile robot in the map coordinate system based on the continuous motion estimation information of the mobile robot collected by the wheel speed sensor and the depth information collected by the RGBD camera. S300: Generate a global path from the real-time position of the mobile robot to the target position of the target object based on the environmental cost map, optimize the local path according to the real-time sensor data, and send the chassis speed command to the mobile robot according to the global path and the local path to control the mobile robot to move to the target object. S400, based on the relative pose between the target object and the mobile robot, control the mobile robot chassis and robotic arm to complete the grasping or placing operation of the target object.

[0009] Preferably, step S100 includes: S101, convert the RGB image to the HSV color space, and segment the target color region in the image according to preset parameters to obtain a binary image after color segmentation; S102, perform edge extraction on the binary image and obtain the inheritance structure between edge contours; S103, Based on OpenCV, use polygons to approximate the edge contour, extract the contour that approximates a quadrilateral within a preset error range, and sort the vertices of the quadrilateral contour in the order of top left-top right-bottom right-bottom left. S104, the sorted quadrilateral contour vertices are used as image feature points, and a mapping relationship is established with the corresponding points of the target object in the world coordinate system. The rotation matrix and translation vector of the target object relative to the camera coordinate system are calculated by the PnP algorithm to realize the pose of the target marker relative to the marker in the camera coordinate system. S105, the image is transformed by the vertices of the quadrilateral contour to reproject the marked area into a standard view image. The standard view image is compared with a preset black and white template, and the template type with the smallest difference between the two is selected as the recognition result of the target mark.

[0010] Preferably, step S102 includes filtering the detected edge contours according to the inheritance structure between the edge contours, retaining the outermost contour, and removing noise contours with an area smaller than a preset area value.

[0011] Preferably, step S105, after reprojecting the marked area into a standard view image, further includes centering the standard view image.

[0012] Preferably, step S200 includes the following steps: S201, the depth information acquired by the RGBD camera is converted into low-level laser scanning data and fused with the data from the two-dimensional lidar SLAM system to construct an environmental map; S202, based on the environmental map and according to the continuous motion estimation information of the mobile robot collected by the wheel speed sensor, the mobile robot is located to obtain the real-time position and attitude information of the mobile robot in the map coordinate system.

[0013] Preferably, the local path optimization based on real-time sensor data in step S300 includes: The system receives the pose information of the target identification relative to the camera coordinate system, combines it with the mileage information measured by the odometry, and the obstacle information fused from the depth information collected by the two-dimensional lidar and the RGBD camera to update the environmental cost map and complete dynamic obstacle avoidance, thereby achieving local path optimization.

[0014] Preferably, the global path is updated at a first frequency, and the local path is updated at a second frequency.

[0015] Preferably, step S300 further includes: When the planning of the global path or the local path is detected to have failed, a safe direction away from the obstacle is calculated based on the current two-dimensional LiDAR SLAM scanning data, and the mobile robot is controlled to move a preset distance in that direction before the planning of the global path and the local path is re-executed.

[0016] Preferably, step S400 further includes: When performing a crawling operation If the target object cannot be identified within the first preset time, the current capture is determined to have failed. If the target object can still be detected after the grasping is completed and the robotic arm is reset, the grasping is determined to have failed, and the method is re-executed. When placement is performed If the target object cannot be detected within the second preset time, the placement is deemed to have failed, and the method is re-executed.

[0017] The present invention also discloses a mobile robot autonomous grasping and placement system, wherein the system uses the method described in any one of the present invention to grasp and place the target object.

[0018] This invention presents a mobile robot autonomous grasping and placement method that integrates visual recognition, 2D LiDAR SLAM localization, wheel velocity measurement, and RGBD camera depth information to achieve multi-sensor collaborative operation. This improves the mobile robot's positioning accuracy and environmental perception capabilities in complex environments, enhances the system's adaptability to low obstacles and environmental changes, and thus improves overall navigation safety and operational stability. By employing a hierarchical path planning structure using global and local paths, the success rate of navigation execution is improved. A clearly structured, collaboratively efficient mobile robot autonomous grasping and placement system is constructed, significantly improving the system's stability and task completion success rate in complex environments, and possessing significant engineering application value.

[0019] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0020] The preferred embodiments of the mobile robot autonomous grasping and placement method and system according to the present invention will be described below with reference to the accompanying drawings. In the drawings: Figure 1 This is a flowchart of a mobile robot autonomous grasping and placement method according to a preferred embodiment of the present invention; Figure 2 A flowchart of a method for target identification and pose estimation of a target object according to a preferred embodiment of the present invention; Figure 3 This is a comparison image before and after centering processing according to a preferred embodiment of the present invention; Figure 4 This is a flowchart of a method for calculating the real-time position and attitude information of a mobile robot based on a two-dimensional lidar SLAM system according to a preferred embodiment of the present invention. Figure 5 The image shows the positioning accuracy test results using the technical solution of this invention. Figure 6 This is a speed command diagram for the navigation process using the technical solution of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0023] In the description of this invention, unless otherwise explicitly defined, terms such as "setup" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0024] In the description of this invention, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0025] Figure 1The flowchart of a preferred embodiment of the present invention for an autonomous grasping and placing method for a mobile robot includes the following steps: Step S100: Based on the acquired RGB image, an image processing method is used to identify the target object and estimate the pose of the target object relative to the camera coordinate system; Step S200: Based on a two-dimensional LiDAR SLAM system, and according to the continuous motion estimation information of the mobile robot acquired by the wheel speed sensor and the depth information acquired by the RGBD camera, the real-time position and attitude information of the mobile robot in the map coordinate system are calculated; Step S300: Based on the environmental cost map, a global path is generated from the real-time position of the mobile robot to the target position of the target object, local path optimization is performed based on real-time sensor data, and a chassis speed command is sent to the mobile robot according to the global path and the local path to control the mobile robot to move to the target object; Step S400: According to the relative pose relationship between the target object and the mobile robot, the mobile robot chassis and robotic arm are controlled to complete the grasping or placing operation of the target object.

[0026] This invention presents a mobile robot autonomous grasping and placement method that integrates visual recognition, 2D LiDAR SLAM localization, wheel velocity measurement, and RGBD camera depth information to achieve multi-sensor collaborative operation. This improves the mobile robot's positioning accuracy and environmental perception capabilities in complex environments, enhances the system's adaptability to low obstacles and environmental changes, and thus improves overall navigation safety and operational stability. By employing a hierarchical path planning structure using global and local paths, the success rate of navigation execution is improved. A clearly structured, collaboratively efficient mobile robot autonomous grasping and placement system is constructed, significantly improving the system's stability and task completion success rate in complex environments, and possessing significant engineering application value.

[0027] In a preferred embodiment, such as Figure 2As shown, step S100 may include: Step S101, converting the RGB image to the HSV color space, segmenting the target color region in the image according to preset parameters, and obtaining a binary image after color segmentation; Step S102, performing edge extraction on the binary image and obtaining the inheritance structure between edge contours; Step S103, using OpenCV to approximate the edge contours with polygons, extracting contours that approximate quadrilaterals within a preset error range, and sorting the vertices of the quadrilateral contours in the order of top left-top right-bottom right-bottom left; Step S104, using the sorted quadrilateral contour vertices as image feature points, establishing a mapping relationship with the corresponding points of the target object in the world coordinate system, and using PnP... The algorithm calculates the rotation matrix and translation vector of the target object relative to the camera coordinate system to estimate the pose of the target marker relative to the camera coordinate system; in step S105, the image is transformed by the vertices of the quadrilateral contour to reproject the marker area into a standard view image, the standard view image is compared with a preset black and white template, and the template type with the smallest difference between the two is selected as the recognition result of the target marker.

[0028] In the above embodiments, the target region is extracted by segmentation in the HSV color space, and candidate quadrilateral regions are obtained by combining edge detection and contour screening. After polygon approximation and vertex sorting, the pose parameters of the target relative to the camera coordinate system are calculated based on the PnP algorithm, and the target recognition is completed by perspective transformation and template matching, thereby improving the recognition stability and anti-interference ability.

[0029] In a specific implementation, step S100 may include color segmentation, edge extraction, polygon approximation, pose estimation, and template matching.

[0030] First, the input RGB image is converted to the HSV color space. Target color regions, such as the red region, are extracted from the image according to preset parameters, thus obtaining target candidate regions. Specifically, since the boundary between the red portion and the background is relatively clear in the simulation environment, color segmentation can effectively reduce subsequent computation and improve processing speed.

[0031] After obtaining the binary segmented image, edge extraction is performed to obtain the inheritance structure between contours. Considering that numbers or letters inside the identifier can also generate edges, to avoid interference with subsequent steps, the detected edge contours can be filtered according to the inheritance structure between the edge contours, retaining only the outermost contour while removing noise contours with an area smaller than a preset area value, thereby obtaining stable candidate regions. This processing can reduce false detections and improve the robustness of the recognition algorithm. The preset area value can be set to around 200-300 pixels to balance long-distance recognition capability and noise resistance performance.

[0032] Based on contour selection, candidate contours are approximated using polygons, retaining only those contours that can be approximated as quadrilaterals within a preset error range. The four vertices are then uniformly sorted in the order of top left, top right, bottom right, and bottom left. These sorted vertices can be used for subsequent pose estimation and image perspective transformation, thus ensuring data consistency between steps in the algorithm.

[0033] Subsequently, the vertices of the sorted quadrilaterals are used as image feature points, and a mapping relationship is established with the corresponding points of the target in the world coordinate system. The rotation matrix and translation vector of the target object relative to the camera coordinate system are calculated using the PnP algorithm to achieve target object pose estimation. This target object is the same target region obtained from the aforementioned HSV segmentation and contour filtering.

[0034] In the target recognition stage, perspective transformation is performed on the original image based on the vertices of the quadrilaterals. The identified area is reprojected into a standard viewpoint image and compared with a black and white template. The template type with the smallest difference between the two is selected as the recognition result of the target identifier. This also determines whether the recognition type is a number or a letter. In actual processing, it was found that the reprojected image often has a misaligned target, easily leading to mismatch with the template. Therefore, after perspective unfolding, further centering processing can be performed to align the projected image with the template, thereby reducing false detections and improving recognition accuracy. Figure 3 As shown, from left to right are: the original projected image, the centered projected image, and the template image.

[0035] By employing strategies such as contour filtering, polygon constraints, and centralized projection, this embodiment improves recognition stability and anti-interference capabilities while ensuring high operating speed. In testing, it was able to stably complete the tasks of identifier detection and type recognition.

[0036] To coordinate with subsequent object grasping and placement steps, the target identifier to be grasped needs to be selected from the detected target identifiers, thereby controlling the robot to reach the appropriate position in front of the object. First, bottom objects can be excluded. Then, the target identifier with the smallest Euclidean distance from the camera coordinate system in the current frame is selected as the grasping target. Testing showed that, at the required distance and angle for object grasping and placement, identifier recognition was very stable in hundreds of tests, with almost no missed or false detections, and the recognition accuracy was close to 100%. The average time for single-frame image recognition is 0.028 seconds, and the frame rate can reach over 30Hz, which meets the application requirements of subsequent modules such as object grasping.

[0037] In a preferred embodiment, such as Figure 4As shown, step S200 may include: step S201, converting the depth information acquired by the RGBD camera into low-level laser scanning data and fusing it with the data from the two-dimensional lidar SLAM system to construct an environmental map; step S202, based on the environmental map and according to the continuous motion estimation information of the mobile robot acquired by the wheel speed sensor, locating the mobile robot to obtain the real-time position and attitude information of the mobile robot in the map coordinate system.

[0038] In this embodiment, a two-dimensional LiDAR-SLAM system is used as the main localization framework, combined with wheel velocities and depth information from an RGBD camera to achieve multi-source data fusion. The LiDAR is used to construct an environmental map and perform real-time pose updates, while the wheel velocities provide stable motion estimation input. The depth data from the RGBD camera is converted into low-level LiDAR scan data and fused with the LiDAR data to improve the system's ability to perceive low-lying obstacles. The accuracy test results of the localization system are as follows: Figure 5 The image shows the localization results during a 40-second motion and a comparison with the ground truth. It can be seen that during the 40-second motion, the SLAM system can perform real-time localization accurately, with an average X-direction position error of 1.8 cm, an average Y-direction position error of 1.4 cm, and an average angle error of 1.4 degrees (in the map coordinate system). Throughout the continuous motion, the position error remained within the centimeter range, and the attitude error remained within a small range, verifying the effectiveness of the fusion localization scheme.

[0039] Step S200 enables environmental mapping and real-time localization of the mobile robot, providing stable pose information for navigation and grasping tasks.

[0040] A 2D SLAM method based on LiDAR was used for localization and mapping. LiDAR can stably perceive the planar structure of the playing field. Compared with visual SLAM methods based on RGBD cameras, it does not rely on image feature points and can avoid localization loss caused by insufficient environmental texture or changes in lighting, thus exhibiting better robustness in real-world environments.

[0041] To further improve positioning stability, wheel velocity measurement information is introduced as an odometry input and fused with LiDAR data. Wheel velocity measurement provides continuous motion estimation information and exhibits good stability in two-dimensional wheeled robot scenarios, effectively reducing positioning drift caused by a single sensor and thus improving overall positioning accuracy.

[0042] Furthermore, considering the high installation position of the LiDAR, which makes it difficult to detect low-lying obstacles, the system introduces an RGBD camera as an auxiliary sensor. By converting the area near the ground in the depth image into laser scan data and fusing it with the LiDAR scan results, the positioning system can simultaneously consider low-height obstacles during mapping and navigation, improving the completeness of environmental perception and operational safety.

[0043] The fused multi-source sensor data is uniformly input into the SLAM system for pose update and map construction, outputting the robot's real-time position and attitude information in the map coordinate system, which is then used in subsequent steps. Test results show that this method can maintain high positioning accuracy during long-term movement, with position errors remaining within the centimeter range, meeting the requirements for mobile robot task execution.

[0044] By fusing lidar, wheel velocity sensors, and depth information, the system improves positioning robustness and environmental adaptability while ensuring real-time performance, providing a stable and reliable pose foundation for subsequent navigation planning and grasping operations.

[0045] In step S300 of the technical solution of this invention, a hierarchical path planning structure is adopted to achieve collaborative work between global path planning and local path optimization within a unified navigation framework. In the global path planning stage, a global path from the current position to the target position is generated based on an environmental cost map, providing overall motion direction and path constraints. In the local path planning stage, the global path is optimized in real time by combining the robot's current pose and sensor data, ensuring that the generated trajectory meets the robot's dynamic constraints and possesses obstacle avoidance capabilities. Through the cooperation of global and local planning, stable path generation and real-time adjustment are achieved.

[0046] In a preferred embodiment, step S300, which optimizes the local path based on real-time sensor data, may include: receiving the pose information of the target identification relative to the camera coordinate system, combining it with the mileage information measured by the odometer, and the obstacle information fused from the depth information collected by the 2D LiDAR and the RGBD camera, updating the environmental cost map, and completing dynamic obstacle avoidance to achieve local path optimization. Continuously receiving the real-time pose information of the target identification relative to the camera coordinate system and the obstacle information fused from the LiDAR and the RGBD camera, updating the environmental cost map, and completing dynamic obstacle avoidance improves safety and environmental adaptability during navigation.

[0047] In a preferred embodiment, to balance computational resources and planning effectiveness, the global path is updated at a first frequency, and the local path is updated at a second frequency. Different update frequencies can be set for the global and local path planning, allowing the planning results to respond promptly to environmental changes. Specifically, the first frequency can be 2Hz, and the second frequency can be 5Hz.

[0048] Control errors may occur during the actual movement of the robot, leading to situations where the planned path is infeasible. In a preferred embodiment, step S300 may further include a planning failure recovery mechanism: when the planning failure of the global path or the local path is detected, a safe direction away from the obstacle is calculated based on the current 2D LiDAR SLAM scan data, and the mobile robot is controlled to move a preset distance in that direction before re-executing the planning of the global path and the local path. This allows the robot to escape the locally infeasible area and resume the navigation process. This strategy can effectively improve the completion rate of the navigation task and the robustness of the system.

[0049] By combining a hierarchical path planning structure with a recovery mechanism, navigation stability is improved while ensuring smooth motion, enabling the robot to continuously generate executable paths in complex environments and achieve autonomous navigation with a high success rate.

[0050] In a specific implementation, the ROS Navigation Stack can be used as the robot navigation framework. It uses `move_base` as the overall framework. First, the global planner calculates a coarse global path in the cost map. Then, the local planner optimizes the local path to meet the robot's dynamic constraints. Finally, the chassis speed command is sent to the robot for execution. In the local planner's dynamic constraints, the maximum speed is constrained to the maximum allowable movement speed of the robot (linear velocity 0.5 m / s in the x-direction, 0.5 m / s in the y-direction, and angular velocity 0.5 rad / s). Testing has shown that the robot can travel at the maximum allowed speed during movement, such as... Figure 6 As shown. Obstacle avoidance capability can be demonstrated by the effectiveness of task completion. Testing showed that it can complete tasks with almost 100% success rate.

[0051] In a preferred embodiment, step S400 may further include a visual detection mechanism: When performing a crawling operation If the target object cannot be identified within the first preset time, the current capture is determined to have failed. If the target object can still be detected after the grasping is completed and the robotic arm is reset, the grasping is determined to have failed, and the method is re-executed. When placement is performed If the target object cannot be detected within the second preset time, the placement is deemed to have failed, and the method is re-executed.

[0052] After navigation reaches the preset operation position, the robot adjusts the chassis posture through closed-loop control according to the relative pose between the target and the robot, so that the robotic arm is in a suitable operating range; the robot moves above the target to complete the gripping or releasing action, and the target status is confirmed by visual detection after execution to determine whether the gripping or placement is successful, thereby improving the reliability of task execution.

[0053] During the grasping process, the robot first navigates to the target area, bringing the marker to be grasped into the camera's field of view. Once the target's pose in the camera coordinate system is detected, the robot adjusts its chassis movement using closed-loop control based on the relative pose between the robot's base and the target, moving the robot to a suitable position for the robotic arm's operation. The robotic arm then moves, positioning the gripper above the target and performing the grasping action. After successful grasping, the robotic arm resets, and the robot retreats to proceed to the next task phase.

[0054] To improve the robustness of the grasping process, if the target cannot be identified for an extended period during grasping, the grasp is considered to have failed. If the target can still be detected after the grasp is complete and the robotic arm has reset, the grasp is considered unsuccessful, and the task flow needs to be re-executed. This method avoids misjudgments caused by relying solely on the gripper's state, thus improving the reliability of grasp success determination.

[0055] During placement, after navigating to the target placement area, the robot adjusts its chassis position based on the target marker's pose information to position the robotic arm within the appropriate range. Considering the limitations of the robotic arm's effective working distance, the robot chassis moves forward a preset distance further, ensuring the robotic arm can stably complete the placement operation, provided the target can be fully observed. The robotic arm is then moved above the placement position, opens its gripper to release the object, and finally resets and retracts after execution.

[0056] If the target cannot be detected during the placement process, the placement is deemed a failure, and a retry process is triggered. Through the aforementioned closed-loop control and visual feedback mechanism, self-monitoring of the status and failure rollback can be achieved during the task, improving the overall task execution success rate and system fault tolerance.

[0057] Testing showed that the grabbing / placement strategy in this invention can complete the task almost 100% of the time. Meanwhile, the retry strategy after failure gives the system good robustness and fault tolerance.

[0058] The autonomous grasping and placement method for mobile robots of the present invention can be controlled using a finite state machine. Steps S100 to S400 can be designed as a visual recognition module, a fusion localization module, a motion planning module, and a grasping and placement execution module, respectively. The finite state machine, as the top-level decision-making module, uniformly schedules the processes of recognition, localization, navigation, grasping, and placement. Each functional module executes according to a preset state logic sequence. When an anomaly occurs at any stage, the system automatically enters a recovery or retry state and re-calls the relevant modules to complete the task processing, achieving multi-module collaboration and anomaly self-recovery capabilities.

[0059] A finite state machine is used as the overall control framework, dividing the task process into multiple states, including the initial state, navigation and movement state, grasping and execution state, placement and execution state, and task switching state. The system transitions between different states according to preset logic, and each state calls the corresponding functional module to complete the specific task operation. For example, in the navigation state, the motion planning module is called to implement path execution, and in the grasping or placement state, the visual recognition and execution module is called to complete the target operation.

[0060] During task execution, the top-level decision-making module continuously receives the execution results from each module and determines whether to proceed to the next stage based on the returned status. When a step fails, such as navigation failure, recognition failure, or grasping failure, the state machine automatically rolls back to the corresponding previous state and re-executes, thereby implementing a task retry and exception recovery mechanism and improving the overall robustness and fault tolerance of the system. Furthermore, the top-level decision-making module is responsible for the sequential management of the task flow and target switching control. After completing the grasping or placement of the current target, it automatically enters the next task loop until all tasks are completed. By unifying the management of all functional modules under the state machine, the system structure is clear, the coupling between modules is reduced, and maintenance and expansion are facilitated.

[0061] Through the above decision-making mechanism, the system can maintain stable operation during complex tasks and automatically recover through state rollback when partial failures occur, thereby effectively improving the overall task success rate.

[0062] The decision logic demonstrated good robustness and fault tolerance during testing, capable of handling various low-probability unexpected situations that may occur during the task. Five tests were conducted for each possible combination of task numbers, for a total of 50 tests. The results are shown in Table 1. The success rate was 100% across the 50 tests, with an average time of 48.39 seconds.

[0063] Table 1

[0064] The present invention also discloses a mobile robot autonomous grasping and placement system, wherein the system uses the method described in any one of the present invention to grasp and place the target object.

[0065] Specifically, based on the embodiments disclosed above in this application, reference is made to... Figure 1 The processes described in the flowcharts of the embodiments can be implemented as computer programs. For example, embodiments of this application also provide a computer program product including a computer program carried on a computer-readable medium, the computer program comprising methods for executing... Figure 1 The flowchart of the embodiment describes the program code for the method. In such an embodiment, the computer program can be downloaded and installed from a network via a communication interface, or installed from memory. When the computer program is executed by a processor, it performs the functions defined in the method of the above embodiment.

[0066] It should be noted that the aforementioned computer-readable media may include, but is not limited to: volatile memory, such as random access memory (RAM); non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); and combinations of the above types of memory.

[0067] In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0068] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0069] In an optional embodiment, this application also provides a computer storage medium that can be used for computer software instructions, including a computer program, which, when executed by a processor, performs the autonomous grasping and placement method for a mobile robot as described in the above embodiments. This storage medium includes, but is not limited to, flash memory, hard disk, and solid-state drive.

[0070] It should be noted that the use of step numbers (letter or number) to refer to certain specific method steps in this invention is merely for the purpose of convenience and brevity in description, and is by no means intended to restrict the order of these method steps. Those skilled in the art will understand that the order of the relevant method steps should be determined by the technology itself and should not be unduly restricted by the existence of step numbers.

[0071] Those skilled in the art will understand that, without conflict, the above-mentioned preferred solutions can be freely combined and superimposed.

[0072] It should be understood that the above embodiments are merely exemplary and not restrictive. Various obvious or equivalent modifications or substitutions that can be made by those skilled in the art regarding the above details without departing from the basic principles of the present invention will be included within the scope of the claims of the present invention.

Claims

1. A method for autonomous grasping and placing of objects by a mobile robot, characterized in that, Including the following steps: S100: Based on the acquired RGB image, an image processing method is used to identify the target object and estimate the pose of the target object relative to the camera coordinate system. S200 is based on a two-dimensional lidar SLAM system and calculates the real-time position and attitude information of the mobile robot in the map coordinate system based on the continuous motion estimation information of the mobile robot collected by the wheel speed sensor and the depth information collected by the RGBD camera. S300: Generate a global path from the real-time position of the mobile robot to the target position of the target object based on the environmental cost map, optimize the local path according to the real-time sensor data, and send the chassis speed command to the mobile robot according to the global path and the local path to control the mobile robot to move to the target object. S400, based on the relative pose between the target object and the mobile robot, control the mobile robot chassis and robotic arm to complete the grasping or placing operation of the target object.

2. The mobile robot autonomous grasping and placement method according to claim 1, characterized in that, Step S100 includes: S101, convert the RGB image to the HSV color space, and segment the target color region in the image according to preset parameters to obtain a binary image after color segmentation; S102, perform edge extraction on the binary image and obtain the inheritance structure between edge contours; S103, Based on OpenCV, use polygons to approximate the edge contour, extract the contour that approximates a quadrilateral within a preset error range, and sort the vertices of the quadrilateral contour in the order of top left-top right-bottom right-bottom left. S104, the sorted quadrilateral contour vertices are used as image feature points, and a mapping relationship is established with the corresponding points of the target object in the world coordinate system. The rotation matrix and translation vector of the target object relative to the camera coordinate system are calculated by the PnP algorithm to realize the pose estimation of the target marker relative to the marker in the camera coordinate system. S105, the image is transformed by the vertices of the quadrilateral contour to reproject the marked area into a standard view image. The standard view image is compared with a preset black and white template, and the template type with the smallest difference between the two is selected as the recognition result of the target mark.

3. The mobile robot autonomous grasping and placement method according to claim 2, characterized in that, Step S102 includes filtering the detected edge contours according to the inheritance structure between the edge contours, retaining the outermost contour, and removing noise contours with an area smaller than a preset area value.

4. The mobile robot autonomous grasping and placement method according to claim 2, characterized in that, In step S105, after reprojecting the marked area into a standard view image, the standard view image is further centered.

5. The mobile robot autonomous grasping and placement method according to claim 1, characterized in that, Step S200 includes the following steps: S201, the depth information acquired by the RGBD camera is converted into low-level laser scanning data and fused with the data from the two-dimensional lidar SLAM system to construct an environmental map; S202, based on the environmental map and according to the continuous motion estimation information of the mobile robot collected by the wheel speed sensor, the mobile robot is located to obtain the real-time position and attitude information of the mobile robot in the map coordinate system.

6. The method for autonomous grasping and placing of a mobile robot according to claim 1, characterized in that, The local path optimization based on real-time sensor data in step S300 includes: The system receives the pose information of the target identification relative to the camera coordinate system, combines it with the mileage information measured by the odometry, and the obstacle information fused from the depth information collected by the two-dimensional lidar and the RGBD camera to update the environmental cost map and complete dynamic obstacle avoidance, thereby achieving local path optimization.

7. The mobile robot autonomous grasping and placement method according to claim 6, characterized in that, The global path is updated at a first frequency, and the local path is updated at a second frequency.

8. The mobile robot autonomous grasping and placement method according to claim 6, characterized in that, Step S300 further includes: When the planning of the global path or the local path is detected to have failed, a safe direction away from the obstacle is calculated based on the current two-dimensional LiDAR SLAM scanning data, and the mobile robot is controlled to move a preset distance in that direction before the planning of the global path and the local path is re-executed.

9. The method for autonomous grasping and placing of a mobile robot according to claim 1, characterized in that, Step S400 further includes: When performing a crawling operation If the target object cannot be identified within the first preset time, the current capture is determined to have failed. If the target object can still be detected after the grasping is completed and the robotic arm is reset, the grasping is determined to have failed, and the method is re-executed. When placement is performed If the target object cannot be detected within the second preset time, the placement is deemed to have failed, and the method is re-executed.

10. A mobile robot autonomous grasping and placement system, characterized in that, The system employs the method described in any one of claims 1-9 to perform the grasping and placement of the target object.

Images