Mobile mechanical arm grabbing system based on reinforcement learning cooperative controller and implementation method thereof

Abstract

The invention discloses a movable mechanical arm grabbing system based on a reinforcement learning cooperative controller and an implementation method thereof.According to the system, control instructions of a movable chassis and a mechanical arm are synchronously generated through the reinforcement learning cooperative controller according to the target position of a workpiece, and synchronous movement of the movable chassis and the mechanical arm is achieved; the laser SLAM technology is used for conducting high-precision mapping and positioning on the factory environment, and the posture of the movable chassis can be automatically adjusted to conduct obstacle avoidance and navigation in the moving process through the Auwhome module. The OpenCV visual recognition and positioning module is combined for precise recognition and positioning of the workpiece, the mechanical arm is controlled through the Moveit module, and finally efficient workpiece automatic grabbing and placing tasks are achieved. The mechanical arm can flexibly cope with workpieces at different positions in the long-distance transportation process, the grabbing posture can be dynamically adjusted, repeated adjustment caused by the fact that the workpieces exceed the movement range of the mechanical arm is avoided, and the overall operation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a mobile robot arm grasping system based on a reinforcement learning collaborative controller and an implementation method thereof. Background Art

[0002] Because of its high working efficiency and low cost, robotic arms have been widely used in intelligent manufacturing scenarios. The mobile robotic arm is composed of a robotic arm mounted on a mobile chassis, which combines the mobility of the mobile chassis with the high flexibility of the robotic arm, thereby expanding its application range. In industrial applications, mobile robotic arms can efficiently complete tasks such as cargo handling, workpiece grasping and placement, greatly improving the automation level and operational flexibility of the production process. Traditional mobile robotic arm control systems usually use teaching methods for operation, which has the problems of complex deployment and insufficient execution flexibility. In addition, traditional systems usually separate the control of the robotic arm from the control of the mobile chassis, which results in the mobile chassis completing the movement first before the robotic arm operates when performing tasks. This control method not only reduces the working efficiency of the overall system, but also cannot guarantee the control accuracy. Summary of the invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a mobile robotic arm grasping system based on reinforcement learning collaborative controller and its implementation method, which breaks through the limitations of traditional step-by-step execution, enables movement and operation to be carried out simultaneously, and improves the operating efficiency and control accuracy of the overall system.

[0004] Technical solution: A mobile robot arm grasping system based on reinforcement learning collaborative controller, including sensors, robot arms, mobile chassis and industrial computers, wherein the sensors are arranged on the robot arms, the robot arms are arranged on the mobile chassis, the industrial computers are arranged in the mobile chassis, and a grasping control system is arranged in the industrial computers; the grasping control system includes an OpenCV visual recognition and positioning module, a data storage module, a reinforcement learning collaborative controller module, a Moveit module and an Autoware module; the sensors include a binocular camera and a laser radar;

[0005] The OpenCV visual recognition and positioning module captures the spatial information of the object in real time through a binocular camera, uses the OpenCV algorithm to identify and locate the workpiece and the placement table, and provides the position information of the target object; and transmits the identified position information of the target object to the reinforcement learning collaborative controller module;

[0006] The data storage module is used to store the position data of multiple workpieces and placement tables, and supports the scheduling and optimization of grasping tasks by recording the dynamic positions of the workpieces and the placement tables; updates the status of the workpieces and the availability information of the placement tables to provide real-time support for grasping and placing tasks; queries and outputs the coordinate information of the workpieces or the placement tables according to the input number, and transmits this information to the reinforcement learning collaborative controller module;

[0007] The reinforcement learning collaborative controller module realizes the collaborative control of the mobile chassis and the robotic arm through the SAC reinforcement learning algorithm, receives the target information from the OpenCV visual recognition and positioning module and the data storage module, and dynamically adjusts the movement of the mobile chassis and the robotic arm through the pre-trained strategy network to optimize the grasping and placing process; transmits the position of the object to be grasped or placed in the robotic arm coordinate system to the Moveit module; and transmits the position and posture of the 2D mobile chassis in the world coordinate system to the Autoware module;

[0008] The Autoware module generates an executable path based on the target point location information and drives the mobile chassis to move to the target point, integrates sensor data for environmental perception, and calculates the optimal movement trajectory in combination with the path planning algorithm to guide the mobile chassis to reach the target area;

[0009] After the Moveit module obtains the target position to be grasped by the robotic arm, it dynamically calculates the motion trajectory according to the task requirements and adjusts the position of the robotic arm in real time to ensure that the gripper accurately reaches the target position.

[0010] A mobile robot arm grasping implementation method based on reinforcement learning collaborative controller is used in the above mobile robot arm grasping system, which realizes automatic grasping or placement of workpieces through collaborative control of the robot arm and the mobile chassis; the steps are as follows:

[0011] S1, use laser SLAM to map the factory environment, perform initial rough calibration of the workpieces to be grasped and the placement table in the factory, and obtain the approximate position coordinates of the workpieces and the placement table;

[0012] S2, the user selects the workpiece to be grasped;

[0013] S3, the data storage module uploads the rough position information of the workpiece to be grasped to the reinforcement learning collaborative controller module according to the previously stored rough position information. After the reinforcement learning collaborative controller module parses the rough position, it dynamically plans the coordinated motion instructions of the robot arm and the mobile chassis, and sends them to the Autoware module and the Moveit module respectively;

[0014] S4, the Autoware module generates an executable path based on the target point position information and drives the mobile chassis to move to the target point. At the same time, after the Moveit module obtains the target position of the robotic arm, it dynamically adjusts the position of the robotic arm during the movement;

[0015] S5, start the OpenCV visual recognition and positioning module to identify the workpiece; pre-process the workpiece image obtained by the binocular camera, and determine the workpiece to be grasped by extracting color and shape features, and upload the position information of the workpiece to the reinforcement learning collaborative controller module;

[0016] S6, after the reinforcement learning collaborative controller module receives the position of the target workpiece, it sends instructions to the Autoware module and the Moveit module respectively to control the coordinated movement of the mobile chassis and the robotic arm to perform synchronous grasping operations;

[0017] S7, after the grab command is executed, determine whether the object is grabbed:

[0018] When the gripper sensor does not detect that the object has been grasped, the robot arm extends and re-identifies and grasps;

[0019] When the gripper sensor detects that an object has been grasped, the data storage module uploads the previously stored placement position to the reinforcement learning collaborative controller module. The reinforcement learning collaborative controller module calculates the target position of the mobile chassis and the target position of the robotic arm, and sends them to the Autoware module and Moveit module respectively;

[0020] S8, the Autoware module generates an executable path based on the target point position information and drives the mobile chassis to move to the target point. At the same time, after the Moveit module obtains the target position of the robotic arm, it drives the robotic arm to extend;

[0021] S9, start the OpenCV visual recognition and positioning module to identify the placement table; after the placement table is identified, its coordinate position is determined and the information is uploaded to the reinforcement learning collaborative controller module;

[0022] S10, after receiving the position of the placement table, the reinforcement learning collaborative controller module simultaneously sends instructions to the Autoware module and the Moveit module respectively to control the coordinated movement of the mobile chassis and the robotic arm to perform a synchronous placement operation;

[0023] S11, when the workpiece is placed, the data storage module publishes the new initial position and uploads it to the reinforcement learning collaborative controller module. The reinforcement learning collaborative controller module calculates the target position of the mobile chassis and the target position of the robotic arm, and sends them to the Autoware module and Moveit module respectively to control the robotic arm to retract and fold and control the mobile chassis to return.

[0024] Furthermore, the initial rough calibration of the workpiece to be grasped and the placement table includes the following steps:

[0025] S11, obtaining continuous frame data from the point cloud data collected by the laser radar, using the generalized iterative closest point algorithm to perform three-dimensional point cloud registration between continuous frames, and calculating the relative pose transformation between continuous frames;

[0026] S12, determining continuous key frames through a key frame screening mechanism, performing three-dimensional point cloud registration on the screened key frames, and using the continuous frame registration results as initial pose estimation for key frame registration;

[0027] S13, using a loop detection mechanism to detect loop frames by matching historical key frames with current key frames; if a loop frame is detected, performing 3D point cloud registration, calculating relative pose transformation of the loop frame, and forming a loop constraint;

[0028] S14, using the laser radar odometer, plane constraint, loop constraint and pose optimization as constraint conditions, constructing a graph optimization model; using the key frame registration results in the pose graph as nodes, combining the pose transformations between frames, between key frames and between loop frames to perform node constraints;

[0029] S15, through the graph optimization model, the optimized pose of the key frame after noise removal is obtained, and the optimized point cloud map information is generated.

[0030] Further, in step S3, the implementation of the reinforcement learning collaborative controller module includes:

[0031] Soft-Actor-Critic reinforcement learning is used as the training method. Based on the off-policy actor-critic framework of maximum entropy reinforcement learning, it maximizes the entropy of the strategy while maximizing the expected reward.

[0032] Adopt dual Q network and dual V network;

[0033] Transformer is used as the neural network structure in the deep reinforcement learning model as the fully connected layer:

[0034] During the calling process, only the trained Actor network is used. The input of the Actor network is the position coordinates of the target object, and the output is the two-dimensional pose coordinates in the geodetic coordinate system of the mobile chassis and the coordinates of the object to be grasped by the robotic arm in the robotic arm coordinate system.

[0035] Further, in step S4, an executable path is generated according to the target point position information and the mobile chassis is driven to move to the target point, which specifically includes the following steps:

[0036] S41, identifies obstacles through point cloud data obtained by laser radar, uses KNN algorithm to preprocess and segment the point cloud, calculates the Euclidean distance between points according to the distance threshold in three-dimensional space, clusters the point cloud, and calculates the distance between surrounding objects and the mobile chassis;

[0037] S42, the planning module generates the motion trajectory of the mobile chassis according to the output of the reinforcement learning collaborative controller module, which is divided into task planning and motion planning; the global trajectory is generated according to the relative position between the current position and the given destination through the global path planning algorithm; the local path planning is combined with the hybrid A* algorithm, considering the mobile chassis state, three-dimensional map information and obstacle position, to generate a local trajectory suitable for obstacle avoidance; the global and local paths are processed by the motion planning module to generate the final motion trajectory;

[0038] S43, path tracking uses a pure tracking algorithm to decompose the global path into multiple discrete path points, and searches for the next closest path point within a certain threshold distance in each control cycle; the trajectory tracking module sets the linear velocity and angular velocity of the mobile chassis according to the relative position of the next path point;

[0039] S44, the bottom control module of the mobile chassis drives the mobile chassis according to the output linear velocity and angular velocity instructions of the trajectory tracking module, and uses a PID controller to adjust the controller output to drive the rotation speed of the two tracks by comparing the difference between the current state and the target state of the mobile chassis.

[0040] Further, in step S5, the specific steps of visual recognition and positioning by the OpenCV visual recognition and positioning module are as follows:

[0041] S51, preprocessing the image of the workpiece to be grasped obtained by the binocular camera;

[0042] S52, converting the image from RGB format to HSV format, using the cvtColor function of OpenCV to complete the conversion, and using a preset color boundary array to detect the target color, identifying different color blocks to determine the workpiece to be grasped;

[0043] S53, using the threshold segmentation method to segment the artifact image, using the threshold function in OpenCV to separate the artifact from the environment in the image;

[0044] S54, extracting the contour of the workpiece by an edge detection algorithm, marking pixels above a high threshold as strong edges, marking pixels below a high threshold but above a low threshold as weak edges, and removing pixels below a low threshold;

[0045] S55, determining the position and posture of the workpiece according to the contour data of the recognition result.

[0046] Compared with the prior art, the present invention has the following significant effects:

[0047] 1. The present invention adopts a reinforcement learning collaborative controller to achieve accurate grasping of workpieces, avoiding frequent adjustments caused by the workpiece exceeding the range of motion of the robot arm, breaking through the limitations of traditional step-by-step execution, allowing movement and operation to be performed simultaneously, and improving the operating efficiency and control accuracy of the overall system;

[0048] 2. The laser SLAM solution adopted by the present invention has higher positioning accuracy and can effectively avoid positioning errors caused by regular scenes in the factory; in addition, the SLAM solution reduces the dependence on infrastructure such as magnetic stripes and QR codes, reducing deployment and maintenance costs; combined with the navigation method of the Autoware module, the mobile robot arm can quickly adapt to new working environments and tasks, achieving higher flexibility and efficiency;

[0049] 3. The OpenCV (Open Source Computer Vision Library, computer vision and machine learning software library) visual recognition and positioning module adopted by the present invention does not rely on a large amount of data for model learning, but recognizes multiple features of the color and shape of the workpiece, has a faster recognition speed, and can achieve higher recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a schematic diagram of a mobile mechanical arm used in the present invention;

[0051] Figure 2 It is a schematic diagram of the overall framework of the mobile mechanical arm grasping system of the present invention;

[0052] Figure 3 It is a flow chart of the implementation method of the mobile robot arm grasping system;

[0053] Figure 4 This is the framework diagram of the laser SLAM algorithm;

[0054] Figure 5 It is a framework diagram of reinforcement learning collaborative controller;

[0055] Figure 6 This is the Autoware module framework diagram;

[0056] Figure 7 It is the flow chart of OpenCV visual recognition and positioning module. DETAILED DESCRIPTION

[0057] The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0058] like Figure 1 The diagram shows the structure of the mobile robot arm. Figure 2 The figure shows the overall framework of the mobile robot gripping system. The mobile robot gripping system of the present invention includes a sensor, a robot arm, a mobile chassis and an industrial computer. The sensor is arranged on the robot arm, the robot arm is arranged on the mobile chassis, the industrial computer is arranged in the mobile chassis, and a gripping control system is arranged in the industrial computer. The sensor includes a binocular camera and a laser radar, and the gripping control system includes an OpenCV visual recognition and positioning module, a data storage module, a reinforcement learning collaborative controller module, a Moveit module and an Autoware module. The functions and introductions of each module are as follows:

[0059] The OpenCV visual recognition and positioning module uses a binocular camera to capture the spatial information of objects in real time. The module uses the OpenCV algorithm to identify and locate the workpiece and the placement table, and provides the location information of the target object. The OpenCV visual recognition and positioning module transmits the location information (x, y, z) of the identified target object to the reinforcement learning collaborative controller module to provide data support for subsequent grasping and placement.

[0060] The data storage module is used to store the position data of multiple workpieces and placement tables. By recording the dynamic positions of workpieces and placement tables, it supports the scheduling and optimization of grasping tasks. The module can also update the status of workpieces (grasped, not grasped) and the availability information of the placement table to provide real-time support for grasping and placing tasks. The data storage module queries and outputs the coordinate information of the relevant objects (workpieces or placement tables) according to the input number, and transmits this information to the reinforcement learning collaborative controller module to provide rough grasping and placement positions.

[0061] The reinforcement learning collaborative controller module realizes the collaborative control of the mobile chassis and the robotic arm in the mobile robotic arm system through the SAC reinforcement learning algorithm. This module receives the target information from the OpenCV visual recognition and positioning module and the data storage module, and dynamically adjusts the movement of the mobile chassis and the robotic arm through the pre-trained strategy network to optimize the grasping and placement process and achieve efficient operation. The instructions transmitted by the reinforcement learning collaborative controller to the Moveit module are the position (x, y, z) of the object to be grasped or placed in the robotic arm coordinate system; the instructions transmitted to the Autoware module are the position (x, y, θ) of the 2D mobile chassis in the world coordinate system. z ), where θ z is the rotation angle of the robot along the Z axis.

[0062] The Autoware module generates an executable path based on the target point location information (from the reinforcement learning collaborative controller) and drives the mobile chassis to move to the target point. This module integrates sensor data (such as lidar, binocular camera, etc.) for environmental perception, and combines the path planning algorithm to calculate the optimal movement trajectory to guide the mobile chassis to the target area.

[0063] After the Moveit module obtains the target position of the robot arm to be grasped (from the reinforcement learning collaborative controller), it dynamically calculates the motion trajectory according to the task requirements (such as grasping or placing) and adjusts the position of the robot arm in real time to ensure that the gripper reaches the target position accurately.

[0064] like Figure 3 The figure shows a flow chart of a method for implementing a mobile robot arm grasping system, which includes the following steps:

[0065] Step 1: Use laser SLAM (Simultaneous Localization And Mapping) to map the factory environment and perform initial rough calibration of the workpieces to be grasped and the placement table in the factory to obtain the approximate position coordinates of the workpieces and the placement table;

[0066] Furthermore, the specific contents include the following:

[0067] Step 11, such as Figure 4 The figure shows the framework of the laser SLAM algorithm. Continuous frame data is obtained from the point cloud data collected by the laser radar. The generalized iterative closest point algorithm (GICP) is used to perform three-dimensional point cloud registration between continuous frames to calculate the relative pose transformation between continuous frames.

[0068] Step 12: Determine the continuous keyframes through the keyframe screening mechanism, and perform 3D point cloud registration on the screened keyframes. Register the 3D point clouds between the keyframes based on GICP, and use the continuous frame registration results as the initial pose estimation for keyframe registration;

[0069] Step 13: Use the loop detection mechanism to detect loop frames by matching historical key frames with the current key frames. Once the loop frame is detected, the GICP algorithm is used to perform 3D point cloud registration and calculate the relative pose transformation of the loop frame to form a loop constraint.

[0070] Step 14, in the graph SLAM optimization, the graph optimization model is constructed with the lidar odometer, plane constraint, loop constraint and pose optimization as constraint conditions. Its purpose is to improve the accuracy of pose estimation through the collaborative optimization of multi-source information, thereby improving the accuracy and consistency of environmental mapping. The key frame registration results in the pose graph are used as nodes, and the pose transformations between frames, key frames and loop frames are combined to perform node constraints, making full use of the observation correlation information of different time steps and spatial positions to reduce the impact of cumulative errors.

[0071] Step 15, through the graph optimization model, obtain the optimized pose of the key frame after removing the noise, and generate the optimized point cloud map information.

[0072] Step 2: The user selects the workpiece to be grasped.

[0073] Step 3: The data storage module uploads the rough position information of the workpiece to be grasped to the reinforcement learning collaborative controller module according to the previously stored rough position information. After the reinforcement learning collaborative controller module parses the rough position, it dynamically plans the coordinated motion instructions of the robot arm and the mobile chassis, and sends them to the Autoware module and the Moveit module respectively.

[0074] Furthermore, the specific contents include the following:

[0075] Step 31, use Soft-Actor-Critic reinforcement learning as the training method, the framework is as follows Figure 5 As shown in the figure, the off-policy actor-critic framework based on maximum entropy reinforcement learning maximizes the entropy of the strategy while maximizing the expected reward as shown in formula (1), that is, ensuring the successful completion of the task while maximizing the exploration, where π * For the optimal strategy, represents the weighted average of all trajectories generated by strategy π. γ is the discount factor of reinforcement learning, r(s t ,a t ) means in state s t Take action a t The reward obtained by the agent, α is the adjustment parameter of the strategy entropy and reward, The entropy of the policy is used to measure the uncertainty or exploration of the policy;

[0076]

[0077] Step 32, designing a double Q network and a double V network;

[0078] Dual Q network, the two Q networks have the same structure, but the models are trained separately to avoid overestimation of Q value.

[0079] The dual V network consists of a V network and a Target V network. The two V networks have the same structure, but only the V network is trained, while the parameters of the Target V network are updated from the parameters of the V network through a soft update mechanism.

[0080] Step 33: Use Transformer as the neural network structure in the deep reinforcement learning model to replace the fully connected layer in traditional deep reinforcement learning:

[0081] In the Actor network, the network is divided into two parts: strategy selection and strategy improvement. The two networks share parameters. In the strategy selection stage, the network input is all the states of the robot in the environment for n steps. The input is passed to the Transformer layer after the fully connected layer is upgraded in dimension, and the association information between each step is generated and merged with the current step information. Subsequently, the mean and variance of the action are generated respectively through two fully connected layers. In the strategy improvement stage, the network receives the experience sequence from the experience replay pool. The sequence data processed by the Transformer layer does not need to be averaged and reduced in dimension, but is directly spliced ​​with the original data after the dimension is upgraded. This design aims to generate action data with sequence dimensions for Q network learning. The objective function J of the Actor network π (φ) is as shown in formula (2), where logπ φ (a t |s t ) means in state s t According to the strategy π φ Select action a t The logarithmic probability of is the entropy of the strategy, Q θ (s t ,a t ) means in state s t Take action a t Q value at:

[0082]

[0083] In the V network, there are two networks, one for estimating the current state value V(s t ) of the V network, one for calculating the target value V(s t+1 ) of the Target V network. The parameters of the Target V network are obtained by soft updating of the V network. The structure of the V network is similar to that of the Actor network selected by the strategy. The output of the V network is a one-dimensional value V(s t ), the objective function of the V network J V (ψ) is as shown in formula (3), where V ψ (s t ) represents the state s estimated by the V network with parameter ψ t The state value under state s t The expected cumulative return under:

[0084]

[0085] In the Q network, there are two networks, namely Q1 network and Q2 network. The structures of these two networks are exactly the same, but the parameters θ of the two networks are independent and updated independently. When selecting the final Q value, the minimum value of the output of these two networks is taken as the final Q value. The architecture of the Q network is very similar to that of the V network. The main difference lies in the input data: the input of the Q network is the superposition of the state dimension and the action dimension, while the input of the V network is the state dimension. The other structures of the Q network are consistent with the V network. The output result of the Q network is a one-dimensional numerical value Q(s t ,a t ), the objective function of the Q network is J Q (θ) is as shown in formula (4), where is the target value of the Q-value function, which is usually calculated using TD (Temporal Difference) as shown in formula (5), where For state s t+1 The V value under the condition is determined by the parameter The Target V network is generated as follows:

[0086]

[0087] During the training process, the neural network is updated every two iterations, the learning rate is set to 5e-4, the discount factor γ is set to 0.98, the size of the experience buffer is set to 1e5, the batch size is set to 256, and the random seed is set to 1.

[0088] Step 34: The reinforcement learning collaborative controller uses only the trained Actor network during the call process. The input of the network is the position (x, y, z) coordinates of the target object, and the output is the two-dimensional pose coordinates (x, y, θ z ), and the (x, y, z) coordinates of the object to be grasped by the robot arm in the robot arm coordinate system.

[0089] Step 4: The Autoware module (Autoware is the world's leading open source autonomous driving software built on ROS (Robot Operating System)) generates an executable path based on the target point location information and drives the mobile chassis to move to the target point. Its framework is as follows: Figure 6 As shown in the figure, the IMU (Inertial Measurement Unit) is used for posture estimation. At the same time, after the Moveit module (Moveit, a motion planning library in the open source Robot Operating System (ROS)) obtains the target position of the robotic arm, it dynamically adjusts the position of the robotic arm during the movement, so that the onboard camera can observe a larger area to identify and accurately locate the workpiece;

[0090] Furthermore, the specific contents include the following:

[0091] Step 41, using the point cloud data obtained by the laser radar to identify obstacles, using the KNN (K-Nearest Neighbor) algorithm to pre-process and segment the point cloud, calculating the Euclidean distance between points based on the distance threshold in the three-dimensional space, clustering the point cloud, and calculating the distance between the surrounding objects and the mobile chassis.

[0092] Step 42, the planning module is responsible for generating the motion trajectory of the mobile chassis based on the output of the reinforcement learning collaborative controller module, which is divided into two aspects: task planning and motion planning. The Autoware module generates a global trajectory based on the relative position of the current position and the given destination through a global path planning algorithm. The local path planning combines the hybrid A* algorithm (HybridA*) to consider factors such as the state of the mobile chassis, three-dimensional map information, and obstacle positions to generate a local trajectory suitable for obstacle avoidance. The global and local paths are processed by the motion planning module to generate the final motion trajectory for subsequent path tracking control of the mobile chassis.

[0093] Step 43, path tracking uses a pure tracking algorithm to decompose the global path into multiple discrete path points, and searches for the next closest path point within a certain threshold distance in each control cycle. The trajectory tracking module sets the linear velocity and angular velocity of the mobile chassis according to the relative position of the next path point so that the mobile chassis moves to the specified path point according to the predefined curvature, reducing the angle mutation during turning or path deviation.

[0094] Step 44, the bottom control module of the mobile chassis drives the mobile chassis according to the output linear velocity and angular velocity instructions of the trajectory tracking module. This part adopts a PID controller, which adjusts the controller output to drive the rotation speed of the two track wheels by comparing the difference between the current state and the target state of the mobile chassis.

[0095] Step 5: Start the OpenCV visual recognition and positioning module to identify the workpiece. The workpiece image is obtained through the binocular camera. The image is first pre-processed by Gaussian filtering, and the color recognition and shape feature extraction are used to achieve accurate recognition and positioning of the workpiece, and the position information of the workpiece is uploaded to the reinforcement learning collaborative controller module.

[0096] Further, such as Figure 7 The figure shows the flow chart of the OpenCV visual recognition and positioning module. The specific steps of the OpenCV visual recognition and positioning module include the following:

[0097] Step 51, using a binocular camera to obtain an image of the workpiece to be grasped, and transmitting the image to an OpenCV visual recognition and positioning module.

[0098] First, the image is preprocessed. The preprocessing step includes Gaussian filtering to remove noise and improve the image clarity and fidelity of feature information. The main template value function of Gaussian filtering is as follows:

[0099]

[0100] Where M(i,j) represents the value of the Gaussian kernel at position (i,j), the value of σ determines the width of the template. The larger the value of σ, the wider the range of the Gaussian filter. k is a constant used for translation operations, which refers to the offset of the center position of the Gaussian kernel.

[0101] Step 52: Identify the workpiece to be grasped based on the color feature.

[0102] By converting the image from RGB format to HSV format, the conversion is completed using the cvtColor function of OpenCV, and the preset color boundary array is used to detect the target color and identify different color blocks to determine the workpiece to be grasped;

[0103] Step 53, using the threshold segmentation method to segment the workpiece image, using the threshold function in OpenCV to separate the workpiece from the environment in the image. The threshold segmentation method compares the gray value of each pixel with a given threshold to achieve effective segmentation of the workpiece and the background;

[0104] Step 54, extracting the workpiece contour by edge detection algorithm.

[0105] The Canny operator is used for edge detection. Its principle is based on the evolution of the Sobel operator. It uses high and low thresholds to classify and filter edge pixels. Pixels above the high threshold are marked as strong edges, pixels below the high threshold but above the low threshold are marked as weak edges, and pixels below the low threshold are removed. Finally, a high-quality contour image is generated, which is convenient for subsequent processing. The general ratio of the two thresholds is 2:1 or 3:1.

[0106] Step 55, after the recognition is completed, the position and posture of the workpiece are determined.

[0107] For cylindrical workpieces, the least squares method is used to determine the center point coordinates and radius of the cylindrical workpiece. According to the contour data of the recognition result, the center position of the workpiece and the cylindrical radius are calculated to provide accurate posture information for the grasping operation. Assuming the coordinates of the center of the circle are (A, B) and the radius is R, the expression of the circle is:

[0108] R 2 =(xA) 2 +(yB) 2 (7)

[0109] Right now:

[0110] x 2 +y 2 +ax+by+c=0 (8)

[0111] Then the edge contour sample set (x i ,y i )The difference between the square of the distance from the point to the center of the circle and the square of the radius is:

[0112]

[0113] According to the principle of least squares, we construct By solving the parameters a, b, c to minimize the value of Q(a, b, c), setting the partial derivative to 0, we can find the extreme point:

[0114]

[0115] By solving the values ​​of a, b, and c, we can get the estimated fitting values ​​of A, B, and R. At this point, the center coordinates and radius of the circle to be grasped have been solved.

[0116] Step 6: After the reinforcement learning collaborative controller module receives the position of the target workpiece, it sends instructions to the Autoware module and the Moveit module respectively to control the coordinated movement of the mobile chassis and the robotic arm to perform synchronous grasping operations; the operation process of this step is the same as steps 3 and 4.

[0117] Step 7: After the grabbing instruction is executed, it is determined whether the object is grabbed;

[0118] When the gripper sensor does not detect that the object has been grasped, the robotic arm extends and re-identifies and grasps the object.

[0119] If the gripper sensor detects that an object has been grasped, the data storage module uploads the previously stored placement position to the reinforcement learning collaborative controller module. After the module parses the rough position, it calculates the target position of the mobile chassis and the target position of the robotic arm, and sends them to the Autoware module and Moveit module respectively. The operation process of this step is the same as step 3.

[0120] In step 8, the Autoware module generates an executable path based on the target point position information and drives the mobile chassis to move to the target point. At the same time, after the Moveit module obtains the target position of the robotic arm, it drives the robotic arm to extend so that the mounted camera can observe a larger area for visual recognition and positioning of the workpiece placement table. The operation process of this step is the same as step 4.

[0121] Step 9: Start the OpenCV visual recognition and positioning module to identify the placement table. After the placement table is identified, its coordinate position is determined and the information is uploaded to the reinforcement learning collaborative controller module; the operation process of this step is the same as step 5.

[0122] Step 10: After the reinforcement learning collaborative controller module receives the position of the placement table, it sends instructions to the Autoware and Moveit modules respectively to control the coordinated movement of the mobile chassis and the robotic arm to perform the synchronous placement operation; the operation process of this step is the same as step 3.

[0123] Step 11: After the workpiece is placed, the data storage module publishes a new initial position and uploads it to the reinforcement learning collaborative controller module. After parsing the rough position, the module calculates the target position of the mobile chassis and the target position of the robotic arm, and sends them to the Autoware module and Moveit module respectively to control the robotic arm to retract and fold and control the mobile chassis to return. The operation process of this step is the same as steps 3 and 4. At this point, the grasping and placement task is completed.

[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A mobile robot arm grasping system based on reinforcement learning collaborative controller, characterized in that: It includes a sensor, a robotic arm, a mobile chassis and an industrial computer. The sensor is arranged on the robotic arm, the robotic arm is arranged on the mobile chassis, the industrial computer is arranged in the mobile chassis, and a grasping control system is arranged in the industrial computer; the grasping control system includes an OpenCV visual recognition and positioning module, a data storage module, a reinforcement learning collaborative controller module, a Moveit module and an Autoware module; the sensor includes a binocular camera and a laser radar; The OpenCV visual recognition and positioning module captures the spatial information of the object in real time through a binocular camera, uses the OpenCV algorithm to identify and locate the workpiece and the placement table, and provides the position information of the target object; And transmit the position information of the identified target object to the reinforcement learning collaborative controller module; The data storage module is used to store the position data of multiple workpieces and placement tables, and supports the scheduling and optimization of grasping tasks by recording the dynamic positions of the workpieces and the placement tables; updates the status of the workpieces and the availability information of the placement tables to provide real-time support for grasping and placing tasks; queries and outputs the coordinate information of the workpieces or the placement tables according to the input number, and transmits this information to the reinforcement learning collaborative controller module; The reinforcement learning collaborative controller module realizes the collaborative control of the mobile chassis and the robotic arm through the SAC reinforcement learning algorithm, receives the target information from the OpenCV visual recognition and positioning module and the data storage module, and dynamically adjusts the movement of the mobile chassis and the robotic arm through the pre-trained strategy network to optimize the grasping and placing process; transmits the position of the object to be grasped or placed in the robotic arm coordinate system to the Moveit module; and transmits the position and posture of the 2D mobile chassis in the world coordinate system to the Autoware module; The Autoware module generates an executable path based on the target point location information and drives the mobile chassis to move to the target point, integrates sensor data for environmental perception, and calculates the optimal movement trajectory in combination with the path planning algorithm to guide the mobile chassis to reach the target area; After the Moveit module obtains the target position to be grasped by the robotic arm, it dynamically calculates the motion trajectory according to the task requirements and adjusts the position of the robotic arm in real time to ensure that the gripper accurately reaches the target position.

2. A mobile robot arm grasping implementation method based on reinforcement learning collaborative controller, characterized in that: Used in the mobile robot gripping system as claimed in claim 1, the automatic gripping or placement of the workpiece is achieved through the coordinated control of the robot arm and the mobile chassis; The steps include: S1, use laser SLAM to map the factory environment, perform initial rough calibration of the workpieces to be grasped and the placement table in the factory, and obtain the approximate position coordinates of the workpieces and the placement table; S2, the user selects the workpiece to be grasped; S3, the data storage module uploads the rough position information of the workpiece to be grasped to the reinforcement learning collaborative controller module according to the previously stored rough position information. After the reinforcement learning collaborative controller module parses the rough position, it dynamically plans the coordinated motion instructions of the robot arm and the mobile chassis, and sends them to the Autoware module and the Moveit module respectively; S4, the Autoware module generates an executable path based on the target point position information and drives the mobile chassis to move to the target point. At the same time, after the Moveit module obtains the target position of the robotic arm, it dynamically adjusts the position of the robotic arm during the movement; S5, start the OpenCV visual recognition and positioning module to identify the workpiece; pre-process the workpiece image obtained by the binocular camera, and determine the workpiece to be grasped by extracting color and shape features, and upload the position information of the workpiece to the reinforcement learning collaborative controller module; S6, after the reinforcement learning collaborative controller module receives the position of the target workpiece, it sends instructions to the Autoware module and the Moveit module respectively to control the coordinated movement of the mobile chassis and the robotic arm to perform synchronous grasping operations; S7, after the grab command is executed, determine whether the object is grabbed: When the gripper sensor does not detect that the object has been grasped, the robot arm extends and re-identifies and grasps; When the gripper sensor detects that an object has been grasped, the data storage module uploads the previously stored placement position to the reinforcement learning collaborative controller module. The reinforcement learning collaborative controller module calculates the target position of the mobile chassis and the target position of the robotic arm, and sends them to the Autoware module and Moveit module respectively; S8, the Autoware module generates an executable path based on the target point position information and drives the mobile chassis to move to the target point. At the same time, after the Moveit module obtains the target position of the robotic arm, it drives the robotic arm to extend; S9, start the OpenCV visual recognition and positioning module to identify the placement table; after the placement table is identified, its coordinate position is determined and the information is uploaded to the reinforcement learning collaborative controller module; S10, after receiving the position of the placement table, the reinforcement learning collaborative controller module simultaneously sends instructions to the Autoware module and the Moveit module respectively to control the coordinated movement of the mobile chassis and the robotic arm to perform a synchronous placement operation; S11, when the workpiece is placed, the data storage module publishes the new initial position and uploads it to the reinforcement learning collaborative controller module. The reinforcement learning collaborative controller module calculates the target position of the mobile chassis and the target position of the robotic arm, and sends them to the Autoware module and Moveit module respectively to control the robotic arm to retract and fold and control the mobile chassis to return.

3. The mobile robot arm grasping implementation method based on reinforcement learning collaborative controller according to claim 1 is characterized in that: The initial rough calibration of the workpiece to be grasped and the placement table includes the following steps: S11, obtaining continuous frame data from the point cloud data collected by the laser radar, using the generalized iterative closest point algorithm to perform three-dimensional point cloud registration between continuous frames, and calculating the relative pose transformation between continuous frames; S12, determining continuous key frames through a key frame screening mechanism, performing three-dimensional point cloud registration on the screened key frames, and using the continuous frame registration results as initial pose estimation for key frame registration; S13, using a loop detection mechanism to detect loop frames by matching historical key frames with current key frames; if a loop frame is detected, performing 3D point cloud registration, calculating relative pose transformation of the loop frame, and forming a loop constraint; S14, using the laser radar odometer, plane constraint, loop constraint and pose optimization as constraint conditions, constructing a graph optimization model; using the key frame registration results in the pose graph as nodes, combining the pose transformations between frames, between key frames and between loop frames to perform node constraints; S15, through the graph optimization model, the optimized pose of the key frame after noise removal is obtained, and the optimized point cloud map information is generated.

4. The mobile robot arm grasping implementation method based on reinforcement learning collaborative controller according to claim 1 is characterized in that: In step S3, the implementation of the reinforcement learning collaborative controller module includes: Soft-Actor-Critic reinforcement learning is used as the training method, and the off-policy actor-critic framework based on maximum entropy reinforcement learning maximizes the entropy of the strategy while maximizing the expected reward; Adopt dual Q network and dual V network; Transformer is used as the neural network structure in the deep reinforcement learning model as the fully connected layer: During the calling process, only the trained Actor network is used. The input of the Actor network is the position coordinates of the target object, and the output is the two-dimensional pose coordinates in the geodetic coordinate system of the mobile chassis and the coordinates of the object to be grasped by the robotic arm in the robotic arm coordinate system.

5. The mobile robot arm grasping implementation method based on reinforcement learning collaborative controller according to claim 1 is characterized in that: In step S4, an executable path is generated according to the target point position information and the mobile chassis is driven to move to the target point, which specifically includes the following steps: S41, identifies obstacles through point cloud data obtained by laser radar, uses KNN algorithm to preprocess and segment the point cloud, calculates the Euclidean distance between points according to the distance threshold in three-dimensional space, clusters the point cloud, and calculates the distance between surrounding objects and the mobile chassis; S42, the planning module generates the motion trajectory of the mobile chassis according to the output of the reinforcement learning collaborative controller module, which is divided into task planning and motion planning; the global trajectory is generated according to the relative position between the current position and the given destination through the global path planning algorithm; the local path planning is combined with the hybrid A* algorithm, considering the state of the mobile chassis, the three-dimensional map information and the obstacle position, to generate a local trajectory suitable for obstacle avoidance; the global and local paths are processed by the motion planning module to generate the final motion trajectory; S43, path tracking uses a pure tracking algorithm to decompose the global path into multiple discrete path points, and searches for the next closest path point within a certain threshold distance in each control cycle; the trajectory tracking module sets the linear velocity and angular velocity of the mobile chassis according to the relative position of the next path point; S44, the bottom control module of the mobile chassis drives the mobile chassis according to the output linear velocity and angular velocity instructions of the trajectory tracking module, and uses a PID controller to adjust the controller output to drive the rotation speed of the two tracks by comparing the difference between the current state and the target state of the mobile chassis.

6. The mobile robot arm grasping implementation method based on reinforcement learning collaborative controller according to claim 1 is characterized in that: In step S5, the specific steps of visual recognition and positioning by the OpenCV visual recognition and positioning module are as follows: S51, preprocessing the image of the workpiece to be grasped obtained by the binocular camera; S52, converting the image from RGB format to HSV format, using the cvtColor function of OpenCV to complete the conversion, and using a preset color boundary array to detect the target color, identifying different color blocks to determine the workpiece to be grasped; S53, using the threshold segmentation method to segment the artifact image, using the threshold function in OpenCV to separate the artifact from the environment in the image; S54, extracting the contour of the workpiece by an edge detection algorithm, marking pixels above a high threshold as strong edges, marking pixels below a high threshold but above a low threshold as weak edges, and removing pixels below a low threshold; S55, determining the position and posture of the workpiece according to the contour data of the recognition result.

Images