A method for using an omnidirectional AGV based on cooperative grabbing of small devices by double mechanical arms

Abstract

The application discloses a kind of based on the use method of omnidirectional AGV of double mechanical arm cooperation and capture small device, comprising;Step one: through touch display, use man-machine interaction page, order is issued;Step two: system is according to order, automatically match beforehand and build good map stored in local;Step three: the attitude information of current omnidirectional AGV is obtained by MPU9250, real-time positioning is carried out using AMCL;Step four: using Mecanum wheel, DC brushless motor, encoder and the drive circuit of adaptation complete the omnidirectional movement of AGV;Step five: global path planning is carried out by Dijkstra algorithm;Step six: the attitude of small device is estimated;Step seven: position information is sent to AI control unit, and small device is captured in real time by direction bounding box algorithm cooperation double mechanical arm, step eight: whether the device required in order is all completed capture is judged.The application effectively improves the transport efficiency of small device, avoids the wear and tear and scratch of device.

Description

Technical Field

[0001] This invention belongs to the field of omnidirectional AGV navigation and computer vision technology, specifically relating to a method for using an omnidirectional AGV based on the collaborative grasping of small devices by two robotic arms. Background Technology

[0002] In the context of next-generation intelligent manufacturing applications, the handling and transportation of small components need to adapt to multi-objective production modes and improve flexibility. For most small components of various types and in small batches, fixed conveyor belt transportation is currently too costly, and the distribution of components to fixed workstations mainly relies on manual labor and fixed-station robots.

[0003] This transportation model still has two problems: First, robots at fixed workstations complete part of the work, which cannot meet the needs of flexibility. Second, manual handling is prone to wear and scratches.

[0004] Currently available on the market are submersible magnetic navigation AGVs (intelligent handling robots) for lifting and unloading, and custom-made trolleys for stationary parts handling. However, some of these AGVs are not suitable for small parts, while others are only suitable for fixed locations, lacking sufficient flexibility. Therefore, they lack practical value for promotion when dealing with small parts, and their cost-effectiveness does not meet market requirements. Summary of the Invention

[0005] To overcome the problems existing in the prior art, the present invention aims to provide a method for using an omnidirectional AGV based on the collaborative gripping of small components by dual robotic arms. Navigation is achieved using an IMU (Inertial Measurement Unit), an RGB-D (Depth Camera), and a LiDAR three-way odometer. A vehicle-mounted collaborative dual robotic arm is employed. After the vehicle-mounted industrial camera identifies the component to be gripped, the flexible robotic gripper grasps and neatly stores the component without damage. This effectively improves the transportation efficiency of small components and avoids wear and scratches.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for using an omnidirectional AGV based on the collaborative grasping of small components by dual robotic arms includes the following steps;

[0008] Step 1: Place an order using the touchscreen display and the human-computer interaction page;

[0009] Step 2: The system automatically matches the pre-built map stored locally based on the order;

[0010] Step 3: Obtain the current attitude information of the omnidirectional AGV through the MPU9250, and use the AMCL algorithm for real-time positioning;

[0011] Step 4: Use Mecanum wheels, DC brushless motors, encoders, and compatible drive circuits to complete the omnidirectional movement of the AGV;

[0012] Step 5: Perform global path planning using Dijkstra's algorithm, and then use the TEB algorithm to plan a local path from the current location to the local target point when local obstacles are detected.

[0013] Step 6: The distance between the small device and the AGV is obtained in real time by RGB-D placed on the AGV. The omnidirectional AGV is navigated to the location of the device to be grasped. The small device is identified by an on-board industrial camera using the Yolov5 algorithm based on an improved attention mechanism and multi-scale features. The pose of the small device is estimated by the Canny contour and pose acquisition algorithm.

[0014] Step 7: Communicate between the dual robotic arms and the omnidirectional AGV via the CAN bus. Use a time series prediction algorithm combined with the attitude of the small device to predict its position and send the position information to the AI ​​control unit (e.g., STM32F407ZET6). Use the orientation bounding box dual robotic arm cooperative grasping algorithm to coordinate the dual robotic arms to grasp the small device in real time, while ensuring that the dual robotic arms can cooperate efficiently to complete the specified task without collision during actual operation.

[0015] Step 8: Determine if all required components in the order have been picked up. If not, the AGV will start from Step 6 again. If all components have been picked up, the AGV will deliver the components to the receiving point and then return to the default starting point to prepare for the next task.

[0016] In step one, orders are placed by opening the human-computer interaction page in the AI ​​computing unit (egi5-8260u industrial computer) via a touch screen.

[0017] In step two, RGB-D and LiDAR dual odometers are combined to use RTAB to construct a 3D dense map. The constructed map is stored in the AI ​​computing unit. After the order is placed, the map with the highest probability is adapted according to the types of each device required in the order.

[0018] The method for matching maps is as follows: The required component types are determined from the order details. A map is constructed using the RTAB algorithm. For each map, a sequence is created to store all component types present in that map. When the system matches a map to the required component types from the order, it first calculates the number of required component types. Only maps where the total number of component types in the sequence (the sum of their indices) is greater than or equal to the number of required component types are traversed. Due to the diverse nature of small components, this method significantly reduces the number of calculations. Afterward, the system traverses the remaining maps, sorts them according to their probability, and stores them for later use.

[0019] In step three, the MPU9250 module is used to obtain the raw data of the AGV's current acceleration and angular velocity. The raw data is processed in the AI ​​control unit to obtain the AGV's current pose information, which is then transmitted to the AI ​​computing unit via the CAN bus.

[0020] For the acceleration data and gyroscope data acquired by the MPU9250, quaternions are used to describe the relationship between the coordinate systems:

[0021] P = [s, x, y, z] T =[s,v] T

[0022] First, the FISR algorithm is used to solve for the reciprocal of the modulus:

[0023]

[0024]

[0025] Multiplying the three acceleration vectors of the IMU by the reciprocal of the magnitude yields the three-dimensional unit vector.

[0026] According to the following formula, each gravity component is estimated based on the current quaternion attitude value, and then compared with the gravity components actually measured by the accelerometer, thereby correcting the quadcopter attitude.

[0027]

[0028] The error between the estimated and measured directions of gravity is calculated using the sum of cross-products:

[0029]

[0030] The gravity error calculated above is proportionally calculated, with a proportionality coefficient of Kp. The result of the calculation is accumulated into the gyroscope data to correct the gyroscope data.

[0031] Through the above calculations, the gyroscope data corrected by the accelerometer is obtained. The corrected gyroscope data is integrated into the quaternion. Finally, the quaternion obtained from the above calculations is normalized to obtain the new quaternion after rotation.

[0032]

[0033] In step four, the rotation direction of the four brushless DC motors is determined by force analysis to achieve omnidirectional movement of the AGV. The serial PID algorithm is used to adjust the PWM value through the AI ​​control unit to control the speed of the Mecanum wheel. The AI ​​control unit uses the quadruple frequency technology to perform inverse kinematics on the four 90° AB phase encoders to obtain the current speed of the AGV, and then transmits it to the AI ​​computing unit through the CAN bus.

[0034] By placing an RGB-D sensor and two robotic arms on an AGV, the following can be obtained:

[0035] V arm =V depthcamera =V AGV实时

[0036] In step five, the 3D dense map established in step two is compressed and filtered to remove ground information using CostMap, and a raster map is generated through 2D projection. An Obstacles layer is added to maintain the obstacle information obtained by fusing the LiDAR and RGB-D scans. On the 2D raster map, Dijkstra's algorithm is used for global path planning to find a feasible shortest path from the starting point to the target point. The global path consists of discrete path points, considering only the static obstacle information during map building. The TEB algorithm is used to complete local path planning for obstacle avoidance. When the omnidirectional AGV gets stuck in navigation, it rotates in place to forcibly clear the residual obstacles in the cost map, performs recovery behavior, and gets out of trouble.

[0037] In step six, the distance between the small device and the AGV is first obtained in real time by using an RGB-D sensor placed on the AGV.

[0038] Improved Yolov5 network: Introduce ShuffleAttention (SA) mechanism, improve the further mining of small target features by adding a new 104*104 feature map, adopt multi-scale feedback to introduce global context information to improve the recognition ability of small targets in the image, and adopt CIoU as a new target box regression loss function to enhance the convergence accuracy of the small device detection model.

[0039] Training weight file: To obtain the small device detection model, the created small device dataset is put into the improved Yolov5 network for training. The learning environment is configured and the number of training categories and label names are modified. After training, the small device detection model is obtained, and the detection effect is tested.

[0040] Based on Canny device contour and pose acquisition: Since the small target detection model has detected the category and simple position information of small devices on the production line, it is necessary to further obtain the device contour and centroid, and obtain the pose of the small device in the robot arm coordinate system. Based on the deep learning detection results, image cropping, Gaussian blur, morphological closing operation, edge detection, and IoU calculation are performed to realize the detection of the small device contour and centroid. Then, its pose is calculated. Finally, the projection distance between the two centers on the plane is L, the angle between the device and the z-axis is θ, and the height difference between the two centers is ;

[0041]

[0042]

[0043] H = Z1 - Z2

[0044] The distance between a small device and the RGB-D sensor is calculated using similar triangles in RGB-D. First, assuming the width of the detected object is w, the distance between the object and the camera is d, and the pixel width of the object in the image is p, the camera focal length f is obtained using the following formula:

[0045] f = (p·d) / w

[0046] The distance d′ between the moving object and the camera is calculated by applying similar triangles:

[0047] d′=(w·f) / p

[0048] The pixel coordinate system is defined as follows: the origin is located at the top left corner o′ of the image, the u-axis is parallel to the x-axis to the right, the v-axis is parallel to the y-axis to the bottom, and the pixel coordinates are scaled by a factor of α on the u-axis and by a factor of β on the v-axis, with a translation distance of [d]. x ,d y ] T The relationship between pixel coordinates and image physical coordinates is expressed as:

[0049]

[0050] The similar triangles formed by the camera focal length f and the distance d between the object and the camera yield the following:

[0051]

[0052] Where f is in meters, α and β are in pixels per meter, and let:

[0053]

[0054] Combining the two equations and writing them in matrix form, we get:

[0055]

[0056] The intermediate formula is called the camera's intrinsic parameter matrix K, and the extrinsic parameters are described by the rotation matrix R and the translation vector t.

[0057] The RGB-D is placed on the AGV, and the distance d between the RGB-D and the small device, as well as the pose descriptions of depthcamera_link and base_link, are obtained.

[0058] The quaternion can be converted to a rotation matrix R using the following formula:

[0059]

[0060] In step seven, the design of the dual-robotic arm grasping collaborative algorithm is as follows:

[0061] By using device type recognition and pose detection based on the fusion of deep learning and image processing, the coordinates of various devices in the robotic arm coordinate system have been obtained. A dual-robotic arm collaborative grasping algorithm is used to grasp multiple types of small devices. To achieve real-time and accurate collaborative grasping by the two robotic arms, the following process was set:

[0062] First, the forward and inverse kinematics models of the six-axis serial robotic arm used in the system were derived and modeled; the specific rules for the cooperative motion of the two robotic arms are as follows:

[0063] An industrial camera captures images of devices placed on a workbench, which are then uploaded to an AI computing unit. Based on deep learning and machine vision, device category and contour detection methods are used to detect the device images, thereby obtaining the device category and pose pixel coordinates. The presence of a device in the image is determined to control whether the program ends. Using the calibration results of the industrial camera, the pixel coordinates of the small devices in the image are converted into the gripping coordinates of the dual robotic arms. The dual robotic arm geometric inverse kinematics solution system calculates the rotation angles of each joint of the dual robotic arms based on the gripping point coordinates of the small device and the real-time speed of the small device. Finally, this data is sent to the dual robotic arm drive module via the CAN bus to control the dual robotic arms to collaboratively grasp the small device.

[0064] Step three yields the coordinate relationship between base_link and imu_link, and step four yields the real-time moving speed of the AGV, denoted as V. AGV实时 Let V be the moving speed of the robotic arm and the moving speed of the RGB-D. arm and Vdepthcamera And V arm =V depthcamera =V AGV实时 Step six obtains the distance d between RGB-D and the small device, the pose descriptions of base_link and depthcamera_link, and the contour and pose of the small device, and calculates the time t when the dual robotic arms start moving.

[0065] In step eight, the AI ​​computing unit determines whether all the required components in the order have been successfully grabbed. If not, the AGV starts executing from step six again.

[0066] Once all items have been captured, the AGV delivers the device to the receiving point, then returns to its default starting point and prepares for the next task.

[0067] The beneficial effects of this invention are:

[0068] This invention realizes an omnidirectional AGV with dual robotic arms collaboratively grasping devices. It can flexibly grasp small devices of various sizes and types, changing the traditional grasping approach that targets a single device and a single action. It enables dynamic combination of different production lines to work together, and can orderly and non-destructively grasp various types of devices as needed. This invention has an accuracy rate of ≥97% in recognizing small devices. It can complete the non-destructive grasping and orderly storage of small devices without collisions by relying on the collaborative action of dual robotic arms and flexible mechanical grippers. It can move in all directions and has practical application value. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of the process of this invention.

[0070] Figure 2 This is a schematic diagram of similar triangles along the X and Y axes provided by the present invention.

[0071] Figure 3 This is a diagram illustrating the steps for calculating the tilt angle of a small device provided by the present invention.

[0072] Figure 4 The overall workflow diagram of the dual robotic arm system provided by this invention. Detailed Implementation

[0073] The present invention will now be described in further detail with reference to the accompanying drawings.

[0074] like Figures 1-4As shown: First, the worker selects the type and quantity of small components to be picked up in the human-machine interface and places an order. The system automatically matches the pre-built navigation map stored locally based on the order. The AGV adjusts its posture according to the map, performs path planning and navigation to the production line of the required component, and uses an onboard industrial camera to determine whether the component produced on this production line is the one to be picked up.

[0075] If the required device is being picked up, a dual-arm onboard robotic system uses flexible grippers to pick it up non-destructively and place it into a tray on the AGV. Simultaneously, the system checks if all devices in the order have been picked up. If not, the AGV navigates to the next production line for another check. If the order is complete, all required devices are delivered by the AGV to the receiving point, then the system returns to the default starting point and prepares for the next task.

[0076] If the device being picked up is not the one that needs to be picked up, the AGV excludes the possibility of using the current map and checks if there is another possible map. If there is, the AGV navigates to the production line according to the next map and checks again. If there is no map, the AGV displays an error message on the human-machine interface and returns to the default starting point, waiting for the next task to be executed.

[0077] The application principle of the present invention will be further explained below with reference to the accompanying drawings:

[0078] like Figure 1 The diagram shown is the overall algorithm flowchart of the method of the present invention. The method of using the omnidirectional AGV with dual robotic arm cooperative grasping device described in the present invention is carried out according to the following steps:

[0079] Step 1: Use the touchscreen to open the human-computer interaction page in the AI ​​computing unit to place the required orders.

[0080] Step 2: A 3D dense map is constructed using the RTAB algorithm, which integrates RGB-D and LiDAR dual-oddscopy. The constructed map is stored in the AI ​​computing unit. After an order is placed, the map is adapted to the most likely candidate based on the types of components required in the order.

[0081] The method for matching maps is as follows: The required component types are determined from the order details. After constructing the map using the RTAB algorithm, a sequence is created for each map to store all component types present in that map. When the system matches maps based on the required component types from the order, it first calculates the number of required component types. Only maps where the total number of component types in the sequence (the sum of their indices) is greater than or equal to the number of required component types are traversed. Due to the diverse nature of small components, this method significantly reduces the number of calculations. Afterward, the system traverses the remaining maps, selects those containing all required components, sorts them according to their probability, and stores them for later use.

[0082] Step 3: The MPU9250 module acquires raw data such as the current acceleration and angular velocity of the AGV. The processing of the raw data is carried out in the AI ​​control unit to obtain the current pose information of the AGV. The communication between the AI ​​control unit and the AI ​​computing unit is carried out through the CAN bus.

[0083] Because of the compactness and lack of singularity of quaternions, quaternions are used to describe the relationship between coordinate systems for acceleration and gyroscope data acquired by the MPU9250:

[0084] P = [s, x, y, z] T =[s,v] T

[0085] First, the FISR algorithm is used to solve for the reciprocal of the modulus:

[0086]

[0087]

[0088] Multiplying the three acceleration vectors of the IMU by the reciprocal of the magnitude yields the three-dimensional unit vector.

[0089] The following formula is used to estimate each gravity component based on the current quaternion attitude value. This estimate is then compared with the gravity components actually measured by the accelerometer to correct the quadcopter attitude.

[0090]

[0091] The error between the estimated and measured directions of gravity is calculated using the sum of cross-products:

[0092]

[0093] A proportional calculation is performed on the gravity error obtained above. The proportionality coefficient is Kp. The result of the calculation will be added to the gyroscope data to correct the gyroscope data.

[0094] Through the above calculations, we obtained gyroscope data corrected by the accelerometer. The next step is to integrate the corrected gyroscope data into a quaternion. Finally, we normalize the quaternion obtained from the above calculations to obtain a new quaternion after rotation.

[0095]

[0096] Step 4: The AI ​​control unit uses a quadruple frequency multiplication technique to perform inverse kinematics on the four 90° AB phase encoders to obtain the current speed of the AGV, and communicates with the AI ​​computing unit via the CAN bus. Force analysis determines the rotation direction of the four brushless DC motors, thus enabling omnidirectional movement of the AGV. A serial PID algorithm is used to adjust the PWM value through the AI ​​control unit to control the wheel speed, thereby achieving speed control of the AGV.

[0097] Step 5: Compress and filter out ground information using CostMap on the 3D dense map created in Step 2, and generate a raster map through 2D projection. Add an Obstacles layer to maintain obstacle information obtained by fusing LiDAR and RGB-D scans.

[0098] On a 2D raster map, Dijkstra's algorithm is used for global path planning to find the shortest feasible path from the starting point to the target point. The global path consists of discrete path points and only considers static obstacle information during map construction. It cannot avoid obstacles that suddenly appear on the map. Therefore, the TEB algorithm is used to complete local path planning for obstacle avoidance.

[0099] When an omnidirectional AGV gets stuck in navigation, it can forcefully clear residual obstacles in the cost map by rotating in place, thus recovering and getting out of trouble.

[0100] Step Six: Considering the cost of omnidirectional AGVs, select a suitable AI computing unit. Considering the performance of this AI computing unit, improve it based on the Yolov5 algorithm.

[0101] The Yolov5 algorithm is improved by using both frequency and time domain scales to enhance recognition accuracy. Finally, the pose of small devices is estimated by using an algorithm based on Canny contour and pose acquisition.

[0102] Improved Yolov5 network: Based on the original Yolov5, a ShuffleAttention (SA) mechanism is introduced to improve the detection accuracy of similar targets. Simultaneously, a new 104*104 feature map is added to further mine features of small targets, and multi-scale feedback is employed to incorporate global contextual information to enhance the recognition ability of small targets in images. CIoU is used as a new bounding box regression loss function to enhance the convergence accuracy of the small device detection model.

[0103] Training weight file: To obtain the small device detection model, the created small device dataset is fed into the improved Yolov5 network for training. The deep learning environment is configured and the number of training categories and label names are modified.

[0104] Based on Canny device contour and pose acquisition: The small target detection model has detected the category and basic position information of small devices on the production line. Further steps are needed to obtain the device contour and centroid, and the pose of the small device in the robot arm coordinate system. Therefore, based on the deep learning detection results, image cropping, Gaussian blurring, morphological closing operation, edge detection, and IoU calculation are performed to achieve small device contour and centroid detection, and then its pose is calculated. Finally, the calculated projection distance between the two center points on the plane is L, the angle between the device and the z-axis is θ, and the height difference between the two center points is H.

[0105]

[0106]

[0107] H = Z1 - Z2

[0108] The distance between a small device and the RGB-D image is calculated using similar triangles from the RGB-D image. First, assuming the width of the detected object is w, the distance between the detected object and the camera is d, and the pixel width of the object in the image is p, the formula for the camera focal length f can be obtained:

[0109] f = (p·d) / w

[0110] like Figure 2 As shown, the distance d′ between the moving object and the camera is calculated using similar triangles:

[0111] d′=(w·f) / p

[0112] The pixel coordinate system is defined as follows: the origin is located at the top left corner o′ of the image, the u-axis is parallel to the x-axis to the right, and the v-axis is parallel to the y-axis downwards. The pixel coordinates are scaled by a factor of α on the u-axis and by a factor of β on the v-axis. The translation distance is [d]. x ,d y ] T The relationship between pixel coordinates and image physical coordinates can be expressed as:

[0113]

[0114] The similar triangles formed by the camera focal length f and the distance d between the object and the camera yield the following:

[0115]

[0116] Where f is in meters, and α and β are in pixels per meter. Also let:

[0117]

[0118] Combining the two equations and writing them in matrix form, we get:

[0119]

[0120] The intermediate expression is called the camera's intrinsic parameter matrix K. The extrinsic parameters are described by the rotation matrix R and the translation vector t.

[0121] With the RGB-D placed on the AGV, the distance d between the RGB-D and the small device, as well as the pose descriptions of depthcamera_link and base_link, can be obtained.

[0122] The quaternion can be converted to the rotation matrix R using the following formula:

[0123]

[0124] In step seven, the dual-robotic arm grasping collaborative algorithm is designed as follows:

[0125] By employing device type recognition and pose detection based on a fusion of deep learning and image processing, the coordinates of various devices in the robotic arm coordinate system have been obtained. Further, a dual-robotic arm collaborative grasping algorithm is needed to grasp various types of small devices. To achieve real-time and accurate collaborative grasping by the two robotic arms, the following process is defined:

[0126] First, the forward and inverse kinematics models of the six-axis serial robotic arms used in the system were derived and modeled. Cooperative motion rules for the two robotic arms and collision prevention rules based on the oriented containment box algorithm were designed. First, an onboard industrial camera captures images of the components placed on the worktable. These images are then uploaded to the AI ​​computing unit. A device category and contour detection method based on deep learning and machine vision is used to detect the device images, obtaining the device category and pose pixel coordinates. The presence of a device in the image is then used to determine whether the program terminates. Further, using the industrial camera calibration results, the pixel coordinates of the small components in the image are converted into gripping coordinates for the two robotic arms. The dual-robotic arm geometric inverse kinematics calculation system calculates the rotation angles of each joint of the two robotic arms based on the gripping point coordinates of the small component and the real-time vehicle speed of the component. Finally, this data is transmitted to the dual-robotic arm drive module via the CAN bus to control the two robotic arms to collaboratively grasp the small component.

[0127] Step three yields the coordinate relationship between base_link and imu_link, and step four yields the real-time moving speed of the AGV, denoted as V. AGV实时 And V arm =V depthcamera =V AGV实时 Step six yields the distance *d* between the RGB-D sensor and the small device, the pose descriptions of the base_link and depthcamera_link, and the contour and pose of the small device. The time *t* at which the dual robotic arms begin to move is then calculated.

[0128] Step 8: Determine if all required components in the order have been successfully picked up. If not, the AGV restarts from Step 6. If all components have been successfully picked up, the AGV delivers the components to the receiving point, then returns to its default starting point and prepares for the next task.

Claims

1. A method for using an omnidirectional AGV based on the collaborative grasping of small components by dual robotic arms, characterized in that, Includes the following steps; Step 1: Submit the required order via the touchscreen display and the human-computer interaction page; Step 2: The system automatically matches the pre-built map stored locally based on the order; Step 3: Obtain the current attitude information of the omnidirectional AGV and use the AMCL algorithm for real-time positioning; Step 4: Use Mecanum wheels, DC brushless motors, encoders, and compatible drive circuits to complete the omnidirectional movement of the AGV; Step 5: Perform global path planning using Dijkstra's algorithm, and then use the TEB algorithm to plan a local path from the current location to the local target point when local obstacles are detected. Step 6: The distance between the small device and the AGV is obtained in real time by RGB-D placed on the AGV. The omnidirectional AGV is navigated to the location of the device to be grasped. The small device is identified by an on-board industrial camera using the Yolov5 algorithm based on an improved attention mechanism and multi-scale features. The pose of the small device is estimated by the Canny contour and pose acquisition algorithm. Step 7: Communicate between the dual robotic arms and the omnidirectional AGV via the CAN bus. Use a time series prediction algorithm combined with the posture of the small device to predict its position and send the position information to the AI ​​control unit. Use the orientation bounding box dual robotic arm cooperative grasping algorithm to coordinate the dual robotic arms to grasp the small device in real time, while ensuring that the dual robotic arms can cooperate efficiently to complete the specified task without collision in actual operation. Step 8: Determine if all required components in the order have been picked up. If not, the AGV will start from Step 6 again. If all components have been picked up, the AGV will deliver the components to the receiving point, return to the default starting point, and prepare for the next task. In step two, RGB-D and LiDAR dual odometers are fused and RTAB is used to construct a 3D dense map. The constructed map is stored in the AI ​​computing unit. After the order is placed, the map is adapted according to the types of each device required in the order. The method for matching maps is as follows: The required device types can be determined from the order. The RTAB algorithm is used to build a map. For each map, a sequence is created to store all device types that exist in the map. When matching maps according to the device types required by the order, the number of required device types is first calculated. Only maps where the total number of device types in the sequence is greater than or equal to the number of required device types will be traversed by type. In step four, the rotation direction of the four brushless DC motors is determined by force analysis to achieve omnidirectional movement of the AGV. The serial PID algorithm is used to adjust the PWM value through the AI ​​control unit to control the speed of the Mecanum wheel. The AI ​​control unit uses the quadruple frequency technology to perform inverse kinematics on the four 90° AB phase encoders to obtain the current speed of the AGV, and then transmits it to the AI ​​computing unit through the CAN bus. By placing an RGB-D sensor and two robotic arms on an AGV, the following can be obtained: ； The real-time moving speed of the AGV is denoted as Let the moving speed of the robotic arm and the moving speed of the RGB-D be denoted as . and ; In step five, the 3D dense map established in step two is compressed and filtered to remove ground information using CostMap, and a raster map is generated through 2D projection. An Obstacles layer is added to maintain the obstacle information obtained by fusing the LiDAR and RGB-D scans. On the 2D raster map, Dijkstra's algorithm is used for global path planning to find a feasible shortest path from the starting point to the target point. The global path consists of discrete path points, considering only the static obstacle information during map building. The TEB algorithm is used to complete local path planning for obstacle avoidance. When the omnidirectional AGV gets stuck in navigation, it rotates in place to forcibly clear the residual obstacles in the cost map, performs recovery behavior, and gets out of trouble.

2. The method of using an omnidirectional AGV based on dual robotic arms collaboratively grasping small components according to claim 1, characterized in that, In step one, orders are placed by opening the human-computer interaction page in the AI ​​computing unit via a touch screen.

3. The method of using an omnidirectional AGV based on dual robotic arms collaboratively grasping small components according to claim 1, characterized in that, In step three, the MPU9250 module is used to obtain the raw data of the AGV's current acceleration and angular velocity. The raw data is processed in the AI ​​control unit to obtain the AGV's current pose information, which is then transmitted to the AI ​​computing unit via the CAN bus. For the acceleration data and gyroscope data acquired by the MPU9250, quaternions are used to describe the relationship between the coordinate systems: First, the FISR algorithm is used to solve for the reciprocal of the modulus: Multiplying the three acceleration vectors of the IMU by the reciprocal of the magnitude yields the three-dimensional unit vector. The gravity components are estimated based on the current quaternion attitude values ​​and compared with the gravity components actually measured by the accelerometer, thereby correcting the quadcopter attitude. The error between the estimated and measured directions of gravity is calculated using the sum of cross-products: The gravity error calculated above is proportionally calculated, with a proportionality coefficient of Kp. The result of the calculation is accumulated into the gyroscope data to correct the gyroscope data. Through the above calculations, the gyroscope data corrected by the accelerometer is obtained. The corrected gyroscope data is then integrated into a quaternion. Finally, the quaternion obtained from the above calculations is normalized to obtain a new quaternion after rotation.

4. The method of using an omnidirectional AGV based on dual robotic arms collaboratively grasping small components according to claim 1, characterized in that, In step six, the distance between the small device and the AGV is first obtained in real time by using an RGB-D sensor placed on the AGV. Improved Yolov5 network: Introduce ShuffleAttention mechanism, improve the mining of small target features by adding a new 104*104 feature map, adopt multi-scale feedback to introduce global context information to improve the recognition ability of small targets in the image, and adopt CIoU as a new target box regression loss function. Training weight file: To obtain the small device detection model, the created small device dataset is put into the improved Yolov5 network for training. The learning environment is configured and the number of training categories and label names are modified. After training, the small device detection model is obtained. Based on Canny device contour and pose acquisition: The device contour and centroid are obtained, and the pose of the small device in the robotic arm coordinate system is acquired. Based on the deep learning detection results, image cropping, Gaussian blurring, morphological closing operation, edge detection, and IoU calculation are performed to detect the small device contour and centroid. Then, its pose is calculated, and finally, the projected distance between the two circle centers on the plane is calculated. The angle between the device and the z-axis is The height difference between the two centers is ; The distance between the small device and the RGB-D sensor is calculated using similar triangles from the RGB-D sensor. First, it is assumed that the width of the detected object is... The distance between the detected object and the camera is recorded as... The pixel width of the object in the photo is To obtain the camera focal length formula: The distance between the moving object and the camera is calculated by applying similar triangles. : The pixel coordinate system is defined as follows: the origin is located at the top left corner of the image. , Axial to the right and The axes are parallel. Axial downward and The axes are parallel, and the pixel coordinates are in On-axis scaling times, in On-axis scaling The translation distance is times, and the translation distance is... The relationship between pixel coordinates and image physical coordinates is expressed as: And due to the camera focal length Distance between the object and the camera The similar triangles formed are: in The unit is meters. and The unit is pixels per meter, and let: Combining the two equations and writing them in matrix form, we get: The intermediate expression is called the camera's intrinsic parameter matrix. The extrinsic parameters are obtained through the rotation matrix. Translation vector describe; The RGB-D sensor is placed on the AGV, and the distance between the RGB-D sensor and the small device is obtained. And the pose descriptions of depthcamera_link and base_link.

5. The method of using an omnidirectional AGV based on dual robotic arms collaboratively grasping small components according to claim 4, characterized in that, In step seven, the dual-robotic arm grasping collaborative algorithm is specifically as follows: Through step six, which uses deep learning and image processing to identify device types and detect poses, the coordinates of various devices in the robotic arm coordinate system have been obtained. The dual robotic arm collaborative grasping algorithm is then used to grasp multiple types of small devices. First, the forward and inverse kinematics models of the six-axis serial robotic arm are used; the specific rules for the cooperative motion of the two robotic arms are as follows: An industrial camera captures images of devices placed on a workbench, which are then uploaded to an AI computing unit. Based on deep learning and machine vision, device category and contour detection methods are used to detect the device images, thereby obtaining the device category and pose pixel coordinates. The presence of a device in the image is determined to control whether the program ends. Using the calibration results of the industrial camera, the pixel coordinates of the small devices in the image are converted into the gripping coordinates of the dual robotic arms. The dual robotic arm geometric inverse kinematics solution system calculates the rotation angles of each joint of the dual robotic arms based on the gripping point coordinates of the small device and the real-time speed of the small device. Finally, the rotation angles of each joint of the dual robotic arms are sent to the dual robotic arm drive module via the CAN bus to control the dual robotic arms to collaboratively grasp the small device. Step three yields the coordinate relationship between base_link and imu_link, and step four yields the real-time moving speed of the AGV, denoted as [missing information]. ,and Step six yields the distance between the RGB-D sensor and the small device. The pose descriptions of the base_link and depthcamera_link, as well as the contours and poses of the small device, are used to calculate the time when the dual robotic arms begin to move. .

6. The method of using an omnidirectional AGV based on dual robotic arms collaboratively grasping small components according to claim 1, characterized in that, In step eight, the AI ​​computing unit determines whether all the required components in the order have been successfully grabbed. If not, the AGV starts executing from step six again. Once all items have been captured, the AGV delivers the device to the receiving point, then returns to its default starting point and prepares for the next task.

Images