Mobile mechanical arm grabbing control method and system based on machine vision

Abstract

The invention belongs to the technical field of mobile mechanical arm grabbing, and provides a mobile mechanical arm grabbing control method and system based on machine vision, and the method comprises the steps: collecting an image in the moving process of a mobile mechanical arm according to a preset grabbing path, recognizing a dynamic obstacle in the image through an RT-DETR algorithm, and generating a prediction track of the dynamic obstacle, collision detection is conducted on a preset grabbing path of the movable mechanical arm, an obstacle avoidance track switching instruction is generated by combining the current time point of the mechanical arm with the predicted collision time point of the dynamic obstacle, trajectory tracking is conducted on the obstacle avoidance track of the mechanical arm, a model reference self-adaptive control algorithm is adopted, a reference model is established, and obstacle avoidance is achieved. According to the method, mechanical arm vibration caused by obstacle avoidance track switching is effectively restrained, interference of vibration on grabbing stability is reduced, and the operation precision and reliability of the mechanical arm in a dynamic obstacle avoidance scene are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of mobile manipulator grabbing, and particularly relates to a mobile manipulator grabbing control method and system based on machine vision. BACKGROUND

[0002] In modern industrial production and human-machine cooperation and many other scenes, the application of mobile manipulators is becoming more and more widespread, and the demand for intelligence and safety of mobile manipulators is also increasingly prominent. The traditional mobile manipulator has many defects and deficiencies when facing complex dynamic environments.

[0003] When encountering situations that require obstacle avoidance, the obstacle avoidance strategy of the traditional manipulator has obvious shortcomings. On the one hand, in the assessment of obstacle avoidance urgency, there is a lack of scientific and reasonable assessment method, which cannot accurately judge the urgency of obstacle avoidance, resulting in slow response or overreaction when dealing with emergency situations. On the other hand, during the obstacle avoidance process, the manipulator is prone to vibration, and the traditional technology is difficult to effectively identify and suppress such vibration. For example, it is difficult to accurately determine the vibration source joint, so it is difficult to take targeted measures to reduce the interference of vibration on grabbing, making it difficult for the manipulator to stably complete the grabbing task after obstacle avoidance, seriously affecting the grabbing precision and operation reliability, and making it difficult to meet the requirements of industrial grabbing, human-machine cooperation and other scenes for high-precision and high-reliability operation of mobile manipulators.

[0004] Therefore, the application provides a mobile manipulator grabbing control method and system based on machine vision. SUMMARY

[0005] In order to make up for the deficiencies of the prior art and solve at least one technical problem proposed in the background art.

[0006] In a first aspect, the application provides a mobile manipulator grabbing control method based on machine vision, comprising: Collecting images during the movement of the mobile manipulator according to the preset grabbing path, identifying dynamic obstacles in the images through an RT-DETR algorithm, and generating a predicted trajectory of the dynamic obstacles, and performing collision detection with the preset grabbing path of the mobile manipulator to determine whether there is a collision risk for the mobile manipulator according to the preset grabbing path; If there is a collision risk, the avoidance urgency of the manipulator for the dynamic obstacle is evaluated in combination with the current time point of the manipulator and the predicted collision time point of the dynamic obstacle; If the avoidance urgency is high, an obstacle avoidance trajectory switching instruction is generated, the interference error of the vibration of the manipulator caused by obstacle avoidance on the grabbing of the manipulator is identified through trajectory tracking of the obstacle avoidance trajectory of the manipulator; If the disturbance error is strong, a model reference adaptive control algorithm is used to establish a reference model, and an adaptive compensation torque is determined through gradient descent method to reduce the disturbance error caused by the robot arm vibration due to obstacle avoidance.

[0007] In a second aspect, the application provides a mobile robot arm grasping control system based on machine vision, comprising: A dynamic obstacle identification module: collects images during movement of the mobile robot arm according to the preset grasping path, identifies dynamic obstacles in the images through the RT-DETR algorithm, and generates a predicted trajectory of the dynamic obstacles, and performs collision detection with the preset grasping path of the mobile robot arm to determine whether there is a collision risk for the mobile robot arm according to the preset grasping path; A collision risk assessment module: if there is a collision risk, the emergency of the robot arm for the dynamic obstacle is assessed in combination with the current time point of the robot arm and the predicted collision time point of the dynamic obstacle; An interference error analysis module: if the emergency of obstacle avoidance is high, an obstacle avoidance trajectory switching instruction is generated, the interference error of the robot arm vibration caused by obstacle avoidance on the robot arm grasping is identified through trajectory tracking of the robot arm obstacle avoidance trajectory; An adaptive compensation module: if the disturbance error is strong, a model reference adaptive control algorithm is used to establish a reference model, and an adaptive compensation torque is determined through gradient descent method to reduce the disturbance error caused by the robot arm vibration due to obstacle avoidance.

[0008] The beneficial effects of the application are as follows: 1、The application provides a mobile robot arm with a synchronous high-definition RGB and RGBD depth camera, which realizes reliable acquisition and identity tracking of obstacle vision and depth information through efficient and accurate dynamic obstacle identification and cross-frame identity binding capability of the RT-DETR algorithm; then the obstacle trajectory is generated through coordinate conversion, trajectory smoothing and LSTM model prediction, and the collision detection logic of point-to-point and point-to-sphere region is used for small and large volume obstacles respectively to accurately judge the collision risk of the preset grasping path; at the same time, the emergency of obstacle avoidance is evaluated by comparing the obstacle avoidance time and the threshold, which effectively improves the safety, reliability and adaptability of the mobile robot arm in complex dynamic environment, and can be flexibly adapted to different scene requirements such as industrial grasping and human-robot collaboration.

[0009] 2、The application collects vibration signals through an acceleration sensor and a joint encoder when the obstacle avoidance emergency is high, identifies the main vibration source joint through preprocessing and Fourier transform, and then calculates the instantaneous tracking error and error integral value through trajectory tracking to judge the vibration interference strength; when the interference error is strong, a reference model is established based on the model reference adaptive control algorithm combined with the joint motion parameters, the trajectory error is converted into a joint angle error, the adaptive compensation torque is determined through the gradient descent method and is superimposed on the control torque, a complete control link from vibration perception, interference evaluation to accurate compensation is formed, the mechanical arm vibration caused by the obstacle avoidance trajectory switching is effectively inhibited, the interference of the vibration on the grabbing stability is reduced, and the operation precision and reliability of the mechanical arm in the dynamic obstacle avoidance scene are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0010] The application will be further described below with reference to the drawings.

[0011] Figure 1 is a step flow chart of a mobile manipulator grabbing control method based on machine vision according to an embodiment of the application; Figure 2 is a system block diagram of a mobile manipulator grabbing control system based on machine vision according to an embodiment of the application. DETAILED DESCRIPTION

[0012] In order to make the technical means, creative features, purposes and effects of the application easy to understand, the application will be further described below in combination with specific embodiments.

[0013] Embodiment 1 Please refer to Figure 1 The mobile manipulator grabbing control method based on machine vision according to an embodiment of the application includes the following steps: Step 1: Collect images during the movement of the mobile manipulator according to the preset grabbing path, identify dynamic obstacles in the images through the RT-DETR algorithm, and generate the predicted trajectory of the dynamic obstacles, and perform collision detection with the preset grabbing path of the mobile manipulator to determine whether there is a collision risk when the mobile manipulator moves according to the preset grabbing path. It should be noted that the RT-DETR algorithm combines the efficient characteristics of Transformer and the powerful visual processing capability of convolutional neural network (CNN), and can efficiently identify multiple targets in the image in real time, such as pedestrians, vehicles, roadblocks, etc. A high-definition RGB camera is provided for the mobile manipulator to collect the visual features of the obstacles, which facilitates the identification of the RT-DETR algorithm, and an RGBD depth camera is also provided to supplement the depth information of the obstacles for subsequent three-dimensional trajectory calculation. The camera needs to be fixed at the end of the manipulator or at a key viewing angle in the work area to ensure that there is no visual blind area. The RGB image data collected by the RGB camera and the depth data collected by the RGBD depth camera are synchronized by a hardware clock, avoiding mismatch between the visual position and the depth position of the obstacle due to data delay, which affects the accuracy of subsequent trajectory coordinates; Pre-set category dynamic obstacle: According to the scene requirements (such as industrial grabbing, human-robot collaboration), the RT-DETR algorithm is used to pre-set the categories of dynamic obstacles to be identified (such as "moving workpiece", "human arm", "material on the conveyor belt"), avoiding irrelevant targets (such as static equipment shell) from interfering with trajectory analysis; Each frame of obstacle positioning and cross-frame identity binding is completed by the RT-DETR algorithm, specifically: Under the condition that the frame rate is the same, the RGB image frames collected by the RGB camera during the movement of the mobile robot arm along the preset grabbing path are extracted and integrated into a sequence of moving image frames in chronological order; Each frame of obstacle positioning: the real-time collected RGB image frame is used as the input of the RT-DETR algorithm, which quickly scans the image through the attention mechanism of the Transformer architecture and identifies all dynamic obstacles that meet the pre-set category (static targets are filtered out); for each dynamic obstacle, the key information is output, including: a two-dimensional bounding box (determining the position of the obstacle in the image), a category label (such as "moving workpiece"), a confidence (judging the reliability of the identification result), and a temporary ID of the target (used to temporarily store the identity of the obstacle in this frame); Cross-frame identity binding: the RT-DETR algorithm compares the 3 features of the current frame obstacle and the obstacle identified in the previous frame in the sequence of moving image frames: the similarity of the bounding box position (whether the distance between the center coordinates of the obstacles in the two frames is less than a pre-set threshold), the similarity of the appearance features (whether the visual features such as color and texture of the obstacles match, the cosine similarity of the feature vectors extracted by the model is calculated), and the consistency of the motion trend (combining the moving direction of the obstacle in the previous frame, judging whether the target in the current frame conforms to the trend); for the matched obstacle, a unique global ID is assigned to ensure that the identity of the same dynamic obstacle in consecutive frames is always consistent; if a new dynamic obstacle appears, a new global ID is assigned; if the obstacle disappears, it is marked as "has left" and its historical data is preserved; The conversion relationship between pixel coordinates and actual physical coordinates is established, and the two-dimensional bounding box center coordinates output by the RT-DETR algorithm are converted into three-dimensional coordinates in the robot arm world coordinate system; According to the global ID, the three-dimensional coordinates of each dynamic obstacle are integrated into a dynamic obstacle coordinate sequence at equal time intervals, where the time interval is consistent with the image acquisition frame rate; The sliding window smoothing method is used for the dynamic obstacle coordinate sequence to fill in possible inter-frame data gaps, and finally the motion trajectory curve of the dynamic obstacle is generated. For each dynamic obstacle, the generated motion trajectory curve of the dynamic obstacle is input into the LSTM time sequence model as a historical trajectory, the model learns the position change rule, sets a prediction period, and generates a predicted trajectory of the dynamic obstacle in the prediction period; The predicted trajectory of the dynamic obstacle in the prediction period is subjected to collision detection with the preset grabbing path of the mobile manipulator, and specifically: Based on the two-dimensional bounding box and depth data of the dynamic obstacle output by the RT-DETR, the volume of the dynamic obstacle is determined, and the volume limit value of the dynamic obstacle is set by a person skilled in the art according to the characteristics of the obstacle and historical experience. If the volume of the dynamic obstacle is greater than or equal to the volume limit value, the corresponding dynamic obstacle is recorded as a large-volume obstacle; if the volume of the dynamic obstacle is less than the volume limit value, the corresponding dynamic obstacle is recorded as a small-volume obstacle; Based on the small-volume obstacle, the small-volume obstacle and the end of the mobile manipulator are simplified as spatial points, the prediction period is divided into a plurality of prediction time points according to equal time intervals, the time interval is consistent with the image acquisition frame rate, the coordinates of the end of the manipulator at the prediction time points are extracted from the preset grabbing path, the center coordinates at the same prediction time points are extracted from the predicted trajectory of the small-volume obstacle, and the Euclidean distance between the end of the manipulator and the obstacle at the prediction time points is calculated by substituting the Euclidean distance formula, and is recorded as a small-volume distance safety value at the prediction time point. If the small-volume distance safety value at the prediction time point is less than or equal to the small-volume distance safety value threshold, it indicates that there is a collision risk at the prediction time point, and the corresponding prediction time point is recorded as a predicted collision time point. If the small-volume distance safety value at the prediction time point is greater than the small-volume distance safety value threshold, it indicates that there is no collision risk at the prediction time point. Based on the large-volume obstacle, only detecting the center point of the dynamic obstacle will miss the risk of the end of the manipulator touching the edge of the dynamic obstacle. The large-volume obstacle is simplified as a spherical region, the center coordinates of the large-volume obstacle at the prediction time point are taken as the center of the sphere, the radius of the sphere is calculated through the diagonal length of the two-dimensional bounding box of the dynamic obstacle, the Euclidean distance from the end point of the manipulator to the center of the sphere is calculated, and the minimum distance from the end point of the manipulator to the large-volume obstacle at the prediction time point is obtained by subtracting the Euclidean distance from the end point of the manipulator to the center of the sphere from the radius of the sphere, and is recorded as a large-volume distance safety value at the prediction time point. If the large-volume distance safety value at the prediction time point is less than or equal to the large-volume distance safety value threshold, it indicates that there is a collision risk at the prediction time point, and the corresponding prediction time point is recorded as a predicted collision time point. If the large-volume distance safety value at the prediction time point is greater than the large-volume distance safety value threshold, it indicates that there is no collision risk at the prediction time point. It should be noted that the small volume distance safety value threshold and the large volume distance safety value threshold are both greater than 0, and when the obtained small volume distance safety value is less than 0, it indicates that the robot arm has collided with the dynamic obstacle, and the robot arm operation is stopped urgently; Step two: if there is a collision risk, the collision avoidance urgency of the robot arm for the dynamic obstacle is evaluated in combination with the current time point of the robot arm and the predicted collision time point of the dynamic obstacle; Based on any dynamic obstacle, all predicted collision time points of the dynamic obstacle are extracted and integrated in time sequence as a predicted collision time sequence, and the first predicted collision time point in the predicted collision time sequence is subtracted from the current time point of the robot arm to obtain the collision avoidance time of the dynamic obstacle; The collision avoidance time of the dynamic obstacle is compared with the collision avoidance time threshold value: If the collision avoidance time of the dynamic obstacle is greater than or equal to the collision avoidance time threshold value, the collision avoidance urgency of the robot arm for the dynamic obstacle is high; If the collision avoidance time of the dynamic obstacle is less than the collision avoidance time threshold value, the collision avoidance urgency of the robot arm for the dynamic obstacle is low; The technical scheme of the embodiment is: collecting images in the movement process of the mobile robot arm according to the preset grabbing path, identifying the dynamic obstacle in the images through the RT-DETR algorithm, and generating a predicted trajectory of the dynamic obstacle; collision detection is performed on the predicted trajectory of the dynamic obstacle and the preset grabbing path of the mobile robot arm to determine whether there is a collision risk for the mobile robot arm according to the preset grabbing path; if there is a collision risk, the collision avoidance urgency of the robot arm for the dynamic obstacle is evaluated in combination with the current time point of the robot arm and the predicted collision time point of the dynamic obstacle; the present application realizes reliable acquisition and identity tracking of obstacle vision and depth information by providing a mobile robot arm with a synchronous high-definition RGB and RGBD depth camera, combining the high-efficiency and accurate dynamic obstacle identification and cross-frame identity binding capability of the RT-DETR algorithm; then, through coordinate conversion, trajectory smoothing and LSTM model prediction, the obstacle trajectory is generated, and point-to-point and point-to-sphere collision detection logic are respectively adopted for large and small volume obstacles to accurately determine the collision risk of the preset grabbing path; at the same time, the collision avoidance urgency is evaluated by comparing the collision avoidance time with the threshold value, which effectively improves the safety, reliability and adaptability of the mobile robot arm in complex dynamic environments, and can flexibly adapt to different scene requirements such as industrial grabbing and human-machine cooperation.

[0014] Embodiment 2 Please refer to Figure 1 The mobile robot arm grabbing control method based on machine vision of the embodiment of the present application further includes the following steps: Step three: if the collision avoidance urgency is high, a collision avoidance trajectory switching instruction is generated, and the disturbance error of the robot arm grabbing caused by the vibration of the robot arm due to collision avoidance is identified through trajectory tracking of the collision avoidance trajectory of the robot arm. It should be noted that the mechanical arm switches from the preset grabbing path to the obstacle avoidance path, the end or joint has high-frequency vibration, which causes the trajectory tracking error to increase, affects the grabbing stability, and the trajectory tracking is a process in which the mechanical arm controller drives the joint motor to make the mechanical arm end follow the expected trajectory in real time; Taking the obstacle avoidance trajectory switching instruction generation as the time starting point, a preset trajectory switching analysis period is set, vibration signals in the obstacle avoidance process of the mechanical arm are extracted through signal processing based on the acceleration sensor and joint encoder data, wherein the vibration signals include the vibration signals of the joints of the mechanical arm and the vibration signals of the end of the mechanical arm; It should be noted that the trajectory switching analysis period refers to the time period when the mechanical arm switches from the preset grabbing path to the obstacle avoidance path due to the high urgency of obstacle avoidance, and the mechanical arm has not reached the stable operation of the obstacle avoidance path. In this time period, the vibration caused by the inertia of the mechanical arm during obstacle avoidance affects the grabbing stability of the mechanical arm when switching to the obstacle avoidance path; The vibration signals are preprocessed, including removing environmental noise by using low-pass filtering to retain the vibration signals of the mechanical arm itself; the preprocessed vibration signals are subjected to Fourier transform to convert the time domain signals into frequency domain signals, generate a vibration spectrum graph, record the frequency corresponding to the peak value in the vibration spectrum graph as the main vibration frequency, and read the phase value corresponding to the main vibration frequency in the vibration spectrum graph; Based on any joint of the mechanical arm, the phase value of the joint of the mechanical arm is subtracted from the phase value of the end of the mechanical arm to obtain a phase difference; If the main vibration frequency of the joint of the mechanical arm is equal to the main vibration frequency of the end of the mechanical arm, and the phase difference is greater than 0, the corresponding joint of the mechanical arm is recorded as a main vibration source joint; otherwise, the corresponding joint of the mechanical arm is recorded as a non-vibration source joint; It should be noted that the otherwise condition includes: the main vibration frequency of the joint of the mechanical arm is not equal to the main vibration frequency of the end of the mechanical arm, and the phase difference is greater than 0; the main vibration frequency of the joint of the mechanical arm is equal to the main vibration frequency of the end of the mechanical arm, and the phase difference is less than or equal to 0; the main vibration frequency of the joint of the mechanical arm is not equal to the main vibration frequency of the end of the mechanical arm, and the phase difference is less than or equal to 0; Based on the main vibration source joint, the trajectory of the mechanical arm is tracked, the trajectory switching analysis period is divided into a plurality of trajectory switching analysis time points at equal time intervals, the coordinates of the end of the mechanical arm are extracted at the trajectory switching analysis time points, and the instantaneous tracking error of the trajectory switching time point is calculated by the Euclidean distance formula and the theoretical coordinates of the obstacle avoidance trajectory; The instantaneous tracking errors of all trajectory switching time points are integrated to obtain an error integral value; In some embodiments, the error integral value is compared with an error integral threshold, specifically: If the error integral value is greater than or equal to the error integral threshold value, it indicates that the interference error of the robot arm vibration caused by obstacle avoidance on the robot arm grabbing is strong; If the error integral value is less than the error integral threshold value, it indicates that the interference error of the robot arm vibration caused by obstacle avoidance on the robot arm grabbing is weak; Step four: if the interference error is strong, a model reference adaptive control algorithm is used to establish a reference model, and an adaptive compensation torque is determined by gradient descent method to reduce the interference error of the robot arm vibration caused by obstacle avoidance on the robot arm grabbing; The actual joint angle of the main vibration source joint is obtained through the joint encoder, the actual joint angular velocity of the main vibration source joint is calculated by differentiating the actual joint angle of the main vibration source joint, and the actual joint angular acceleration of the main vibration source joint is calculated by differentiating the actual joint angular velocity of the main vibration source joint; All joint angles of the main vibration source joint in the obstacle avoidance trajectory are integrated into a sequence of ideal joint angles of the main vibration source joint in time sequence, the damping coefficient matrix, the stiffness coefficient matrix and the gain matrix are set according to the rigidity of the main vibration source joint, and the reference model equation is established based on the simplified dynamics equation of the robot arm: , wherein, represents the ideal joint angle of the main vibration source joint at the tth trajectory switching time point, represents the ideal joint angular velocity of the main vibration source joint at the tth trajectory switching time point, represents the ideal joint angular acceleration of the main vibration source joint at the tth trajectory switching time point, represents the sequence of ideal joint angles of the main vibration source joint at the tth trajectory switching time point, and D represents the damping coefficient matrix, represents the stiffness coefficient matrix, represents the gain matrix; The instantaneous tracking error at the trajectory switching time point is converted into joint angle error through forward kinematics of the robot arm , specifically: , wherein, represents the instantaneous tracking error at the tth trajectory switching time point, is the pseudo-inverse of the Jacobian matrix at the tth trajectory switching time point; It should be noted that the pseudo-inverse of the Jacobian matrix describes the mapping relationship between the end position and the joint angle, which is updated in real time with the actual joint angle of the main vibration source joint, so as to ensure that the trajectory error of the robot arm end can be accurately converted into error indicators at the joint level; According to the joint angle error of the main vibration source joint at the trajectory switching time point, a target function is determined, an actual dynamics equation of the robot arm is established, an error between the actual dynamics equation of the robot arm and a reference model equation is minimized, an estimated disturbance torque of the actual dynamics equation of the robot arm is determined through a gradient descent method, an adaptive compensation torque is obtained by taking the opposite of the estimated disturbance torque, a control torque output by a basic controller is added to the adaptive compensation torque to obtain a final joint driving torque of the vibration source joint, and the final joint driving torque is sent to a joint motor driver, so as to reduce the interference error of the robot arm vibration caused by obstacle avoidance on the robot arm grabbing. It should be noted that the adaptive compensation torque is equal in size and opposite in direction to the estimated disturbance torque. The technical scheme of the embodiment is as follows: if the obstacle avoidance emergency is high, an obstacle avoidance trajectory switching instruction is generated, the interference error of the robot arm vibration caused by obstacle avoidance on the robot arm grabbing is identified through trajectory tracking of the robot arm obstacle avoidance trajectory; if the interference error is strong, a model reference adaptive control algorithm is used to establish a reference model, and an adaptive compensation torque is determined through a gradient descent method to reduce the interference error of the robot arm vibration caused by obstacle avoidance on the robot arm grabbing; in the case of high obstacle avoidance emergency, the acceleration sensor and the joint encoder are used to collect vibration signals, the main vibration source joint is identified through preprocessing and Fourier transform, the instantaneous tracking error and the error integral value are calculated through trajectory tracking to judge the vibration interference strength; when the interference error is strong, a reference model is established based on the model reference adaptive control algorithm and in combination with the joint motion parameters, the trajectory error is converted into a joint angle error, the adaptive compensation torque is determined through a gradient descent method and is superimposed on the control torque to form a complete control link from vibration perception, interference evaluation to accurate compensation, the robot arm vibration caused by obstacle avoidance trajectory switching is effectively inhibited, the interference of the vibration on the grabbing stability is reduced, and the operation precision and reliability of the robot arm in the dynamic obstacle avoidance scene are significantly improved.

[0015] Embodiment 3 Please refer to Figure 2 As shown in the figure, the mobile robot arm grabbing control system based on machine vision of the embodiment of the present application includes the following modules: A dynamic obstacle identification module: collects images in the process of movement of the mobile robot arm according to a preset grabbing path, identifies a dynamic obstacle in the images through an RT-DETR algorithm, and generates a predicted trajectory of the dynamic obstacle, and performs collision detection with the preset grabbing path of the mobile robot arm to determine whether there is a collision risk for the mobile robot arm according to the preset grabbing path; A collision risk evaluation module: if there is a collision risk, the obstacle avoidance emergency of the robot arm for the dynamic obstacle is evaluated in combination with the current time point of the robot arm and the predicted collision time point of the dynamic obstacle; An interference error analysis module: if the obstacle avoidance emergency is high, a trajectory switching instruction of obstacle avoidance is generated, the trajectory tracking of the obstacle avoidance trajectory of the robot arm is carried out, and the interference error of the robot arm vibration caused by the obstacle avoidance on the robot arm grabbing is identified; An adaptive compensation module: if the interference error is strong, a model reference adaptive control algorithm is used to establish a reference model, an adaptive compensation torque is determined through the gradient descent method, and the interference error of the robot arm vibration caused by the obstacle avoidance on the robot arm grabbing is reduced.

[0016] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A mobile robot arm grasping control method based on machine vision, characterized by: include: The system collects images of the mobile robot arm as it moves along the preset grasping path. It uses the RT-DETR algorithm to identify dynamic obstacles in the images and generate a predicted trajectory of the dynamic obstacles. It then performs collision detection with the preset grasping path of the mobile robot arm to determine whether there is a collision risk along the preset grasping path. If there is a collision risk, the robot arm's obstacle avoidance urgency is evaluated based on the robot arm's current time point and the predicted collision time point of the dynamic obstacle. If the obstacle avoidance urgency is high, an obstacle avoidance trajectory switching instruction is generated. By tracking the obstacle avoidance trajectory of the robot arm, the interference error of the robot arm vibration caused by obstacle avoidance on the robot arm grasping is identified; If the interference error is strong, the model reference adaptive control algorithm is used to establish a reference model, and the adaptive compensation torque is determined by the gradient descent method to reduce the interference error of the manipulator vibration caused by obstacle avoidance on the manipulator grasping.

2. A mobile robot arm grasping control method based on machine vision according to claim 1, characterized in that: The process of generating the predicted trajectory of a dynamic obstacle is as follows: The RGB image data collected by the RGB camera and the depth data collected by the RGBD depth camera are synchronized by the hardware clock. Dynamic obstacles are pre-classified and the RT-DETR algorithm is used to complete obstacle positioning in each frame and cross-frame identity binding to identify dynamic obstacles. Establish the conversion relationship between pixel coordinates and actual physical coordinates, and convert the center coordinates of the two-dimensional bounding box output by the RT-DETR algorithm into three-dimensional coordinates in the world coordinate system of the robot arm; Classify by global ID and integrate the 3D coordinates of each dynamic obstacle into a dynamic obstacle coordinate sequence at equal time intervals, where the time interval is consistent with the image acquisition frame rate; The sliding window smoothing method is used on the dynamic obstacle coordinate sequence to fill the possible data gaps between frames and finally generate the motion trajectory curve of the dynamic obstacle; For each dynamic obstacle, an LSTM time series model is used, and the generated motion trajectory curve of the dynamic obstacle is input into the model as the historical trajectory. The model learns the position change law, sets the prediction period, and generates the predicted trajectory of the dynamic obstacle within the prediction period.

3. A mobile robot arm grasping control method based on machine vision according to claim 1, characterized in that: The determination of whether there is a collision risk for the mobile manipulator along the preset grasping path includes respectively determining whether there is a collision risk for small obstacles and large obstacles along the preset grasping path. The identification process of small obstacles and large obstacles is as follows: Determine the volume of dynamic obstacles based on the 2D bounding box and depth data of dynamic obstacles output by RT-DETR; If the volume of a dynamic obstacle is greater than or equal to the volume limit, the corresponding dynamic obstacle will be recorded as a large-volume obstacle; if the volume of a dynamic obstacle is less than the volume limit, the corresponding dynamic obstacle will be recorded as a small-volume obstacle.

4. A mobile robot arm grasping control method based on machine vision according to claim 3, characterized in that: The process of determining whether there is a collision risk with a small obstacle along the preset grasping path of the mobile robot arm is as follows: Based on small-volume obstacles, the small-volume obstacle and the end of the mobile robot arm are simplified into spatial points. The prediction period is divided into several prediction time points at equal time intervals. The coordinates of the robot arm end at the prediction time point are extracted from the preset grasping path. The center coordinates at the same prediction time point are extracted from the predicted trajectory of the small-volume obstacle. Substituted into the Euclidean distance formula, the Euclidean distance between the end of the robot arm and the obstacle at the prediction time point is calculated and recorded as the small-volume distance safety value at the prediction time point. If the small volume distance safety value at the predicted time point is less than or equal to the small volume distance safety value threshold, it means that there is a collision risk at the predicted time point, and the corresponding predicted time point is recorded as the predicted collision time point; otherwise, it means that there is no collision risk at the predicted time point.

5. A mobile robot arm grasping control method based on machine vision according to claim 4, characterized in that: The process of determining whether there is a collision risk with a large obstacle along the preset grasping path of the mobile robot arm is as follows: Based on large-volume obstacles, the large-volume obstacles are simplified into spherical areas. The center coordinates of the large-volume obstacles at the predicted time point are used as the center of the sphere. The radius of the sphere is calculated by the diagonal length of the dynamic obstacle's two-dimensional bounding box. The Euclidean distance from the end point of the manipulator to the center of the sphere is calculated. The Euclidean distance from the end point of the manipulator to the center of the sphere is subtracted from the radius of the sphere to obtain the minimum distance from the end point of the manipulator to the large-volume obstacle at the predicted time point. This is recorded as the large-volume distance safety value at the predicted time point. If the large volume distance safety value at the predicted time point is less than or equal to the large volume distance safety value threshold, it means that there is a collision risk at the predicted time point, and the corresponding predicted time point is recorded as the predicted collision time point; otherwise, it means that there is no collision risk at the predicted time point.

6. A mobile robot arm grasping control method based on machine vision according to claim 5, characterized in that: The process of evaluating the urgency of the robot arm's avoidance of dynamic obstacles is as follows: Based on any dynamic obstacle, all predicted collision time points of the dynamic obstacle are extracted and integrated into a predicted collision time sequence in chronological order. The first predicted collision time point in the predicted collision time sequence is extracted and subtracted from the current time point of the robot arm to obtain the obstacle avoidance time of the dynamic obstacle; If the obstacle avoidance time of the dynamic obstacle is greater than or equal to the obstacle avoidance time threshold, the obstacle avoidance urgency of the robot arm for the dynamic obstacle is high; otherwise, the obstacle avoidance urgency of the robot arm for the dynamic obstacle is low.

7. A mobile robot arm grasping control method based on machine vision according to claim 4, characterized in that: The process of identifying the interference error of the robot arm vibration caused by obstacle avoidance on the robot arm grasping is as follows: Starting from the time when the obstacle avoidance trajectory switching instruction is generated, a trajectory switching analysis period is preset. Based on the acceleration sensor and joint encoder data, the vibration signal during the robot arm's obstacle avoidance process is extracted through signal processing. The vibration signal includes the vibration signal of the robot arm joint and the vibration signal of the robot arm end. By analyzing and processing the vibration signal, the main vibration source joint is determined; Based on the main vibration source joints, the robot arm's obstacle avoidance trajectory is tracked. The trajectory switching analysis period is divided into several trajectory switching analysis time points at equal time intervals. The coordinates of the robot arm end are extracted at the trajectory switching analysis time points and calculated with the theoretical coordinates of the obstacle avoidance trajectory using the Euclidean distance formula to obtain the instantaneous tracking error at the trajectory switching time points. Integrate the instantaneous tracking errors at all trajectory switching time points to obtain the error integral value; If the error integral value is greater than or equal to the error integral threshold, it means that the interference error of the robot arm vibration caused by obstacle avoidance on the robot arm grasping is strong; otherwise, it means that the interference error of the robot arm vibration caused by obstacle avoidance on the robot arm grasping is weak.

8. A mobile robot arm grasping control method based on machine vision according to claim 7, characterized in that: The process of determining the vibration source joint is as follows: Preprocess the vibration signal and perform Fourier transform to convert the time domain signal into a frequency domain signal to generate a vibration spectrum. The frequency corresponding to the peak in the vibration spectrum is recorded as the main vibration frequency, and the phase value corresponding to the main vibration frequency in the vibration spectrum is read. Based on any joint of the robotic arm, the phase value of the robotic arm joint is subtracted from the phase value of the robotic arm end to obtain the phase difference; If the main vibration frequency of the manipulator joint is equal to the main vibration frequency of the manipulator end, and the phase difference is greater than 0, the corresponding manipulator joint is recorded as the main vibration source joint.

9. A mobile robot arm grasping control method based on machine vision according to claim 8, characterized in that: The process of determining the adaptive compensation torque by the gradient descent method is as follows: The actual joint angle of the main vibration source joint is obtained through the joint encoder, the actual joint angle of the main vibration source joint is differentiated to obtain the actual joint angular velocity of the main vibration source joint, and the actual joint angular acceleration of the main vibration source joint is differentiated to obtain the actual joint angular velocity of the main vibration source joint; All joint angles of the main vibration source joints in the obstacle avoidance trajectory are integrated into the ideal joint angle sequence of the main vibration source joints in chronological order. According to the stiffness of the main vibration source joints, the damping coefficient matrix, stiffness coefficient matrix, and gain matrix are set. Based on the simplified dynamic equation of the manipulator, the reference model equation is established. The instantaneous tracking error at the trajectory switching time point is converted into joint angle error through the forward solution of the robot arm kinematics; According to the joint angle error of the main vibration source joint at the trajectory switching time point, the objective function is determined, and the actual dynamic equation of the robot arm is established. The error between the actual dynamic equation of the robot arm and the reference model equation is minimized. The estimated interference torque of the actual dynamic equation of the robot arm is determined by the gradient descent method, and the inverse of the estimated interference torque is taken to obtain the adaptive compensation torque.

10. A mobile robot arm grasping control system based on machine vision, characterized in that: The system is used to perform the method according to any one of claims 1 to 9, and the system comprises: Dynamic obstacle recognition module: This module collects images of the mobile robot arm as it moves along the preset grasping path, identifies dynamic obstacles in the images using the RT-DETR algorithm, and generates a predicted trajectory of the dynamic obstacles. This module then performs collision detection with the preset grasping path of the mobile robot arm to determine whether there is a collision risk along the preset grasping path. Collision risk assessment module: If there is a collision risk, the robot arm's current time point and the predicted collision time point of the dynamic obstacle are combined to evaluate the robot arm's obstacle avoidance urgency; Interference error analysis module: If the obstacle avoidance urgency is high, an obstacle avoidance trajectory switching instruction is generated. By tracking the robot arm's obstacle avoidance trajectory, the interference error caused by the robot arm's vibration caused by obstacle avoidance on the robot arm's grasping is identified; Adaptive compensation module: If the interference error is strong, the model reference adaptive control algorithm is used to establish a reference model and determine the adaptive compensation torque through the gradient descent method to reduce the interference error of the robot arm vibration caused by obstacle avoidance on the robot arm grasping.

Images