Multi-degree-of-freedom mechanical arm obstacle avoidance control method based on machine vision

Abstract

The invention relates to the technical field of robot control, in particular to a multi-degree-of-freedom mechanical arm obstacle avoidance control method based on machine vision, which comprises the following steps: acquiring 360-degree panoramic image data around a mechanical arm through a multi-camera system, and combining two-dimensional and three-dimensional information acquired by an RGB camera and a depth camera to construct an environment model. A pre-trained convolutional neural network model is used for identifying fixed and moving obstacles in an image, a Kalman filter is combined to predict the motion trail of the moving obstacle, an improved fast expansion random tree algorithm is used for path planning, it is ensured that the efficient and accurate obstacle avoidance capacity of the mechanical arm is kept in a dynamic environment, and the obstacle avoidance efficiency of the mechanical arm is improved. The motion state of the mechanical arm is controlled and monitored through real-time feedback, and deviation caused by inertia or errors is avoided. The dynamic adaptive capacity and the safe operation performance of the mechanical arm in the complex environment can be improved, and the mechanical arm is widely applied to robots in the fields of industry, medical treatment and the like.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a multi-degree-of-freedom robot arm obstacle avoidance control method based on machine vision. Background Art

[0002] With the rapid development of industrial automation, medical devices and intelligent service robots, the application demand of multi-degree-of-freedom robotic arms in complex environments is gradually increasing. However, robotic arms often need to work in dynamic and changing environments when performing tasks, which places higher requirements on their obstacle avoidance capabilities. Traditional robotic arm obstacle avoidance technologies usually rely on sensors such as ultrasonic, infrared sensors or lidar. Although these technologies can identify obstacles, they often show limitations in complex environments. For example, for obstacles with irregular shapes or fast movements, the accuracy and reaction speed of these sensors are difficult to meet actual needs. In addition, path planning in static environments is relatively simple, while in complex dynamic environments, robotic arms need to have the capabilities of real-time perception, path correction and dynamic obstacle avoidance to ensure the safe execution of tasks.

[0003] Existing obstacle avoidance control technologies generally face technical difficulties when facing complex environments. First, traditional sensors obtain limited environmental information and cannot capture the dynamic changes of obstacles in the environment in real time, resulting in delayed obstacle avoidance responses. Second, existing path planning algorithms may have planning delays and path discontinuity problems in complex scenarios and cannot effectively respond to real-time changes in the environment. Finally, the control system lacks robustness and accuracy, especially when the robot arm moves at high speed or encounters uncertain disturbances, which can easily lead to deviation accumulation or oscillation, resulting in system instability. Summary of the invention

[0004] Based on the above objectives, the present invention provides a multi-degree-of-freedom robotic arm obstacle avoidance control method based on machine vision.

[0005] The multi-degree-of-freedom robot arm obstacle avoidance control method based on machine vision includes the following steps:

[0006] S1, environmental perception: multiple cameras installed on the robotic arm collect 360-degree panoramic image data around the robotic arm. The cameras include RGB cameras and depth cameras, which are used to obtain two-dimensional images and three-dimensional depth data of the environment;

[0007] S2, image preprocessing: preprocessing the collected image data, including denoising, edge detection, image enhancement and segmentation, to ensure image clarity and extraction of effective data;

[0008] S3, obstacle recognition: Analyze possible obstacles in the image through a pre-trained convolutional neural network model. Obstacles include fixed obstacles (such as walls and tables) and mobile obstacles (such as people and vehicles);

[0009] S4, distance calculation and trajectory prediction: The distance between the obstacle and the robotic arm is obtained through the depth camera, and the motion trajectory of the moving obstacle is predicted in combination with the Kalman filter to determine the potential collision risk;

[0010] S5, obstacle avoidance path planning: Based on the motion trajectory prediction results, an improved fast-expanding random tree algorithm is used to plan an optimal obstacle avoidance path to ensure that the robot arm maintains efficiency and accuracy when avoiding obstacles, while avoiding delays caused by repeated path planning;

[0011] S6, real-time path correction: During the movement of the robot arm, the changes in the surrounding environment are monitored in real time. If a new obstacle is detected or the position of the obstacle changes, the preset path planning module is restarted to correct the path to ensure that the robot arm can quickly adjust the movement trajectory;

[0012] S7, dynamic feedback control: The motion state and posture of the robot arm are continuously monitored through the preset feedback control module to ensure that the robot arm accurately executes according to the planned path, while avoiding deviations caused by inertia or mechanical errors, ensuring smooth and safe operation of the robot arm.

[0013] Optionally, the S1 includes:

[0014] S11, camera layout design: optimize the layout of camera installation according to the structural characteristics of the robot arm and the workspace requirements;

[0015] S12, camera parameter configuration: optimizing the parameters of the RGB camera and the depth camera to adapt to different ambient lighting conditions;

[0016] S13, multi-camera data synchronization and fusion: In order to ensure the consistency of information when multiple cameras collect images, a time synchronization mechanism is used to calibrate the shooting time of multiple cameras through accurate timestamps to ensure that the images of the RGB camera and the depth camera are collected at the same time. The two-dimensional image and three-dimensional depth image of the same frame are fused through the data fusion algorithm to obtain the environment modeling data;

[0017] S14, panoramic field of view expansion: by adjusting the camera viewing angle and combining the collected data of multiple cameras, a panoramic stitching algorithm is used to synthesize the field of view coverage of multiple cameras into a 360-degree panoramic image;

[0018] S15, environmental depth modeling: convert the three-dimensional depth data collected by the depth camera into a three-dimensional point cloud, align the point cloud data from different perspectives through coordinate transformation, and generate a complete three-dimensional environmental model.

[0019] Optionally, the S2 specifically includes:

[0020] S21, image denoising processing: performing denoising processing on the collected RGB image and depth image respectively to reduce the interference of noise on image processing and improve the clarity of the image;

[0021] S22, edge detection: extract the edge of the denoised image using the Canny edge detection algorithm;

[0022] S23, image enhancement: using histogram equalization to enhance the image and improve the contrast of the image;

[0023] S24, Image segmentation: Use the GrabCut algorithm to segment the image and separate the robotic arm and possible obstacles in the foreground from the background.

[0024] Optionally, the S3 specifically includes:

[0025] S31, Convolutional neural network model training: Use an image dataset containing a large number of fixed obstacles and moving obstacles to pre-train the convolutional neural network model;

[0026] S32, fixed obstacle recognition: fixed obstacles in the image are recognized through a convolutional neural network model;

[0027] S33, moving obstacle recognition: the moving obstacles in the image are recognized by combining the convolutional neural network model with the frame difference method;

[0028] S34, obstacle category and position output: The final output of the convolutional neural network model includes the category and position of the obstacle, and the precise position of the obstacle is marked by a bounding box.

[0029] Optionally, the S4 specifically includes:

[0030] S41, distance calculation: obtain the distance between the obstacle and the robotic arm through the depth camera;

[0031] S42, mobile obstacle trajectory estimation: preliminarily estimating the motion trajectory of the mobile obstacle, extracting the speed and direction information of the obstacle by using inter-frame motion analysis technology, calculating the motion vector field through continuous frame images, and determining the preliminary motion trajectory of the mobile obstacle;

[0032] S43, Kalman filter trajectory prediction: Use Kalman filter to predict the motion trajectory of moving obstacles to improve the accuracy of obstacle avoidance response to dynamic obstacles;

[0033] S44, Collision risk assessment: Combined with the trajectory prediction results, the potential collision risk between the robot arm and the moving obstacle is assessed.

[0034] Optionally, the S5 specifically includes:

[0035] S51, Obstacle Space Modeling: Based on the obstacle distance and motion trajectory collected by the depth camera, the obstacle model in the robot workspace is constructed;

[0036] S52, initialization of improved rapidly expanding random tree algorithm: after the obstacle space modeling is completed, the improved rapidly expanding random tree algorithm is initialized to plan the optimal obstacle avoidance path;

[0037] S53, path search and node expansion: starting from the current position of the robot, randomly generate sample points, select the existing nodes closest to the sample points in the search space, and connect the new nodes by expanding the tree;

[0038] S54, heuristic search improvement: based on the traditional fast expansion random tree algorithm, a heuristic search is introduced, which uses a heuristic function to estimate the distance between the current node and the target node, and gives priority to expanding nodes that are closer to the target;

[0039] S55, path optimization: Based on the initially planned obstacle avoidance path, the path is adjusted through a local path optimization method to eliminate redundant nodes or unnecessary turning points.

[0040] Optionally, the S6 specifically includes:

[0041] S61, environmental change detection: During the movement of the robot arm, the depth camera and RGB camera are used to monitor the changes in the surrounding environment in real time;

[0042] S62, obstacle update: when a new obstacle is detected or the position of an existing obstacle changes, the spatial position and motion state information of the obstacle is updated, and the motion trajectory of the moving obstacle is corrected using a Kalman filter;

[0043] S63, path re-planning trigger: after the environment changes or the obstacle position is updated, the preset path re-planning module is automatically triggered, and the improved fast expansion random tree algorithm is called to search and plan the path again. The path planning process will consider the new obstacle position and motion state, and generate an optimal path to avoid the new obstacle;

[0044] S64, path correction execution: after the path replanning is completed, the current motion trajectory of the robot arm is dynamically updated;

[0045] S65, Continuous Path Correction Monitoring: When the robot arm executes a new path, it continuously monitors environmental changes and keeps real-time updates on obstacle positions and trajectories.

[0046] Optionally, the S7 includes:

[0047] S71, robot arm motion state monitoring: real-time monitoring of the robot arm motion state through a preset sensor network, including a position sensor, an acceleration sensor, and an angular velocity sensor;

[0048] S72, feedback data fusion: The data collected by multiple sensors are fused and processed, and the extended Kalman filter algorithm is used to filter the motion state of the robot arm to reduce the influence of sensor noise and improve feedback accuracy;

[0049] S73, error calculation: based on the deviation between the actual motion state and the planned path, calculate the position error and attitude error.

[0050] Optionally, the S7 further includes:

[0051] S74, real-time control adjustment: according to the calculated error value, the feedback control module adjusts the motion parameters of the robot arm in real time through the sliding mode control algorithm;

[0052] S75, Stability Analysis and Adjustment: Analyze the stability of the preset robot arm feedback control system through Lyapunov function;

[0053] S76, Posture and motion smoothness control: During real-time adjustment, the angular velocity and acceleration of each joint of the robot arm are controlled to ensure the smoothness of the robot arm's motion.

[0054] Beneficial effects of the present invention:

[0055] The present invention, through multi-camera arrangement and fusion technology, can achieve 360-degree blind-spot-free perception of the environment around the robot arm. RGB cameras and depth cameras are used to collect two-dimensional and three-dimensional data, and combined with the synchronous fusion of multi-source images, an accurate three-dimensional environmental model is constructed. This method can quickly adapt to environmental changes and promptly identify new obstacles or changes in the position of existing obstacles. The obstacle recognition model based on convolutional neural networks can effectively distinguish between fixed obstacles and mobile obstacles, and update in real time according to their dynamic trajectories. Through high-precision distance calculation and trajectory prediction of the Kalman filter, the future position of the mobile obstacle is accurately predicted, and the obstacle avoidance ability of the robot arm in complex environments is improved.

[0056] The present invention uses an improved rapidly expanding random tree algorithm and introduces a heuristic search method to improve the efficiency of path planning and avoid the planning delay problem that may occur in the traditional RRT algorithm. By continuously optimizing the path, the path length and the movement time of the robot arm are further reduced to ensure the efficiency of the robot arm in the obstacle avoidance process. In addition, the real-time path correction module continuously monitors environmental changes and can automatically trigger path re-planning when changes in the position of obstacles are detected, ensuring that the robot arm can respond quickly to environmental changes. The entire planning system can maintain the optimality and continuity of the path in highly dynamic work scenarios, thereby ensuring the flexibility and accuracy of the robot arm in complex working environments.

[0057] The present invention realizes real-time feedback monitoring of the motion state of the robot arm through sensor networks and extended Kalman filter technology, and can accurately obtain motion data such as position, velocity and acceleration. Combining the stability analysis of sliding mode control and Lyapunov function, the robustness and stability of the control system under complex working conditions are guaranteed, and the system instability caused by external disturbances or uncertainty of control input is avoided. The rapid convergence of errors is achieved through sliding mode control to ensure that the robot arm can stably execute the planned path. At the same time, PID control is used to smoothly adjust the movement of the robot arm to avoid jitter or error accumulation caused by rapid adjustment, which improves the operational stability and safety of the robot arm, and enables it to have excellent performance in scenarios with high precision requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 A schematic diagram of a method flow chart of an embodiment of the present invention;

[0060] Figure 2 Schematic diagram of S5 process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0062] It should be noted that the references to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments" and the like in the specification indicate that the embodiments described may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0063] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0064] like Figure 1-Figure 2 As shown, the multi-degree-of-freedom robot arm obstacle avoidance control method based on machine vision includes the following steps:

[0065] S1, environmental perception: multiple cameras installed on the robotic arm collect 360-degree panoramic image data around the robotic arm. The cameras include RGB cameras and depth cameras, which are used to obtain two-dimensional images and three-dimensional depth data of the environment;

[0066] S2, image preprocessing: preprocessing the collected image data, including denoising, edge detection, image enhancement and segmentation, to ensure image clarity and extraction of effective data;

[0067] S3, obstacle recognition: Analyze possible obstacles in the image through a pre-trained convolutional neural network model. Obstacles include fixed obstacles (such as walls and tables) and mobile obstacles (such as people and vehicles);

[0068] S4, distance calculation and trajectory prediction: The distance between the obstacle and the robotic arm is obtained through the depth camera, and the motion trajectory of the moving obstacle is predicted in combination with the Kalman filter to determine the potential collision risk;

[0069] S5, obstacle avoidance path planning: Based on the motion trajectory prediction results, an improved fast-expanding random tree algorithm is used to plan an optimal obstacle avoidance path to ensure that the robot arm maintains efficiency and accuracy when avoiding obstacles, while avoiding delays caused by repeated path planning;

[0070] S6, real-time path correction: During the movement of the robot arm, the changes in the surrounding environment are monitored in real time. If a new obstacle is detected or the position of the obstacle changes, the preset path planning module is restarted to correct the path to ensure that the robot arm can quickly adjust the movement trajectory;

[0071] S7, dynamic feedback control: The motion state and posture of the robot arm are continuously monitored through the preset feedback control module to ensure that the robot arm accurately executes according to the planned path, while avoiding deviations caused by inertia or mechanical errors, ensuring smooth and safe operation of the robot arm.

[0072] S1 includes:

[0073] S11, camera layout design: According to the structural characteristics of the robot arm and the requirements of the working space, the layout of the installed cameras is optimized. The RGB camera and the depth camera are evenly distributed at the joints and ends of the robot arm to ensure that the robot arm can collect 360-degree panoramic image data during movement. The camera layout avoids the formation of blind spots in the working area of ​​the robot arm to ensure that the surrounding environment information can be obtained in real time regardless of the posture of the robot arm.

[0074] S12, camera parameter configuration: optimize the parameters of the RGB camera and the depth camera to adapt to different ambient lighting conditions. The resolution, frame rate and exposure time of the RGB camera are automatically adjusted according to the ambient light intensity to ensure that a clear two-dimensional image is obtained. The depth camera optimizes the collection of three-dimensional depth information by adjusting the projection mode and parallax calculation accuracy to ensure accurate depth data is obtained under different object surfaces and distances.

[0075] S13, multi-camera data synchronization and fusion: In order to ensure the consistency of information when multiple cameras collect images, a time synchronization mechanism is used to calibrate the shooting time of multiple cameras through accurate timestamps to ensure that the images of the RGB camera and the depth camera are collected at the same time. The two-dimensional image and the three-dimensional depth image of the same frame are fused through the data fusion algorithm to obtain the environment modeling data. The fusion algorithm is expressed as:

[0076] I fused (x,y)=α·I RGB (x,y)+(1-α)·I Depth (x,y);

[0077] Among them, I fused (x, y) is the pixel value after fusion, α is the weight factor, which depends on the clarity of the RGB image and the accuracy of the depth image. RGB (x,y) and I Depth (x, y) are the pixel values ​​of the RGB image and the depth image respectively;

[0078] S14, panoramic field of view expansion: by adjusting the camera viewing angle and combining the collected data of multiple cameras, a panoramic stitching algorithm is used to synthesize the field of view coverage of multiple cameras into a 360-degree panoramic image. The stitching process needs to consider image distortion correction and smooth transition of the stitching boundary to ensure seamless integration of the panoramic image. The panoramic stitching algorithm can be completed by the RANSAC (random sampling consensus) algorithm in conjunction with the image feature point matching technology;

[0079] S15, environmental depth modeling: convert the 3D depth data collected by the depth camera into a 3D point cloud, align the point cloud data from different perspectives through coordinate transformation, and generate a complete 3D environmental model. The point cloud generation formula is expressed as:

[0080]

[0081] Among them, P (x, y, z) is the coordinate of the three-dimensional point, Z is the depth value, c x and c y is the image center, f x and f y is the focal length parameter of the camera;

[0082] Through reasonable camera layout, optimized parameter configuration and efficient data fusion algorithm, 360-degree all-round perception of the robot's surrounding environment is achieved. Through panoramic image stitching and 3D modeling technology, a high-precision environmental perception system is built to provide accurate environmental information for subsequent obstacle avoidance control.

[0083] S2 specifically includes:

[0084] S21, image denoising: denoising is performed on the collected RGB image and depth image respectively to reduce the interference of noise on image processing and improve the clarity of the image. The RGB image is denoised using a Gaussian filter and the depth image is denoised using a bilateral filter to ensure that edge information is retained while removing noise;

[0085] S22, edge detection: The Canny edge detection algorithm is used to extract the edges of the denoised image. First, the image is Gaussian smoothed, then the gradient strength and direction of the image are calculated, and then non-maximum suppression is used to ensure that the main edges are retained. Finally, the double threshold method is used to remove false edges to ensure that only the real edges related to obstacles are retained;

[0086] S23, Image Enhancement: Use histogram equalization to enhance the image, improve the image contrast, make the edges of objects in the image clearer, and adjust the image grayscale distribution to enhance the details of bright and dark parts in the image, so as to ensure that the features of obstacles are more prominent;

[0087] S24, Image segmentation: Use the GrabCut algorithm to segment the image and separate the robot arm and possible obstacles in the foreground from the background. The GrabCut algorithm classifies the pixels in the image into foreground and background by constructing a Gaussian mixture model, and iteratively optimizes the image using the maximum flow minimum segmentation algorithm to finally obtain accurate foreground and background segmentation results. The energy function of GrabCut is expressed as:

[0088] E(G)=∑ i∈P D(i,G)+∑ (i,j)∈N V(i,j,G);

[0089] Where D(i,G) is the similarity between pixel i and its label G, V(i,j,G) is the relationship between adjacent pixels, P is the pixel set, and N is the pixel neighborhood;

[0090] The collected RGB images and depth images are fully preprocessed, including denoising, edge detection, image enhancement and segmentation processing, to ensure high-quality image data input and effectively improve the accuracy of subsequent obstacle recognition and obstacle avoidance path planning.

[0091] S3 specifically includes:

[0092] S31, convolutional neural network model training: The convolutional neural network model is pre-trained using an image dataset containing a large number of fixed obstacles and moving obstacles. Through supervised learning, the convolutional neural network model can learn the characteristics of different obstacles, including shape, color, texture, etc., to ensure that obstacles in various environments can be accurately identified. The network structure includes multiple convolutional layers, pooling layers, and fully connected layers. The output is the category and position of the obstacle. The loss function uses the cross entropy loss function, which is expressed as:

[0093]

[0094] Among them, y i is the true category label, p i The category probability output by the model;

[0095] S32, fixed obstacle recognition: The convolutional neural network model is used to recognize fixed obstacles in the image. Fixed obstacles include walls, tables, machinery and equipment, etc. The positions and shapes of these obstacles are relatively stable. The model outputs the category of the obstacle and its position coordinates in the image based on the edge information, shape and color features in the image.

[0096] S33, moving obstacle recognition: the moving obstacles in the image are recognized by combining the convolutional neural network model with the frame difference method;

[0097] First, the pixel differences are calculated through continuous frame images to extract the moving object areas. Then, the convolutional neural network model classifies these areas and identifies moving obstacles such as pedestrians and vehicles. The calculation formula for the inter-frame difference is:

[0098] D(x,y)=|I t (x,y)-I t-1 (x,y)|;

[0099] Among them, D(x,y) is the difference between frames, I t (x,y) and I t-1 (x, y) are the pixel values ​​of the current frame and the previous frame respectively;

[0100] S34, obstacle category and position output: The final output of the convolutional neural network model includes the category and position of the obstacle. The precise position of the obstacle is marked by a bounding box. The output is in the form of the category label of the obstacle and its position information, such as the coordinates of the four corners (x1, y1, x2, y2), to ensure accurate positioning in the image.

[0101] The pre-trained convolutional neural network model can efficiently and accurately identify fixed and moving obstacles in images, including walls, tables, pedestrians, vehicles, etc., and provides a reliable data basis for subsequent distance calculation and obstacle avoidance path planning through the output of category and location information.

[0102] S4 specifically includes:

[0103] S41, distance calculation: The distance between the obstacle and the robotic arm is obtained through the depth camera, and the actual distance of the obstacle is determined by the pixel difference in the depth image using the parallax calculation method, which is expressed as:

[0104]

[0105] Where D(x,y) is the actual distance of the obstacle at the pixel point (x,y), f is the focal length of the camera, B is the baseline distance of the dual cameras, and d(x,y) is the disparity value of the pixel point;

[0106] The depth camera updates the depth information of each pixel in real time to generate a depth map of obstacles in the environment;

[0107] S42, mobile obstacle trajectory estimation: preliminarily estimate the motion trajectory of the mobile obstacle, extract the speed and direction information of the obstacle by using the inter-frame motion analysis technology, calculate the motion vector field through continuous frame images, and determine the preliminary motion trajectory of the mobile obstacle. The calculation formula of the motion vector field is expressed as:

[0108]

[0109] Among them, v(x,y) is the movement speed of the pixel point (x,y), I t (x,y) and I t -1(x,y) are the pixel values ​​of the current frame and the previous frame respectively, and Δt is the time difference between the two frames;

[0110] S43, Kalman filter trajectory prediction: Use the Kalman filter to predict the motion trajectory of the moving obstacle to improve the accuracy of the obstacle avoidance response to dynamic obstacles. The state update equation of the Kalman filter is:

[0111]

[0112] in, is the predicted state at the next moment, is the current state, A is the state transfer matrix, B is the control matrix, u k is the control input, w k is the process noise;

[0113] The filter updates the predicted state at each moment, dynamically adjusts it based on the measurement noise and uncertainty of the motion process, and finally obtains the accurate predicted trajectory of the obstacle;

[0114] S44, Collision risk assessment: Combined with the trajectory prediction results, the potential collision risk between the robot arm and the moving obstacle is assessed. According to the current motion state (position, speed, direction) of the robot arm and the predicted trajectory of the obstacle, the collision time and distance are calculated. The risk assessment formula is:

[0115]

[0116] Among them, T collision is the estimated collision time, D min is the minimum distance between the robot arm and the obstacle, v relative is the relative speed between the robot arm and the obstacle, if T collision If it falls below the set safety threshold, the obstacle avoidance mechanism is triggered;

[0117] Through distance calculation and trajectory prediction, the distance between the robot arm and the obstacle can be accurately calculated, and the motion trajectory of the moving obstacle can be predicted through the Kalman filter, and the potential collision risk can be evaluated in real time, providing accurate data support for the robot arm's obstacle avoidance decision.

[0118] S5 specifically includes:

[0119] S51, obstacle space modeling: Based on the obstacle distance and motion trajectory collected by the depth camera, an obstacle model in the robot workspace is constructed. The voxel grid method is used to divide the space into several three-dimensional small units (voxels). The occupancy state of each voxel is represented as 0 or 1, representing free space and obstacles respectively. This model can be used in subsequent path planning algorithms. The formula for obstacle modeling is:

[0120]

[0121] Among them, V(x,y,z) is the state value of the voxel in three-dimensional space, and x,y,z are the coordinates of the voxel in space;

[0122] S52, initialization of improved rapidly expanding random tree algorithm: after the obstacle space modeling is completed, the improved rapidly expanding random tree algorithm is initialized to plan the optimal obstacle avoidance path. The improved rapidly expanding random tree algorithm introduces heuristic search, and by combining the distance information between obstacles and targets, it preferentially expands the nodes facing the target, thereby accelerating the path planning process and reducing the number of iterations;

[0123] S53, path search and node expansion: Starting from the current position of the robot, randomly generate sample points, select the existing nodes closest to the sample points in the search space, connect the new nodes by expanding the tree, and optimize the node expansion according to the following path cost function:

[0124]

[0125] Among them, C(P) is the path cost, d i is the moving distance of the robot arm on path segment i, v i is the moving speed of the robot on this path segment. The algorithm ensures that the robot chooses the shortest and most efficient obstacle avoidance path by minimizing the path cost.

[0126] S54, heuristic search improvement: Based on the traditional fast expansion random tree algorithm, a heuristic search is introduced. The heuristic function is used to estimate the distance between the current node and the target node, and the nodes closer to the target are expanded first. The expression of the heuristic function is:

[0127] H(n)=α·d(n,goal)+(1-α)·d(n,obstacle);

[0128] Among them, H(n) is the heuristic value of node n, d(n,goal) is the distance from the node to the goal, d(n,obstacle) is the distance from the node to the nearest obstacle, and α is the heuristic weight coefficient, which adjusts the tendency of the search;

[0129] S55, path optimization: Based on the initially planned obstacle avoidance path, the path is adjusted through the local path optimization method to eliminate redundant nodes or unnecessary turning points, further reducing the path length and robot arm movement time. The path optimization strategy is to eliminate unnecessary nodes or readjust the path segments according to the continuity and smoothness of the path to ensure the efficiency and feasibility of the path;

[0130] The improved rapidly expanding random tree algorithm combines heuristic search and path optimization technology to plan an optimal obstacle avoidance path for the robotic arm in a complex environment. It not only improves the efficiency of path planning, but also ensures the accuracy and real-time performance of the robotic arm in the obstacle avoidance process, avoiding the delay problem caused by repeated path planning.

[0131] S6 specifically includes:

[0132] S61, environmental change detection: During the movement of the robot arm, the depth camera and RGB camera are used to monitor the changes in the surrounding environment in real time. By comparing the continuous frame image data, the inter-frame difference method is used to detect whether there are new obstacles or the position of existing obstacles has changed. The calculation formula of the inter-frame difference is expressed as:

[0133] ΔI(x,y)=|I t (x,y)-I t-1 (x,y)|;

[0134] Among them, ΔI(x,y) is the inter-frame difference of pixel point (x,y), I t (x, y) is the image data of the current frame, I t-1 (x, y) is the image data of the previous frame. When the difference exceeds the preset threshold, it is determined that the environment has changed significantly;

[0135] S62, obstacle update: when a new obstacle is detected or the position of an existing obstacle changes, the spatial position and motion state information of the obstacle is updated, and the motion trajectory of the moving obstacle is corrected using the Kalman filter to ensure the accuracy of the real-time position and predicted trajectory of the obstacle. The state update equation of the Kalman filter is:

[0136]

[0137] in, For the prediction of the obstacle position at the next moment, is the current obstacle position state, A is the state transfer matrix, B is the control matrix, u k is the control input, w k is the process noise;

[0138] S63, path re-planning trigger: after the environment changes or the obstacle position is updated, the preset path re-planning module is automatically triggered, and the improved fast expansion random tree algorithm is called to search and plan the path again. The path planning process will consider the new obstacle position and motion state, and generate an optimal path to avoid the new obstacle;

[0139] S64, path correction execution: after the path re-planning is completed, the current motion trajectory of the robot arm is dynamically updated, and the preset control system adjusts the joint angle and speed of the robot arm according to the new planned path to ensure that the robot arm can smoothly turn to the new path while avoiding sudden emergency stops or accelerations to ensure the smoothness of the path correction;

[0140] S65, continuous path correction monitoring: When the robot arm executes a new path, it continuously monitors environmental changes and keeps real-time updates of obstacle positions and trajectories. If new environmental changes are detected, the system will trigger path replanning again to form a closed-loop control mechanism to ensure the robot arm's obstacle avoidance ability in a dynamic environment;

[0141] Through real-time path correction, it is possible to monitor environmental changes in real time during the movement of the robotic arm, and automatically trigger the path correction mechanism when a new obstacle is detected or an existing obstacle is displaced. The improved rapidly expanding random tree algorithm is used to replan the path to ensure that the robotic arm can quickly adjust its motion trajectory and safely avoid obstacles. This closed-loop control mechanism ensures the dynamic obstacle avoidance capability of the robotic arm, enabling it to adapt to complex and changing working environments.

[0142] S7 includes:

[0143] S71, robot arm motion state monitoring: The motion state of the robot arm is monitored in real time through a preset sensor network, including position sensors, acceleration sensors and angular velocity sensors. The position information is obtained through an encoder. The angle and position data of each robot arm joint are collected regularly and fed back to the control system. The acceleration sensor is used to detect the linear acceleration of the robot arm, and the angular velocity sensor is used to monitor the rotational acceleration of the robot arm to ensure that the system can fully perceive the motion state of the robot arm. The motion state data is expressed by the following expression:

[0144]

[0145] Among them, q i is the angle of the i-th joint of the robot, is the angular velocity, is the angular acceleration;

[0146] S72, feedback data fusion: The data collected by multiple sensors are fused and processed, and the extended Kalman filter algorithm is used to filter the motion state of the robot arm to reduce the impact of sensor noise and improve feedback accuracy. The state update and measurement update equations of the extended Kalman filter are as follows:

[0147]

[0148] in, is the predicted value of the motion state, u k is the control input, w k and v k are process noise and measurement noise, h(·) is the measurement function;

[0149] S73, error calculation: Based on the deviation between the actual motion state and the planned path, calculate the position error and attitude error. The error calculation formula is as follows:

[0150] e pos =P desired -P actual ;

[0151] e att =R desired -R actual ;

[0152] Among them, e pos is the position error, P desired is the expected position on the planned path, P actual is the current actual position, e att is the attitude error, R desired is the expected posture matrix, R actual is the current actual posture matrix;

[0153] The robotic arm can monitor its own motion state in real time through sensor networks and data fusion technology, and compare the actual state with the planned path to ensure the accuracy of position and posture during movement, reduce errors caused by inertia, noise or other environmental factors, and lay the foundation for smooth and safe robotic arm operation.

[0154] The S7 also includes:

[0155] S74, real-time control adjustment: According to the calculated error value, the feedback control module adjusts the motion parameters of the robot arm in real time through the sliding mode control algorithm to ensure error convergence and maintain the motion stability of the robot arm. The control law of the sliding mode control is as follows:

[0156] u(t)=u eq (t)+u sw (t);

[0157] Where u(t) is the control input, u e q(t) is the equivalent control quantity, u s w(t) is the switching control quantity, and the sliding surface is defined as:

[0158]

[0159] Among them, s(t) is the sliding surface, e(t) is the position error, and c is the adjustment coefficient. The sliding mode control ensures that the robot can execute according to the planned path and quickly eliminate the deviation by dynamically adjusting the control input;

[0160] S75, Stability analysis and adjustment: The stability of the preset manipulator feedback control system is analyzed by Lyapunov function to ensure the robustness of the manipulator feedback control system. The Lyapunov function V(t) is defined as:

[0161]

[0162] According to the Lyapunov stability condition, its time derivative is guaranteed is negative to ensure the asymptotic stability of the system:

[0163]

[0164] When the above conditions are met, the control system of the robot arm is stable, avoiding oscillation or overshoot;

[0165] S76, attitude and motion smoothness control: In the real-time adjustment process, the angular velocity and acceleration of each joint of the robot arm are controlled to ensure the smoothness of the robot arm movement and avoid violent shaking or unstable operation caused by sudden acceleration or deceleration. The proportional-integral-differential (PID) control algorithm is used to adjust the angular velocity and acceleration. The control law is expressed as:

[0166]

[0167] Among them, K p , K i , K d are the gain parameters of the PID controller. By adjusting these parameters, the robot arm can be ensured to move smoothly and its posture can be adjusted properly.

[0168] Real-time feedback adjustment of the robotic arm is achieved through sliding mode control, Lyapunov stability analysis and PID control, ensuring that the robotic arm can accurately execute the planned path during movement, and avoiding unnecessary jitter or errors through smooth control, thereby improving the safety and stability of the robotic arm's operation.

[0169] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0170] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-degree-of-freedom robot arm obstacle avoidance control method based on machine vision, characterized in that: The following steps are involved: S1, environmental perception: multiple cameras installed on the robotic arm collect 360-degree panoramic image data around the robotic arm. The cameras include RGB cameras and depth cameras, which are used to obtain two-dimensional images and three-dimensional depth data of the environment; S2, image preprocessing: preprocessing the collected image data, including denoising, edge detection, image enhancement and segmentation; S3, obstacle recognition: obstacles in the image are analyzed through a pre-trained convolutional neural network model. Obstacles include fixed obstacles and mobile obstacles. S4, distance calculation and trajectory prediction: The distance between the obstacle and the robotic arm is obtained through the depth camera, and the motion trajectory of the moving obstacle is predicted in combination with the Kalman filter; S5, obstacle avoidance path planning: Based on the motion trajectory prediction results, an optimal obstacle avoidance path is planned using the improved rapidly expanding random tree algorithm; S6, real-time path correction: During the movement of the robot arm, the changes in the surrounding environment are monitored in real time. If a new obstacle is detected or the position of the obstacle changes, the preset path planning module is restarted to correct the path; S7, dynamic feedback control: The motion state and posture of the robot arm are continuously monitored through the preset feedback control module to ensure that the robot arm accurately executes according to the planned path while avoiding deviations caused by inertia or mechanical errors.

2. The multi-degree-of-freedom robot arm obstacle avoidance control method based on machine vision according to claim 1, characterized in that: The S1 includes: S11, camera layout design: optimize the layout of camera installation according to the structural characteristics of the robot arm and the workspace requirements; S12, camera parameter configuration: optimizing the parameters of the RGB camera and the depth camera to adapt to different ambient lighting conditions; S13, multi-camera data synchronization and fusion: adopt time synchronization mechanism, calibrate the shooting time of multiple cameras through timestamp, ensure that the images of RGB camera and depth camera are collected at the same time, and the 2D image and 3D depth image of the same frame are fused through data fusion algorithm to obtain environment modeling data; S14, panoramic field of view expansion: by adjusting the camera viewing angle and combining the collected data of multiple cameras, a panoramic stitching algorithm is used to synthesize the field of view coverage of multiple cameras into a 360-degree panoramic image; S15, environmental depth modeling: convert the three-dimensional depth data collected by the depth camera into a three-dimensional point cloud, align the point cloud data from different perspectives through coordinate transformation, and generate a three-dimensional environmental model.

3. The multi-degree-of-freedom robot arm obstacle avoidance control method based on machine vision according to claim 2, characterized in that: The S2 specifically includes: S21, image denoising processing: performing denoising processing on the collected RGB image and depth image respectively to reduce the interference of noise on image processing and improve the clarity of the image; S22, edge detection: extract the edge of the denoised image using the Canny edge detection algorithm; S23, image enhancement: using histogram equalization to enhance the image and improve the contrast of the image; S24, Image segmentation: Use the GrabCut algorithm to segment the image and separate the robotic arm and obstacles in the foreground from the background.

4. The multi-degree-of-freedom robot arm obstacle avoidance control method based on machine vision according to claim 3, characterized in that: The S3 specifically includes: S31, Convolutional neural network model training: Use image datasets containing fixed obstacles and moving obstacles to pre-train the convolutional neural network model; S32, fixed obstacle recognition: Identify fixed obstacles in the image through a convolutional neural network model; S33, moving obstacle recognition: the moving obstacles in the image are recognized by combining the convolutional neural network model with the frame difference method; S34, obstacle category and position output: The final output of the convolutional neural network model includes the category and position of the obstacle, and the position of the obstacle is marked by a bounding box.

5. The multi-degree-of-freedom robot arm obstacle avoidance control method based on machine vision according to claim 4, characterized in that: The S4 specifically includes: S41, distance calculation: obtain the distance between the obstacle and the robotic arm through the depth camera; S42, moving obstacle trajectory estimation: preliminarily estimating the moving obstacle's trajectory, and extracting the obstacle's speed and direction information using inter-frame motion analysis technology; S43, Kalman filter trajectory prediction: Use Kalman filter to predict the motion trajectory of moving obstacles; S44, Collision risk assessment: Combined with the trajectory prediction results, the potential collision risk between the robot arm and the moving obstacle is assessed.

6. The multi-degree-of-freedom robot arm obstacle avoidance control method based on machine vision according to claim 5, characterized in that: The S5 specifically includes: S51, Obstacle Space Modeling: Based on the obstacle distance and motion trajectory collected by the depth camera, the obstacle model in the robot workspace is constructed; S52, initialization of improved rapidly expanding random tree algorithm: after the obstacle space modeling is completed, the improved rapidly expanding random tree algorithm is initialized to plan the optimal obstacle avoidance path; S53, path search and node expansion: starting from the current position of the robot, randomly generate sample points, select the existing nodes closest to the sample points in the search space, and connect the new nodes by expanding the tree; S54, heuristic search improvement: based on the traditional fast expansion random tree algorithm, a heuristic search is introduced, which uses a heuristic function to estimate the distance between the current node and the target node, and gives priority to expanding nodes that are closer to the target; S55, path optimization: Based on the initially planned obstacle avoidance path, the path is adjusted through a local path optimization method to eliminate redundant nodes or unnecessary turning points.

7. The multi-degree-of-freedom robot arm obstacle avoidance control method based on machine vision according to claim 6, characterized in that: The S6 specifically includes: S61, environmental change detection: During the movement of the robot arm, the depth camera and RGB camera are used to monitor the changes in the surrounding environment in real time; S62, obstacle update: when a new obstacle is detected or the position of an existing obstacle changes, the spatial position and motion state information of the obstacle is updated, and the motion trajectory of the moving obstacle is corrected using a Kalman filter; S63, path re-planning trigger: after the environment changes or the obstacle position is updated, the preset path re-planning module is automatically triggered, and the improved fast expansion random tree algorithm is called to perform path search and planning again to generate an optimal path avoiding new obstacles; S64, path correction execution: after the path replanning is completed, the current motion trajectory of the robot arm is dynamically updated; S65, Continuous Path Correction Monitoring: When the robot arm executes a new path, it continuously monitors environmental changes and keeps real-time updates on obstacle positions and trajectories.

8. The multi-degree-of-freedom robot arm obstacle avoidance control method based on machine vision according to claim 1, characterized in that: The S7 includes: S71, robot arm motion state monitoring: real-time monitoring of the robot arm motion state through a preset sensor network, including a position sensor, an acceleration sensor, and an angular velocity sensor; S72, feedback data fusion: The data collected by multiple sensors are fused and processed, and the extended Kalman filter algorithm is used to filter the motion state of the robot arm to reduce the influence of sensor noise and improve feedback accuracy; S73, error calculation: based on the deviation between the actual motion state and the planned path, calculate the position error and attitude error.

9. The multi-degree-of-freedom robot arm obstacle avoidance control method based on machine vision according to claim 8, characterized in that: The S7 further comprises: S74, real-time control adjustment: according to the calculated error value, the feedback control module adjusts the motion parameters of the robot arm in real time through the sliding mode control algorithm; S75, Stability Analysis and Adjustment: Analyze the stability of the preset robot arm feedback control system through Lyapunov function; S76, Posture and motion smoothness control: During real-time adjustment, the angular velocity and acceleration of each joint of the robot arm are controlled to ensure the smoothness of the robot arm's motion.

Images