Vision-based object sorting robot path planning method and system

Abstract

The present invention discloses a method and system for path planning of an article sorting robot arm based on vision, and relates to the technical field of robot arm path planning. After the light source is automatically controlled, image data in the sorting area is collected, and after the target image is screened out, the detected objects are divided into obstacles and objects to be sorted, and marked on the real-time environmental map of the sorting area. After planning and acquiring the grasping path, the robot arm of the sorting robot grasps the objects to be sorted according to the target path, and the grasping scene data is used as feedback to optimize the movement speed and grasping path of the robot arm of the sorting robot; the grasping feedback value and feedback threshold are constructed from the grasping feedback data, and the corresponding sorting processing strategy is adopted according to the relationship between the grasping feedback value and the feedback threshold. The efficiency in the actual grasping process is improved, invalid grasping or inefficient grasping is reduced, and the grasping feedback effect of the robot arm is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot arm path planning, and in particular to a vision-based object sorting robot arm path planning method and system. Background Art

[0002] Robotic arm path planning refers to the process of planning and optimizing the motion trajectory of the robotic arm using the principles of forward and inverse kinematics and sampling planning algorithms during robot operation. This process aims to ensure that the robotic arm can avoid obstacles, ensure speed and accuracy during movement, so as to successfully reach the designated location and complete the task. Path planning plays a vital role in the operation of the robotic arm. It is widely used in many fields such as industry, medical care, and services, improving production efficiency, reducing surgical risks, and also improving people's quality of life.

[0003] In the Chinese invention patent with application publication number CN117506909A, a material sorting robot path planning method and system based on an improved ant colony algorithm is disclosed, including constructing a three-dimensional space plane of the robot, using an equal-space method to extract the grid points required for the three-dimensional path from the three-dimensional space; modeling threats and terrain obstacles in the environment; constructing an objective function during the operation of the robot; constructing a set of preliminary checkpoints and calculating the number of sections; obtaining the theoretical shortest path of the robot according to the starting point and end point of the robot's trajectory path; taking the section composed of the Y axis and the Z axis, rounding the intersection of the theoretical shortest path and the section, and constructing several rectangular planes with the rounded intersection as the center, uniformly selecting several preliminary checkpoints within each rectangular range; constructing an improved ant algorithm to optimize the robot's trajectory path. The present invention solves the problem that the robot's running trajectory is long and the path optimization performance is poor in the existing method.

[0004] When the robotic arm of a sorting robot grabs items on a sorting conveyor belt, grabbing efficiency and grabbing accuracy are two important feedback indicators. Existing sorting robots mainly rely on collecting image data in the sorting area, detecting, identifying and segmenting the images, and determining the location, type and other information of the items to be sorted. Then, they use a sensor network constructed by photoelectric sensors, distance sensors and the like to collect distance data and other related data, and use this as feedback to assist the sorting robot in making sorting and grabbing decisions, and complete the grabbing and sorting of items.

[0005] However, in this vision-based sorting method, when planning the grasping path for the robot arm before sorting, high requirements are placed on the light conditions in the sorting area and the subsequent image quality. Therefore, when the light conditions in the sorting area are poor, especially when the conveyor belt speed is fast and the position of the items changes rapidly during high-speed movement, the grasping difficulty of the robot arm will be further increased. If the grasping path of the robot arm cannot be adjusted in time, the grasping efficiency of the robot will be greatly affected.

[0006] To this end, the present invention provides a vision-based object sorting robot arm path planning method and system. Summary of the invention

[0007] 1. Technical issues to be resolved

[0008] In view of the deficiencies of the prior art, the present invention provides a method and system for path planning of an article sorting robot arm based on vision. By automatically controlling the light source and collecting image data in the sorting area, the detected articles are divided into obstacles and articles to be sorted, and marked on the real-time environmental map of the sorting area. After planning and acquiring the grasping path, the robot arm of the sorting robot grasps the articles to be sorted according to the target path, and the grasping scene data is used as feedback to optimize the movement speed and grasping path of the robot arm of the sorting robot; the grasping feedback value and feedback threshold are constructed from the grasping feedback data, and the corresponding sorting processing strategy is adopted according to the relationship between the grasping feedback value and the feedback threshold. The efficiency in the actual grasping process is improved, and invalid grasping or inefficient grasping is reduced, thereby solving the technical problems recorded in the background technology.

[0009] (II) Technical solution

[0010] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0011] The path planning method of the object sorting robot arm based on vision includes:

[0012] After analyzing the grabbing scene in the grabbing area, the scene complexity is generated from the grabbing scene data. If the scene complexity exceeds expectations, several light sources are evenly arranged in the sorting area.

[0013] After the light source is automatically controlled, the image data in the sorting area is collected, and after the target image is screened out, each item on the conveyor belt is distinguished through image detection and recognition;

[0014] Differentiate the detected objects into obstacles and objects to be sorted, mark them on the real-time environment map of the sorting area, update the movement trajectory and speed information of the objects, and predict their speed and position;

[0015] After planning and acquiring the grasping path, the mechanical arm of the sorting robot grasps the items to be sorted according to the target path, and uses the collected grasping scene data as feedback to optimize the movement speed and grasping path of the mechanical arm of the sorting robot;

[0016] After collecting the grasping feedback data during the grasping process, the grasping feedback value and feedback threshold are constructed based on the grasping feedback data, and the corresponding sorting processing strategy is adopted according to the relationship between the grasping feedback value and the feedback threshold.

[0017] Furthermore, the grabbing scenes of the current sorting cycle in the sorting area are analyzed, where the speed of object position change, the density of objects on the conveyor belt, and the light environment change data in the sorting area are recorded and summarized to generate a grabbing scene data set; the scene complexity is generated from the grabbing scene data set The way is as follows:

[0018] ;

[0019] Where: For time The eigenvector when , including item speed ,density and light conditions , is the weight matrix of the eigenvector; and is the weight coefficient, which ranges from 0 to 1, and the sum of the two is 1; is the bias vector of the constant term; is the weight matrix of the convolution operation, , is the convolution operation.

[0020] Furthermore, the light direction and intensity of the LED light source are automatically controlled using the trained light automatic control model with real-time light condition data as input; the image acquisition device acquires real-time image data of the sorting area, and obtains multiple identical sorting image data within the same time interval;

[0021] The obtained sorted image data is used as input, and the image quality is evaluated using the trained image quality evaluation model. According to the evaluation results, high-quality sorted images are selected as target images.

[0022] Furthermore, the frequency of adjusting the light conditions is determined based on the image quality evaluation results. Constraints are as follows:

[0023] ;

[0024] in, nis the number of images, It is i Image quality score and j The difference in image quality scores is is the average of the differences, To adjust the frequency; weight coefficient: , ,and .

[0025] Furthermore, the target image is used as input and the pre-trained YOLOv4 algorithm is used to detect objects in the target image and frame the location of each object;

[0026] A depth camera is used to obtain the 3D depth data of the object. The image of the depth camera is aligned and fused with the image of the ordinary camera to form an RGB-D image. After preprocessing, the object area in the depth image is segmented based on the change of the depth value using a clustering algorithm to distinguish the various items on the conveyor belt.

[0027] Furthermore, the surveying and mapping data of the acquired image data are fused, and a real-time environmental map covering the sorting area is constructed from the fused data, and each item is marked on the real-time environmental map; high-quality images are continuously arranged according to timestamps to obtain continuous image frames, and after detecting the items in the image, the optical flow method is used to estimate the direction and speed of the item movement and obtain real-time estimation data.

[0028] Further, based on the real-time estimation data, the movement trajectory and speed information of the object are updated on the real-time environment map;

[0029] If the current item is an item to be sorted, the remaining items will be regarded as obstacles. Based on the motion trajectory and speed information, the long short-term memory neural network will predict the positions of the items to be sorted and obstacles respectively, and the obtained position prediction data will be marked and updated on the real-time environment map.

[0030] Furthermore, based on the location data and moving speed data of obstacles and items to be sorted, a pre-trained path planning algorithm is used to plan the grabbing path of the robot arm for the items to be sorted;

[0031] The robot sorting digital twin model is constructed based on the robot's specification and performance data, conveyor belt operation data, and light condition data. The robot sorting digital twin model is used to verify the grasping path. If the sorting robot completes the sorting within the expected time, it will be used as the target path. If not, it will be optimized.

[0032] Furthermore, the mechanical arm of the sorting robot grabs the items according to the target path, and the sensor network and image acquisition device monitor the sorting process in real time, obtain the movement, grabbing and delivery status data of the mechanical arm of the corresponding sorting robot, and generate a grabbing scene data set after aggregation.

[0033] Furthermore, the similarity between the current grasping scenario and the expected grasping scenario is analyzed. If the similarity is lower than expected, the location and speed data of the item, as well as the specification attribute data of the item are collected and used as feedback data. The improvement of sorting efficiency is taken as the optimization goal, and the movement speed and grasping path of the robot arm are optimized by the pre-trained genetic algorithm.

[0034] Furthermore, after the current sorting cycle is completed, the grasping feedback data during the grasping process is collected, and the grasping feedback value is constructed from the sorting feedback data in the following manner:

[0035] ;

[0036] Where: is the length of the sorting cycle, For the moment The crawling speed, For the moment The accuracy of the crawling, For the moment The crawl error rate is , and is the weight coefficient, and its value falls between 0 and 1. , The value falls between 0 and 1.

[0037] Furthermore, if we capture the feedback value Exceeding feedback threshold , no further processing is done;

[0038] If you grab the feedback value At the feedback threshold To provide a crawl feedback value As the optimization target, the conveying speed of the conveyor belt is optimized using a pre-trained genetic algorithm;

[0039] If you grab the feedback value Below feedback threshold , under the current grasping scenario conditions, increase the number of enabled sorting robots, and use collaborative planning algorithms for task allocation and path coordination.

[0040] A vision-based object sorting robot path planning system includes a scene analysis unit that analyzes the grasping scene in the grasping area and generates scene complexity from the grasping scene data. If the scene complexity exceeds expectations, a number of light sources are evenly arranged in the sorting area.

[0041] The image detection unit automatically controls the light source to collect image data in the sorting area, selects the target image, and distinguishes the items on the conveyor belt through image detection and recognition;

[0042] The position prediction unit distinguishes the detected objects into obstacles and objects to be sorted, marks them on the real-time environmental map of the sorting area, updates the movement trajectory and speed information of the objects, and predicts their speed and position;

[0043] The path planning unit, after planning and acquiring the grasping path, enables the mechanical arm of the sorting robot to grasp the items to be sorted according to the target path, and uses the collected grasping scene data as feedback to optimize the movement speed and grasping path of the mechanical arm of the sorting robot;

[0044] The feedback optimization unit collects the grasping feedback data during the grasping process, constructs the grasping feedback value and feedback threshold based on the grasping feedback data, and adopts the corresponding sorting processing strategy according to the relationship between the grasping feedback value and the feedback threshold.

[0045] (III) Beneficial effects

[0046] The present invention provides a method and system for path planning of an object sorting robot arm based on vision, which has the following beneficial effects:

[0047] 1. Construct the complexity of the current grasping scene from the collected and recorded data, analyze and judge the current sorting difficulty of the sorting robot, and control and adjust the light conditions in the sorting area in real time through the installed light source. By improving the light conditions, the accuracy of recognition can be improved when the machine vision recognizes the objects.

[0048] 2. By constraining the frequency of adjusting the light conditions according to the image quality in the current stage, the adjustment frequency is adapted to the actual image quality, avoiding excessive adjustment and optimization of the current grasping scene, which will have a great impact on the current grasping state. After screening the current sorting image, the subsequent path planning and grasping effect can be improved; by detecting and segmenting the objects to be grasped, distribution errors can be avoided during sorting and the distribution efficiency can be improved.

[0049] 3. By estimating the moving speed and direction of the object, processing can be performed in advance when grasping to improve the accuracy and efficiency of grasping; verifying the feasibility of the grasping path through the robot sorting digital twin model can improve the efficiency of the actual grasping process and avoid multiple invalid grasping or inefficient grasping that affects the overall grasping efficiency of the sorting robot.

[0050] 4. Based on the collected feedback data, the movement speed and grasping path of the robot arm are optimized to improve the sorting efficiency at the current stage. The grasping feedback value is constructed by the collected feedback data to judge whether the grasping effect of the current robot arm has been improved. Targeted processing measures can be taken for the subsequent sorting process, and timely processing can be carried out when the current item sorting effect is not good.

[0051] 5. By optimizing and adjusting the speed of the conveyor belt, the moving speed of the items to be sorted can be adapted. By reducing the moving speed of the items to be sorted and reducing the grasping pressure of the subsequent robotic arm, the accuracy of sorting and grasping can be improved. Introducing new sorting robots and coordinating between sorting robots can ensure that the current sorting task can be completed effectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a schematic diagram of the process flow of the path planning method for a sorting robot arm of the present invention;

[0053] Figure 2 It is a schematic diagram of the structure of the path planning system of the sorting robot arm of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] See also Figure 1 The present invention provides a method for path planning of an object sorting robot arm based on vision, comprising:

[0056] Step 1: After analyzing the grabbing scene in the grabbing area, the scene complexity is generated from the grabbing scene data. If the scene complexity exceeds expectations, several light sources are evenly arranged in the sorting area;

[0057] The step 1 includes the following contents:

[0058] Step 101: when it is necessary to sort the items on the conveyor belt, determine the sorting area, and when the conveyor belt is in operation, analyze the grabbing scene of the current sorting cycle in the sorting area, wherein the speed of the change of the position of the items, the density of the items on the conveyor belt, and the change data of the light environment in the sorting area are recorded, and the recorded and acquired grabbing scene data are aggregated to generate a grabbing scene data set;

[0059] Step 102: Generate scene complexity from the data in the captured scene data set , the complexity of the current sorting environment is evaluated according to the scene complexity, as follows:

[0060] ;

[0061] Where: For time The eigenvector when , including item speed ,density and light conditions , is the weight matrix of the eigenvector; and is the weight coefficient, which ranges from 0 to 1, and the sum of the two is 1; is the bias vector of the constant term; is the weight matrix of the convolution operation, , is the convolution operation;

[0062] Preset the complexity threshold based on historical data and the complexity of the captured scene; if the scene complexity If the complexity threshold is exceeded, multiple high-frame-rate image acquisition devices are installed above and on both sides of the conveyor belt, and several automatically controllable LED light sources are evenly arranged;

[0063] When using, combine the contents in steps 101 and 102:

[0064] When using a sorting robot to sort different items on a conveyor belt, the current sorting process and grasping scene are collected, and the complexity of the current grasping scene is constructed from the collected and recorded data. The current sorting difficulty of the sorting robot is analyzed and judged. If the current sorting difficulty is large, in order to improve the sorting efficiency and grasping accuracy, the current sorting state and sorting process can be adjusted.

[0065] However, in this vision-based sorting method, when planning the grasping path for the robot arm before sorting, the grasping path planning has high requirements on the light conditions in the sorting area and the subsequent image imaging quality. Therefore, when the light conditions in the sorting area are poor, especially in the grasping scenario where the conveyor belt speed is fast and the position of the objects changes rapidly during high-speed movement, the grasping difficulty of the robot arm will be further increased. If the grasping path of the robot arm cannot be adjusted in time, the grasping efficiency of the robot will be greatly affected.

[0066] Step 2: After the light source is automatically controlled, image data in the sorting area is collected, and after the target image is screened out, each item on the conveyor belt is distinguished through image detection and recognition;

[0067] The step 2 includes the following contents:

[0068] Step 201: training a machine learning algorithm with the labeled sample data to obtain a trained automatic light control model;

[0069] After setting the target value for the light parameters in the sorting area, the light direction and intensity of the LED light source are automatically controlled using the trained light automatic control model with real-time light condition data as input; the image acquisition device acquires real-time image data of the sorting area and obtains multiple identical sorting image data within the same time interval;

[0070] When in use, when the current grasping difficulty of the sorting robot arm is large, as the first improvement method, the light conditions in the sorting area are controlled and adjusted in real time through the installed light source. By improving the light conditions, the accuracy of recognition can be improved when the machine vision recognizes the objects.

[0071] Step 202: training a machine learning algorithm with the labeled sample data to obtain a trained image quality evaluation model;

[0072] The obtained sorting image data is used as input, and the image quality is evaluated using the trained image quality evaluation model. The high-quality sorting images are selected as the target images based on the evaluation results. On the basis of light adjustment control, the image quality can be screened to further improve the capture effect.

[0073] Frequency of adjusting light conditions based on image quality evaluation results under dimensionless conditions Constraints are as follows:

[0074] ;

[0075] in, n is the number of images, It is iImage quality score and j The difference in image quality scores is is the average of the differences, To adjust the frequency; weight coefficient: , ,and ;

[0076] Step 203: Using the target image as input, use the pre-trained YOLOv4 algorithm to detect objects in the target image and identify the location of each object;

[0077] Use a depth camera to obtain the 3D depth data of the object, align and fuse the image of the depth camera with that of the ordinary camera to form an RGB-D image, and use a clustering algorithm to segment the object area in the depth image based on the change in depth value after preprocessing to distinguish the various objects on the conveyor belt;

[0078] When using, combine the contents in steps 201 to 203:

[0079] By constraining the adjustment frequency of the light conditions according to the image quality in the current stage, the adjustment frequency is adapted to the actual image quality, avoiding excessive adjustment to optimize the current grasping scene and causing interference to the current grasping state. Furthermore, when the current image acquisition environment is good and the current sorting image is screened, the subsequent path planning and grasping effects can be further improved. As a further content, in order to achieve better results, when there are many items on the conveyor belt and different items are stacked together, the items to be grasped can be detected and segmented to avoid distribution errors during sorting, thereby improving distribution efficiency while ensuring the current sorting speed.

[0080] It should be noted that when the power supply in the grabbing area is insufficient, the lighting will be unstable, such as the light state will fluctuate and gradually dim. The specific reasons are as follows:

[0081] Voltage fluctuation refers to the unstable change of power supply voltage within a certain range. This change will directly affect the stability and quality of lighting. Specifically: Incandescent lamp: If the voltage exceeds the rated value by 10%, the power loss will increase to 21%, which will greatly shorten the life of the incandescent lamp, and may even reach a 70% reduction. On the contrary, if the voltage is lower than the rated value, the luminous efficiency of the incandescent lamp will drop sharply. Fluorescent lamp: Voltage fluctuation will cause the brightness of the fluorescent lamp to gradually decrease. In the case of large voltage fluctuations, the fluorescent lamp may have a continuous flickering feeling, affecting the lighting effect. Other lamps: For high-voltage type lamps, such as pumps, if the voltage continues to decrease by 20% to 30%, the lamp may go out after 0.5 to 1 second. In addition, for equipment such as experimental equipment that requires strict control of output accuracy, voltage fluctuations will also affect its accuracy.

[0082] The choice of power supply method will also affect the stability and quality of lighting. For example, in road and bridge lighting projects, the traditional low-voltage power supply network may cause a large voltage drop in the power supply line due to the voltage drop of the transformer and the voltage drop along the length of the cable. This will lead to an increase in the number of substations and an increase in cable diameter, thereby increasing investment and operating costs.

[0083] As another reason, the staff and various types of equipment in the grasping area are in a mobile state, which will also form various forms of obstructions, resulting in certain fluctuations in light conditions. In this scenario, in order to ensure the grasping effect, it is necessary to respond to and control the lighting conditions in a targeted manner in real time.

[0084] Step 3: Differentiate the detected objects into obstacles and objects to be sorted, mark them on the real-time environment map of the sorting area, update the movement trajectory and speed information of the objects, and predict their speed and position;

[0085] The step three includes the following contents:

[0086] Step 301: The image acquisition device and the laser radar are used to image and map the sorting area, and the radar mapping data of the acquired image data are fused. A real-time environment map covering the sorting area is constructed from the fused data, and each item on the conveyor belt is marked on the real-time environment map;

[0087] Arrange the high-quality images continuously according to the timestamps to obtain continuous image frames. After detecting the objects in the images, use the optical flow method to estimate the moving direction and speed of the objects and obtain real-time estimation data.

[0088] Step 302: Update the movement trajectory and speed information of the object on the real-time environment map according to the real-time estimation data;

[0089] If the current item is an item to be sorted, the remaining items are regarded as obstacles. Based on the motion trajectory and speed information, the long short-term memory neural network predicts the positions of the items to be sorted and obstacles respectively, and marks and updates the acquired position prediction data on the real-time environment map.

[0090] When used, combine the contents in steps 301 and 302:

[0091] After collecting relevant data of the sorting area, when the items to be sorted are in a moving state, the moving speed and direction of the items are estimated and processed in advance when they are grasped, which can improve the accuracy and efficiency of grasping.

[0092] Step 4: After planning and acquiring the grasping path, the mechanical arm of the sorting robot grasps the items to be sorted according to the target path, and uses the collected grasping scene data as feedback to optimize the movement speed and grasping path of the mechanical arm of the sorting robot;

[0093] The step 4 includes the following contents:

[0094] Step 401: Based on the position data and moving speed data of obstacles and items to be sorted, a pre-trained path planning algorithm is used to plan a grabbing path for the robot arm to sort the items;

[0095] The robot sorting digital twin model is constructed based on the robot's specification performance data, conveyor belt operation data, and light condition data. The grasping path is used as input, and the robot sorting digital twin model verifies the feasibility of the grasping path. If the sorting robot completes the sorting within the expected time, it is used as the target path. If not, it is optimized.

[0096] During use, in order to further improve the grasping efficiency of the sorting robot, after completing the grasping path planning of the sorting robot's mechanical arm, the feasibility of the grasping path is verified through the robot sorting digital twin model, which can improve the efficiency of the actual grasping process and avoid multiple invalid grasps or inefficient grasps that affect the overall grasping efficiency of the sorting robot.

[0097] Step 402: The sorting robot is made to grab the items according to the target path, and the sensor network and the image acquisition device monitor the sorting process in real time, obtain the robot arm movement, grabbing and delivery status data of the corresponding sorting robot, and generate a grabbing scene data set after aggregation;

[0098] Analyze the similarity between the current grasping scene and the expected grasping scene. If the similarity is lower than expected, it means that the current sorting state fails to meet expectations. Collect the location and speed data of the items, and the specification attribute data of the items, such as shape and weight, etc., and use these data as feedback data. Take them as input, and take improving the sorting efficiency as the optimization goal. The pre-trained genetic algorithm is used to optimize the movement speed and grasping path of the robot arm.

[0099] When using, combine the contents in steps 401 and 402:

[0100] By performing a similarity analysis between the grasping state in the current grasping process and the expected state, it is verified whether the current grasping effect is consistent with the expected effect and whether it has reached the expected level. If it fails to reach the expected level, the movement speed and grasping path of the robot arm are optimized based on the collected feedback data to improve the sorting efficiency at the current stage.

[0101] Step 5: After collecting the grasping feedback data during the grasping process, the grasping feedback value and the feedback threshold are constructed from the grasping feedback data, and a corresponding sorting processing strategy is adopted according to the relationship between the grasping feedback value and the feedback threshold;

[0102] The step five includes the following contents:

[0103] Step 501: After the current sorting cycle ends, the grasping feedback data in the grasping process is collected to obtain the grasping speed and grasping accuracy of each item. Under dimensionless conditions, the grasping feedback value is constructed from the sorting feedback data in the following manner:

[0104] ;

[0105] Where: is the length of the sorting cycle, For the moment The crawling speed, For the moment The accuracy of the capture, For the moment The crawl error rate is , and is the weight coefficient, and its value falls between 0 and 1. , The value falls between 0 and 1;

[0106] Feedback values ​​from several consecutive captures Building feedback thresholds , as follows:

[0107] ;

[0108] in, , is the number of feedback values ​​captured during the sorting cycle, is the mean of the grasping feedback value; For the data column i A crawl feedback value;

[0109] When in use, after the sorting robot completes a stage of sorting tasks, the grasping feedback value is constructed by the collected feedback data, which can be used to judge whether the current grasping effect of the robot arm has been improved, and targeted processing measures can be taken for the subsequent sorting process. When the current item sorting effect is not good, timely processing can be carried out;

[0110] Step 502: Based on the captured feedback value With feedback threshold The corresponding sorting and processing strategies are adopted according to the relationship between them, as follows:

[0111] If you grab the feedback value Exceeding feedback threshold , no further processing is done;

[0112] If you grab the feedback value At the feedback threshold To provide a crawl feedback value As the optimization target, the conveying speed of the conveyor belt is optimized using a pre-trained genetic algorithm;

[0113] If you grab the feedback value Below feedback threshold Under the current grasping scenario conditions, the number of enabled sorting robots is increased, and collaborative planning algorithms are used for task allocation and path coordination to avoid collisions and interferences between robots, thereby improving the overall sorting efficiency.

[0114] When using, combine the contents in steps 501 and 502:

[0115] When the feedback effect of item sorting and grasping does not exceed expectations, the speed of the conveyor belt is optimized and adjusted to adapt the moving speed of the items to be sorted. By reducing the moving speed of the items to be sorted, the grasping pressure of the subsequent robotic arm is reduced, thereby improving the accuracy of sorting and grasping. If this method still cannot improve the current feedback effect, a new sorting robot is introduced to coordinate the sorting robots to ensure that the current sorting task can be completed effectively.

[0116] See also Figure 2 The present invention provides a vision-based object sorting robot arm path planning system, comprising:

[0117] The scene analysis unit analyzes the grabbing scene in the grabbing area and generates scene complexity from the grabbing scene data. If the scene complexity exceeds expectations, several light sources are evenly arranged in the sorting area.

[0118] The image detection unit automatically controls the light source to collect image data in the sorting area, selects the target image, and distinguishes the items on the conveyor belt through image detection and recognition;

[0119] The position prediction unit distinguishes the detected objects into obstacles and objects to be sorted, marks them on the real-time environmental map of the sorting area, updates the movement trajectory and speed information of the objects, and predicts their speed and position;

[0120] The path planning unit, after planning and acquiring the grasping path, enables the mechanical arm of the sorting robot to grasp the items to be sorted according to the target path, and uses the collected grasping scene data as feedback to optimize the movement speed and grasping path of the mechanical arm of the sorting robot;

[0121] The feedback optimization unit collects the grasping feedback data during the grasping process, constructs the grasping feedback value and feedback threshold based on the grasping feedback data, and adopts the corresponding sorting processing strategy according to the relationship between the grasping feedback value and the feedback threshold.

[0122] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0123] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0124] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only some logical function divisions. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A vision-based object sorting robot path planning method, characterized by: include, After analyzing the grabbing scene in the grabbing area, the scene complexity is generated from the grabbing scene data. If the scene complexity exceeds expectations, several light sources are evenly arranged in the sorting area. After the light source is automatically controlled, the image data in the sorting area is collected, and after the target image is screened out, each item on the conveyor belt is distinguished through image detection and recognition; Differentiate the detected objects into obstacles and objects to be sorted, mark them on the real-time environment map of the sorting area, update the movement trajectory and speed information of the objects, and predict their speed and position; After planning and acquiring the grasping path, the mechanical arm of the sorting robot grasps the items to be sorted according to the target path, and uses the collected grasping scene data as feedback to optimize the movement speed and grasping path of the mechanical arm of the sorting robot; After collecting the grasping feedback data during the grasping process, the grasping feedback value and feedback threshold are constructed based on the grasping feedback data, and the corresponding sorting processing strategy is adopted according to the relationship between the grasping feedback value and the feedback threshold; Analyze the grabbing scenes of the current sorting cycle in the sorting area, record the speed of object position change, the density of objects on the conveyor belt, and the light environment change data in the sorting area, and summarize them to generate a grabbing scene data set; the method of generating the scene complexity Cjp from the grabbing scene data set is as follows: Where: X(t) is the scene feature vector at time t, It includes object velocity V(t), density ρ(t) and light intensity L(t), W is the weight matrix of scene feature vector; α and β are weight coefficients, b is the bias vector of constant term; W C is the weight matrix of the convolution operation, h(t τ )=e -t , * is the convolution operation.

2. The method for path planning of a vision-based object sorting robot arm according to claim 1, characterized in that: Using real-time light condition data as input, the trained light automatic control model is used to automatically control the light direction and intensity of the LED light source; the image acquisition device acquires real-time image data of the sorting area and obtains multiple identical sorting image data within the same time interval; The obtained sorted image data is taken as input, and the image quality is evaluated using the trained image quality evaluation model. A high-quality sorted image is selected as the target image based on the evaluation result.

3. The method for path planning of a vision-based object sorting robot arm according to claim 2, characterized in that: The adjustment frequency Xo of the light condition is constrained according to the image quality evaluation result. The constraint method is as follows: Where n is the number of images, Ty ij is the difference between the i-th image quality score and the j-th image quality score, Ty a is the average value of the difference, Xo is the adjustment frequency; weight coefficient: 0≤S1≤1, 0≤S2≤1, and S1+S2=1.

4. The method for path planning of a vision-based object sorting robot arm according to claim 3, characterized in that: Take the target image as input and use the pre-trained YOLOv4 algorithm to detect objects in the target image and frame the location of each object; A depth camera is used to obtain the 3D depth data of the object. The image of the depth camera is aligned and fused with the image of the ordinary camera to form an RGB-D image. After preprocessing, the object area in the depth image is segmented based on the change of the depth value using a clustering algorithm to distinguish the various items on the conveyor belt.

5. The method for path planning of a vision-based object sorting robot arm according to claim 4, characterized in that: The surveying and mapping data of the acquired image data are fused, and a real-time environment map covering the sorting area is constructed from the fused data, and each item is marked on the real-time environment map; The high-quality images are arranged continuously according to the timestamps to obtain continuous image frames. After detecting the objects in the image, the optical flow method is used to estimate the moving direction and speed of the objects and obtain real-time estimation data.

6. The method for path planning of a vision-based object sorting robot arm according to claim 5, characterized in that: Based on the real-time estimation data, the object's movement trajectory and speed information are updated on the real-time environment map; If the current item is an item to be sorted, the remaining items will be regarded as obstacles. Based on the motion trajectory and speed information, the long short-term memory neural network will predict the positions of the items to be sorted and obstacles respectively, and the obtained position prediction data will be marked and updated on the real-time environment map.

7. The method for path planning of a vision-based object sorting robot arm according to claim 6, characterized in that: Based on the location data and moving speed data of obstacles and items to be sorted, the robot arm uses a pre-trained path planning algorithm to plan the grabbing path for the items to be sorted. The robot sorting digital twin model is constructed based on the robot's specification performance data, conveyor belt operation data and light condition data. The robot sorting digital twin model is used to verify the grasping path. If the sorting robot completes the sorting within the expected time, it will be used as the target path. If not, it will be optimized.

8. The method for path planning of a vision-based object sorting robot arm according to claim 7, characterized in that: The sorting robot's mechanical arm grabs items according to the target path, and the sensor network and image acquisition device monitor the sorting process in real time, obtain the corresponding sorting robot's mechanical arm movement, grabbing and delivery status data, and summarize them to generate a grabbing scene data set.

9. The method for path planning of a vision-based object sorting robot arm according to claim 8, characterized in that: Analyze the similarity between the current grasping scene and the expected grasping scene. If the similarity is lower than expected, collect the location and speed data of the item, and the specification attribute data of the item, and use them as feedback data. Take improving the sorting efficiency as the optimization goal, and use the pre-trained genetic algorithm to optimize the movement speed and grasping path of the robot arm.

10. The method for path planning of a vision-based object sorting robot arm according to claim 9, characterized in that: After the current sorting cycle is over, the grasping feedback data during the grasping process is collected, and the grasping feedback value Fks is constructed from the sorting feedback data in the following way: Where: T is the length of the sorting cycle, S(t) is the grasping speed at time t, A(t) is the grasping accuracy at time t, E(t) is the grasping error rate at time t, F1, F2 and F3 are weight coefficients, and the values ​​fall between 0 and 1, w(t,λ)=e -λt ,e=2.713λ value falls between 0 and 1.

11. The method for path planning of a vision-based object sorting robot arm according to claim 10, characterized in that: If the captured feedback value Fks is greater than the feedback threshold Qb, no further processing will be performed; If the grasping feedback value Fks is greater than or equal to the feedback threshold value Qa and less than or equal to the feedback threshold value Qb, the grasping feedback value Fks is provided as the optimization target, and the conveying speed of the conveyor belt is optimized using the pre-trained genetic algorithm; If the grasping feedback value Fks is less than the feedback threshold Qa, under the current grasping scenario conditions, the number of enabled sorting robots is increased, and the collaborative planning algorithm is used for task allocation and path coordination.

12. A vision-based object sorting robot path planning system, applying the method described in any one of claims 1 to 11, characterized in that: include, The scene analysis unit analyzes the grabbing scene in the grabbing area and generates scene complexity from the grabbing scene data. If the scene complexity exceeds expectations, several light sources are evenly arranged in the sorting area. The image detection unit automatically controls the light source to collect image data in the sorting area, selects the target image, and distinguishes the items on the conveyor belt through image detection and recognition; The position prediction unit distinguishes the detected objects into obstacles and objects to be sorted, marks them on the real-time environmental map of the sorting area, updates the movement trajectory and speed information of the objects, and predicts their speed and position; The path planning unit, after planning and acquiring the grasping path, enables the mechanical arm of the sorting robot to grasp the items to be sorted according to the target path, and uses the collected grasping scene data as feedback to optimize the movement speed and grasping path of the mechanical arm of the sorting robot; The feedback optimization unit collects the grasping feedback data during the grasping process, constructs the grasping feedback value and feedback threshold based on the grasping feedback data, and adopts the corresponding sorting processing strategy according to the relationship between the grasping feedback value and the feedback threshold.

Images