Navigation method and sorting system of multimodal mobile sorting robot

Abstract

The invention discloses a navigation method and a sorting system for a multimodal mobile sorting robot, and relates to the field of robot navigation technology. When light fluctuations in a sorting area exceed expectations, an image acquisition mechanism is triggered before an object to be sorted arrives at a predetermined position; multimodal features are extracted after the acquired auditory data and visual data are fused, and the object to be sorted is identified based on the multimodal features; after the object to be sorted is positioned using visual data and auditory data respectively, if the degree of deviation between the two position data exceeds expectations, the position data of the object to be sorted is corrected; a sorting value is generated from the information of the object to be sorted, and after the object to be sorted is marked with the sorting value and the real-time position, a moving path for the sorting robot to move to the object to be sorted is planned; and the multimodal data is used to identify and locate the object to be sorted.

Description

Technical Field

[0001] The present invention relates to the technical field of robot navigation, and in particular to a navigation method and a sorting system of a multi-modal mobile sorting robot. Background Art

[0002] Sorting robots are indispensable and efficient automated equipment in modern logistics and warehousing systems. By integrating advanced sensor technology, precise mechanical structure and intelligent control system, they can realize rapid identification, accurate grasping and efficient sorting of goods, significantly improving the efficiency and accuracy of warehousing operations. They are an important force in promoting the transformation of the logistics industry towards intelligence and automation.

[0003] The multimodal navigation of the sorting robot also integrates a variety of high-precision sensors such as cameras, infrared sensors, and lidar, as well as advanced information fusion and path planning algorithms. By capturing and processing multi-dimensional information of the surrounding environment in real time, this technology enables the sorting robot to achieve precise navigation in complex and ever-changing scenes, effectively avoid obstacles, and plan the optimal sorting path. The application of this technology has greatly improved the sorting efficiency and accuracy in logistics, warehousing, and manufacturing, and provided strong technical support for automated and intelligent production.

[0004] In the application document with application publication number CN116295434A, a navigation method of a navigation robot based on visual recognition is disclosed, including a navigation robot, wherein the navigation robot is equipped with an imager, and the method includes imaging, modeling, marking obstacle points, eliminating obstacles, path planning and running path; a spatial coordinate system is established through the imaging picture of the navigation robot, and coordinates of obstacles in the imaging picture are established to eliminate them at the same time, multiple path plans are planned according to the remaining coordinate positions of the spatial coordinate system, and the optimal path plan is selected in combination with the parameter data of the navigation robot for actual navigation.

[0005] Combined with the above application and the contents of the prior art:

[0006] Before using a sorting robot to sort items, it is necessary to identify the items to be sorted, for example, determine the location data, category information and sorting requirements of the items to be sorted, and navigate the robot. Existing recognition methods are usually based on machine vision. For example, after collecting image data of the items to be sorted, the image data is detected and identified, and relevant information of the items to be sorted is extracted, and the items to be sorted are sorted based on the relevant information.

[0007] However, when existing sorting robots navigate based on vision, they usually have high requirements for image quality data. Low-quality image data will affect subsequent image detection and recognition. For example, it is difficult to accurately identify the items to be sorted in the image. When the light conditions in the sorting area are difficult to maintain at a high level, for example, shadows often appear in the sorting area, and the light conditions of the sorting robots and image acquisition devices often fluctuate greatly, the imaging quality is difficult to guarantee. At this time, if it is still based only on machine vision, the reliability of identifying and positioning the items to be sorted will be affected to a certain extent, thereby affecting the sorting efficiency.

[0008] To this end, the present invention provides a navigation method and a sorting system for a multi-modal mobile sorting robot. Summary of the invention

[0009] 1. Technical issues to be resolved

[0010] In view of the deficiencies in the prior art, the present invention provides a navigation method and a sorting system for a multimodal mobile sorting robot, which extracts multimodal features by fusing the collected auditory data and visual data, and identifies the items to be sorted based on the multimodal features; after locating the items to be sorted by visual data and auditory data respectively, if the deviation between the two position data exceeds expectations, the position data of the items to be sorted is corrected; a sorting value is generated from the information of the items to be sorted, and after marking the items to be sorted with the sorting value and the real-time position, a moving path is planned for the sorting robot to move to the items to be sorted; the multimodal data is used to identify and locate the items to be sorted, so as to improve the efficiency and reliability of sorting, thereby solving the technical problems recorded in the background technology.

[0011] (II) Technical solution

[0012] To achieve the above objectives, the present invention is implemented through the following technical solutions: a navigation method for a multimodal mobile sorting robot, comprising: when the light fluctuation in the sorting area exceeds expectations, triggering an image acquisition mechanism before the object to be sorted arrives at a predetermined position; wherein the advance acquisition time of the object image data is constrained according to the light fluctuation value and the position change of the object to be sorted. , the constraints are as follows:

[0013] ;

[0014] Weight coefficient, , ; is the number of time nodes, Is the item to be sorted Time point The distance between time nodes, is the average distance, Is the item to be sorted Time point The difference in change value between time nodes, is the average value of light fluctuation;

[0015] The collected auditory data and visual data are fused to extract multimodal features, and the items to be sorted are identified based on the multimodal features; after the items to be sorted are located using the visual data and auditory data respectively, if the deviation between the two position data exceeds the expectation, the position data of the items to be sorted is corrected;

[0016] Generate a sorting value from the information of the items to be sorted, mark the items to be sorted with the sorting value and real-time position, and then plan a moving path for the sorting robot to move to the items to be sorted;

[0017] Plan the grabbing path of the sorting robot's mechanical arm and grab the items to be sorted and place them in the designated disposal area.

[0018] Furthermore, the photoelectric sensors located at various positions in the sorting area collect light data, summarize and generate a light data set in the sorting area, and generate a light fluctuation value based on the change state of the light data. , as follows:

[0019] ;

[0020] Where: is the total number of photoelectric sensors, is the total number of time nodes, For time node Top, The degree of a photoelectric sensor.

[0021] Furthermore, when the items to be sorted with sound-emitting elements attached are in a continuous moving state on the conveyor belt, the advance collection time of the image data of the items is constrained according to the light fluctuation value and the position change of the items to be sorted, and the image collection mechanism is triggered before arriving at the predetermined position, and an image collection instruction is issued to the outside.

[0022] Furthermore, after receiving the image acquisition instruction, the image acquisition device acquires the visual data of the items to be sorted on the conveyor belt, and the auditory data in the sorting area is acquired through the microphone array, and the data is aligned through timestamps and registration.

[0023] Furthermore, the different modal features extracted from the aligned data are combined to form multimodal features;

[0024] The multimodal features of the items to be sorted are used as input, and the trained model for identifying the items to be sorted is used for identification. If the items to be sorted contained therein are identified, they are used as the target image and the identification data is obtained.

[0025] Further, the coordinates of the target image are converted into actual space coordinates to determine the visual position of the object to be sorted, and the position information of the sound-emitting element on the object to be sorted is located by a microphone array and used as the auditory position;

[0026] The auditory position and visual position data of the items to be sorted are aggregated to generate a position data set of the items to be sorted.

[0027] Furthermore, a deviation value is generated from the position data in the position data set of the items to be sorted. If the deviation value is higher than the deviation threshold, a position correction instruction is sent to the outside. After receiving the position correction instruction, a supplementary positioning method is introduced to correct the position data of the items to be sorted, wherein the distance between the auditory position and the visual position is obtained and used as the deviation distance. When under dimensionless conditions, the deviation value is generated according to the following formula :

[0028] ;

[0029] in, For the The deviation distance of each time node, is the mean of the deviation distances, is the weight coefficient, ; is the number of time nodes, is the accepted value for the deviation distance.

[0030] Furthermore, the image acquisition device collects multi-angle image data of the objects to be sorted in the sorting area, uses a text detection algorithm to detect the text area in the image, and uses an OCR algorithm to perform character recognition to extract corresponding text information; the type of the objects to be sorted is identified based on the text information, and the description information and material specification data of the objects to be sorted are extracted to generate an information set of the objects to be sorted.

[0031] Furthermore, the information of the items to be sorted is used as input, and the trained sorting evaluation model is used to score, and the corresponding sorting value is obtained, and the sorting priority of the items to be sorted is determined according to the size of the sorting value;

[0032] Use tracking algorithms to track items to be sorted in real time, update the location of items in real time and record their movement trajectory.

[0033] Furthermore, an electronic map covering the sorting area is established, and after obstacles located in the sorting area are determined, the obstacles are marked on the electronic map; based on the obstacles in the sorting area and the sorting values, real-time location information and predicted location information of the items to be sorted, a path planning algorithm is used to plan a moving path for the sorting robot to move to the items to be sorted, and the moving path is displayed on the electronic map.

[0034] The sorting system of the multimodal mobile sorting robot includes the following:

[0035] The grasping path planning unit identifies the posture of the end effector of the robotic arm of the sorting robot, plans the grasping path of the robotic arm using the RRT or RRT* algorithm, and then smoothes the grasping path using the BSpline algorithm;

[0036] The path testing unit takes the grasping path as input and uses the robot arm sorting digital twin model to verify whether the grasping path is feasible. If not, the grasping path is optimized. If feasible, a grasping instruction is issued;

[0037] The sorting control unit controls the end effector of the robot arm to grab the items to be sorted after receiving the grabbing instruction;

[0038] The placement unit determines the placement point of the items to be sorted according to the preset rules or task requirements, and then places the items to be sorted in the designated disposal area according to the item identification results and preset classification rules.

[0039] (III) Beneficial effects

[0040] The present invention provides a navigation method and a sorting system for a multi-modal mobile sorting robot, which have the following beneficial effects:

[0041] 1. Analyze and judge the current image acquisition conditions and image detection and recognition conditions based on the light fluctuation value, and collect images of the items to be sorted before they arrive at the predetermined location to avoid omissions in image acquisition and failure to fully supplement in scenes with poor lighting conditions.

[0042] 2. It enables mutual verification and confirmation between sound data and image data, which reduces the risk of errors when used to identify and locate items to be sorted.

[0043] 3. Rapidly locate the items to be sorted based on the sound signals emitted, build deviation values ​​based on the degree of deviation between positioning, judge the degree of deviation between different positioning, and improve the reliability of current position detection data.

[0044] 4. Reposition the items to be sorted, correct and adjust the previous positioning data based on the new location data obtained, trigger the positioning correction mechanism when the reliability of the existing positioning method decreases, enable the ranging radar that is not in continuous operation, and improve the accuracy of subsequent sorting when sorting the items to be sorted.

[0045] 5. Evaluate the importance and priority of the items to be sorted. According to the sorting priority, the corresponding sorting order can be determined in the subsequent sorting process; plan the moving path for the sorting robot, and adjust the moving path of the sorting robot so that the robotic arm of the sorting robot can quickly grab and sort the items to be sorted.

[0046] 6. Determine the feasibility of the grasping path based on the simulation test data. When the feasibility is insufficient, optimize the current grasping path based on the simulation test data. On the basis of planning the moving path of the sorting robot, further improve the efficiency of grasping and sorting.

[0047] 7. In scenarios with poor lighting conditions in the sorting area, when using a sorting robot to sort items, the introduction of sound elements as a supplement and the use of multimodal data to identify and locate the items to be sorted can improve the efficiency and reliability of sorting. In addition, by determining the sorting priority of each item to be sorted, important items can be sorted and grabbed first, thereby improving the overall sorting efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a navigation flow chart of the mobile sorting robot of the present invention;

[0049] Figure 2 This is a schematic diagram of a navigation method for a mobile sorting robot of the present invention;

[0050] Figure 3 The figure is a schematic diagram of the structure of the sorting system of the sorting robot of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] See also Figure 1 and 2 The present invention provides a navigation method for a multi-modal mobile sorting robot, comprising:

[0053] Step 1: Collect light condition data in the sorting area. When the light fluctuation in the sorting area exceeds expectations, trigger the image acquisition mechanism before the items to be sorted arrive at the predetermined location.

[0054] The step 1 includes the following contents:

[0055] Step 101: After determining the sorting area, several groups of photoelectric sensors are set in the sorting area, and the photoelectric sensors located at various positions in the sorting area collect light data, and summarize and generate a light data set in the sorting area;

[0056] Generate light fluctuation value from the changing state of light data , the degree of light dimness change in the sorting area is determined based on the obtained light fluctuation value, as follows:

[0057] ;

[0058] Where: is the total number of photoelectric sensors, is the total number of time nodes, For time node Top, The degree of a photoelectric sensor;

[0059] When in use, before identifying and sorting items on the conveyor belt, collect and analyze the light conditions in the sorting area, analyze and judge the current image acquisition conditions and image detection and recognition conditions based on the light fluctuation value, and verify the current difficulty of sorting items. If the current sorting scene is poor and inconsistent with expectations, other identification and positioning methods can be introduced to improve the reliability of subsequent sorting and grasping;

[0060] Step 102: After obtaining the light fluctuation value in the sorting area, under dimensionless conditions, the advance collection time of the item image data is constrained according to the light fluctuation value and the position change constraint of the item to be sorted. , the constraints are as follows:

[0061] ;

[0062] Weight coefficient, , ; is the number of time nodes, Is the item to be sorted Time point The distance between time nodes, is the average distance, Is the item to be sorted Time point The difference in change value between time nodes, is the average value of light fluctuation;

[0063] When the items to be sorted with sound-emitting elements attached are in a continuous moving state on the conveyor belt, the arrival time of the items to be sorted at the predetermined location is predicted, and the time before the items arrive at the predetermined location is predicted. The image acquisition mechanism is triggered before, at which time, an image acquisition instruction is sent to the outside;

[0064] When using, combine the contents in steps 101 and 102:

[0065] Taking into account the inconsistency between the sorting scene environment and expectations, in order to prevent the failure to collect image data of the sorted items in a timely and effective manner, image collection is performed on the sorted items before they arrive at the predetermined location to avoid omissions in image collection and failure to fully supplement in scenes with poor lighting conditions.

[0066] However, when existing sorting robots sort items based on vision, they usually have high requirements for image quality data. Low-quality image data will affect subsequent image detection and recognition. For example, it is difficult to accurately identify the items to be sorted in the image. When the light conditions in the sorting area are difficult to maintain at a high level, for example, shadows often appear in the sorting area, and the light conditions of the sorting robots and image acquisition devices often fluctuate greatly, the imaging quality is difficult to guarantee. At this time, if the items to be sorted are still sorted only based on machine vision, the reliability of identifying and positioning the items to be sorted will be affected to a certain extent, thereby affecting the sorting efficiency.

[0067] Step 2: After fusing the collected auditory data and visual data, extract multimodal features, identify the items to be sorted based on the multimodal features, and obtain identification data;

[0068] The step 2 includes the following contents:

[0069] Step 201: After receiving the image acquisition instruction, the image acquisition devices at least located on both sides of the objects to be sorted acquire image data of the objects to be sorted on the conveyor belt as visual data; the microphone array acquires ambient sound data in the sorting area as auditory data;

[0070] After aligning the visual data and auditory data through timestamps, the data is then aligned in space through calibration and registration to ensure the temporal consistency of data in different modalities.

[0071] When in use, before sorting the items to be sorted, by simultaneously collecting sound data, and by aligning the two different modal data, the accuracy is improved when the data is used for identifying and locating the items to be sorted.

[0072] Step 202: extract features from the aligned visual data and auditory data, such as image features, sound spectrum features, etc., and combine features of different modes to form multi-modal features; wherein, when the sound-emitting element emits sound to the items to be sorted, the sound will also be reflected outward after passing through or penetrating the items to be sorted, and the sound features after these reflections will also reflect the attribute information of the items to be sorted;

[0073] The multimodal convolutional neural network is trained with the labeled multimodal features to obtain a trained recognition model of items to be sorted;

[0074] The multimodal features of the items to be sorted are used as input, and the trained model for identifying the items to be sorted is used for identification. If the items to be sorted contained therein are identified, the images are used as target images and identification data is obtained, thus completing the identification process of the items to be sorted.

[0075] When using, combine the contents in steps 201 and 202:

[0076] After acquiring image data and sound data to construct multimodal features, compared with the recognition of single image data, the sound data and image data can be mutually verified and confirmed, and the risk of errors is lower when used to identify and locate items to be sorted.

[0077] Step 3: After the items to be sorted are located by the visual data and the auditory data respectively, if the deviation between the two position data exceeds the expectation, the position data of the items to be sorted is corrected;

[0078] The step three includes the following contents:

[0079] Step 301: Convert the coordinates of the target image into actual space coordinates, determine the visual position of the object to be sorted, locate the position information of the sound-emitting element on the object to be sorted through the microphone array, and use it as the auditory position; after summarizing the auditory position and visual position data of the object to be sorted, generate a position data set of the object to be sorted;

[0080] Step 302: Generate a deviation value from the position data of the items to be sorted , verify the overlap between the auditory position and the visual position according to the deviation value, wherein, after obtaining the distance between the auditory position and the visual position, it is used as the deviation distance. When under dimensionless conditions, the deviation value is generated according to the following formula :

[0081] ;

[0082] in, For the The deviation distance of each time node, is the mean of the deviation distances, is the weight coefficient, ; is the number of time nodes, is the acceptance value of the deviation distance;

[0083] Pre-set deviation thresholds based on historical data and expected deviations in location identification;

[0084] If the deviation value If it is higher than the deviation threshold, it means that the current auditory positioning and visual positioning may be inaccurate, and the overlap between the two is lower than expected. It is necessary to correct the real-time position of the items to be sorted, or introduce a new positioning method. At this time, a position correction instruction is sent to the outside.

[0085] When in use, after adding a sound element to each item to be sorted, when the items to be sorted are in the running state on the conveyor belt, the sound element is also in the sounding state, and the items to be sorted are quickly located based on the sound signal emitted. At the same time, after imaging the items to be sorted, the deviation value is constructed based on the degree of deviation between positioning. , judge the degree of deviation between different positioning, and thus judge the reliability of the current position detection data.

[0086] Step 303: After receiving the position correction instruction, the position data of the items to be sorted are corrected, wherein:

[0087] After selecting an anchor in the sorting area, the distance measuring radar is used to measure the distance between the anchor and the items to be sorted. The distance data obtained is used to determine the location data of the items to be sorted through the three-sided positioning algorithm, which is used as the radar position. After summarizing the auditory position, visual position and radar position,

[0088] If there is overlap between the three position data, the overlapping position is used as the real-time position of the item to be sorted;

[0089] If the three position data are collinear, the middle point of the collinearity is used as the real-time position of the item to be sorted;

[0090] If the three position data are not collinear or overlapping, connect the three positions and use the center of the enclosed area as the real-time position of the item to be sorted;

[0091] When using, combine the contents in steps 301 to 303:

[0092] When the degree of deviation between different positioning methods exceeds expectations, a third positioning method is introduced for the current items to be sorted, and the items to be sorted are repositioned. The previous positioning data is corrected and adjusted based on the new position data obtained. By setting a positioning correction mechanism, the positioning correction mechanism is triggered when the reliability of the existing positioning method is reduced, and the ranging radar that is not in a continuous operation state is enabled to improve the accuracy of subsequent sorting when sorting the items to be sorted.

[0093] Step 4: Generate a sorting value from the information of the items to be sorted, mark the items to be sorted with the sorting value and the real-time position, and then plan a moving path for the sorting robot to move to the items to be sorted;

[0094] The step 4 includes the following contents:

[0095] Step 401: The image acquisition device acquires multi-angle image data of the items to be sorted in the sorting area, and uses a text detection algorithm to detect the text area in the image. If there is a text area, the OCR algorithm is used to perform character recognition and extract the corresponding text information;

[0096] Identify the type of items to be sorted based on text information, extract description information, material specification data, etc. of the items to be sorted, and generate an information set of the items to be sorted;

[0097] Step 402: train the convolutional neural network with the labeled sample data to obtain a trained sorting product evaluation model; use the information of the items to be sorted as input, use the trained sorting product evaluation model to score, obtain the corresponding sorting value, and determine the sorting priority of the items to be sorted according to the size of the sorting value;

[0098] Mark the items to be sorted with the sorting value and real-time position, use the tracking algorithm to track the items to be sorted in real time, update the item position in real time and record its movement trajectory;

[0099] When using it, considering that the importance of different items to be sorted may be different, the sorting order may be different. When there may be various text descriptions outside the items to be sorted, the relevant information of the items to be sorted is obtained through text information extraction and semantic recognition, and the importance and priority of the items to be sorted are evaluated. According to the sorting priority, the corresponding sorting order can be determined in the subsequent sorting process;

[0100] Step 403: Use laser radar, ultrasonic sensor, etc. to scan and detect the sorting area, obtain terrain information in the sorting area, build an electronic map covering the sorting area based on the scanned detection data, and after obstacles located in the sorting area are determined, mark the obstacles on the electronic map;

[0101] According to the obstacles in the sorting area and the sorting values, real-time location information and predicted location information of the items to be sorted, the path planning algorithm plans the moving path for the sorting robot to move to the items to be sorted, ensuring that the robot reaches the target location smoothly and displays the moving path on the electronic map;

[0102] When using, combine the contents in steps 401 to 403:

[0103] Considering that the items to be sorted are in a moving state and the area to be delivered may be a certain distance away from the current location, in order to increase the sorting speed, it is necessary to plan the moving path for the sorting robot and adjust the moving path of the sorting robot so that the robotic arm of the sorting robot can quickly grab and sort the items to be sorted.

[0104] See also Figure 3 The present invention provides a sorting system of a multimodal mobile sorting robot, comprising the following contents:

[0105] The grasping path planning unit identifies the posture of the end effector of the robotic arm of the sorting robot, such as position and direction, before the sorting robot is about to reach the target position. After using the RRT or RRT* algorithm to plan the grasping path of the robotic arm, the grasping path is smoothed by the BSpline algorithm to obtain the grasping path of the object to be sorted, which can improve the stability of the robotic arm movement.

[0106] The path testing unit collects the specification performance data, operation data and movement data of the sorting robot, etc., trains the machine learning algorithm and generates a digital twin model of the robot arm sorting; takes the grasping path as input, and uses the robot arm sorting digital twin model to verify whether the grasping path is feasible. If not, the grasping path is optimized. If feasible, a grasping instruction is issued through the control interface of the robot arm;

[0107] During use, after the sorting robot moves to the target position, considering the moving state and possible arrival position of the items to be sorted, in order to improve the accuracy of grasping, the motion state and motion path of the robotic arm are planned, which can improve the effectiveness of the grasping path.

[0108] After receiving the grasping instruction, the sorting control unit controls the end effector of the robot arm, such as a gripper or a suction cup, to grasp the items to be sorted and complete the grasping process;

[0109] The placement unit determines the placement point of the items to be sorted according to the pre-set rules or task requirements, and then places the items to be sorted in the designated disposal area according to the item identification results and the pre-set classification rules, thus completing the sorting process of the items to be sorted;

[0110] When in use, after the grasping path of the robotic arm has been obtained, a simulation test is performed on the grasping path of the robotic arm to determine the feasibility of the grasping path based on the simulation test data. When the feasibility is insufficient, the current grasping path is optimized based on the simulation test data. On the basis of planning the moving path of the sorting robot, the efficiency of grasping and sorting is further improved.

[0111] In combination with the above content, when using a sorting robot to sort items under scenario conditions with poor lighting conditions in the sorting area, by introducing sound elements as a supplement and using multimodal data to identify and locate the items to be sorted, the efficiency and reliability of sorting can be improved. Moreover, by determining the sorting priority of each item to be sorted, important items can be sorted and grabbed first, thereby improving the sorting efficiency as a whole.

[0112] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0114] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only some logical function divisions. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0115] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A navigation method for a multimodal mobile sorting robot, characterized in that: include, When the light fluctuation in the sorting area exceeds expectations, the image acquisition mechanism is triggered before the items to be sorted arrive at the predetermined location; the advance acquisition time of the item image data is constrained according to the light fluctuation value and the position change of the items to be sorted. , the constraints are as follows: ； Weight coefficient, , ; is the number of time nodes, Is the item to be sorted Time point The distance between time nodes, is the average distance, Is the item to be sorted Time point The difference in change value between time nodes, is the average value of light fluctuation; The collected auditory data and visual data are fused to extract multimodal features, and the items to be sorted are identified based on the multimodal features; after the items to be sorted are located using the visual data and auditory data respectively, if the deviation between the two position data exceeds the expectation, the position data of the items to be sorted is corrected; Generate a sorting value from the information of the items to be sorted, mark the items to be sorted with the sorting value and real-time position, and then plan a moving path for the sorting robot to move to the items to be sorted; The photoelectric sensors located at various positions in the sorting area collect light data, summarize and generate a light data set in the sorting area, and generate light fluctuation values ​​based on the changing state of the light data. , as follows: ； Where: is the total number of photoelectric sensors, is the total number of time nodes, For time node Previous The degree of a photoelectric sensor; When the items to be sorted with the sound-emitting element attached are in a continuous moving state on the conveyor belt, the advance collection time of the item image data is constrained according to the light fluctuation value and the position change constraints of the items to be sorted. , before arriving at the predetermined position, the image acquisition mechanism is triggered and an image acquisition instruction is issued to the outside; After receiving the image acquisition instruction, the image acquisition device collects the visual data of the items to be sorted on the conveyor belt, collects the auditory data in the sorting area through the microphone array, and aligns the data through timestamps and registration; Combining the different modal features extracted from the aligned data to form multimodal features; The multimodal features of the items to be sorted are used as input, and the trained model for identifying the items to be sorted is used for identification. If the items to be sorted contained therein are identified, they are used as the target image and the identification data is obtained.

2. The navigation method of the multimodal mobile sorting robot according to claim 1, characterized in that: The coordinates of the target image are converted into actual space coordinates to determine the visual position of the items to be sorted. The position information of the sound-emitting elements on the items to be sorted is located through the microphone array and used as the auditory position. The auditory position and visual position data of the items to be sorted are aggregated to generate a position data set of the items to be sorted.

3. The navigation method of the multimodal mobile sorting robot according to claim 2, characterized in that: The deviation value is generated by the position data in the position data set of the items to be sorted. If the deviation value is higher than the deviation threshold, a position correction instruction is issued to the outside. After receiving the position correction instruction, a supplementary positioning method is introduced to correct the position data of the items to be sorted. Among them, after obtaining the distance between the auditory position and the visual position, it is used as the deviation distance. When under dimensionless conditions, the deviation value is generated according to the following formula : ； in, For the The deviation distance of each time node, is the mean of the deviation distances, is the weight coefficient, ; is the number of time nodes, is the accepted value for the deviation distance.

4. The navigation method of the multimodal mobile sorting robot according to claim 3, characterized in that: The image acquisition device collects multi-angle image data of the items to be sorted in the sorting area, uses a text detection algorithm to detect the text area in the image, and uses an OCR algorithm to perform character recognition to extract the corresponding text information; Identify the type of items to be sorted based on text information, extract description information and material specification data of the items to be sorted, and generate an information set of the items to be sorted.

5. The navigation method of the multimodal mobile sorting robot according to claim 4, characterized in that: The information of the items to be sorted is used as input, and the trained sorting evaluation model is used to score them, and the corresponding sorting value is obtained. The sorting priority of the items to be sorted is determined according to the size of the sorting value; the tracking algorithm is used to track the items to be sorted in real time, update the location of the items in real time and record their movement trajectory.

6. The navigation method of the multimodal mobile sorting robot according to claim 5, characterized in that: An electronic map covering the sorting area is established and obstacles are marked on the electronic map. Based on the obstacles in the sorting area and the sorting values, real-time location information and predicted location information of the items to be sorted, a path planning algorithm is used to plan a moving path for the sorting robot to move to the items to be sorted, and the moving path is displayed on the electronic map.

7. A sorting system of a multimodal mobile sorting robot, applying the navigation method according to any one of claims 1 to 6, characterized in that: include: The grasping path planning unit identifies the posture of the end effector of the robotic arm of the sorting robot, plans the grasping path of the robotic arm using the RRT or RRT* algorithm, and then smoothes the grasping path using the BSpline algorithm; The path testing unit takes the grasping path as input and uses the robot arm sorting digital twin model to verify whether the grasping path is feasible. If not, the grasping path is optimized. If feasible, a grasping instruction is issued; The sorting control unit controls the end effector of the robot arm to grab the items to be sorted after receiving the grabbing instruction; The placement unit determines the placement point of the items to be sorted according to the preset rules or task requirements, and then places the items to be sorted in the designated disposal area according to the item identification results and preset classification rules.

Images