Assembly line goods sorting method based on imitation learning

Abstract

The invention belongs to the field of machine vision and deep learning, and discloses an assembly line goods sorting method based on imitation learning, which comprises the following steps: recording a mapping video, a clamping jaw motion video and a task data video; performing three-dimensional reconstruction according to the high-quality mapping video to obtain a feature map, extracting an initial task trajectory in combination with the feature map and the task data video, and optimizing and screening the initial task trajectory to output a final task trajectory; task RGB images are extracted according to task data videos, clamping force of the handheld clamp is collected according to a force sensor on the handheld clamp, the finger width of the handheld clamp is extracted according to the task RGB images, and the handheld clamp position and the handheld clamp angle contained in a final task track are combined to serve as a training data set; and a visual model is trained through an imitation learning method, and the trained visual model is used for being deployed to a mechanical arm to complete an assembly line goods sorting task. According to the method, the convenience of data acquisition and the generalization of the task environment are both considered.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision and deep learning, and specifically relates to an assembly line cargo sorting method based on imitation learning. Background Art

[0002] In modern manufacturing and logistics industries, cargo sorting is a key link in the assembly line. The initial cargo sorting tasks often use traditional manual sorting, hiring a group of skilled workers to distinguish and sort the products. However, with the continuous expansion of production scale and the increasing demand for logistics, the traditional manual sorting method can no longer meet the needs of modern industry for high-efficiency, high-precision and high-flexibility sorting. Therefore, the automated sorting system came into being and became an inevitable trend. Manufacturing companies often achieve efficient sorting of goods by building automated systems. However, on complex and changeable production lines, there are many types of goods and different shapes, and the accuracy and efficiency of the sorting process still face huge challenges.

[0003] Initially, the automated sorting system used mechanical structures to implement the sorting function. By setting the shape and size of the slots, the goods of different shapes on the assembly line were screened and placed in different partitions. However, this screening method was powerless for objects with similar appearances, and this simple and direct method was not environmentally adaptable. It is also a common method to use targeted sensing and detection equipment to classify and identify different objects. Specific objects can be targetedly distinguished by features such as shape and appearance. For example, a color sensor that detects the amount of light received by red, blue, and green to determine the color of the object can be used to distinguish objects of different colors. However, this requires the use of specific sensors for specific object recognition and sorting tasks, which leads to an increase in procurement and maintenance costs. With the rapid development of machine vision and deep learning technologies, vision-based target recognition and positioning methods have brought new solutions to production line cargo sorting systems. Target detection algorithms such as YOLO (You Only Look Once) have achieved remarkable results in the field of image recognition and can accurately identify and locate targets. However, such methods often rely on large labeled data sets for training. Especially when there are many types of goods with different shapes, each type of goods needs to be accurately labeled, which undoubtedly greatly increases the complexity of the work and labor costs. Therefore, how to ensure the accuracy and efficiency of the sorting system becomes a major challenge.

[0004] Existing technologies, such as the patent application with publication number CN118403810A, propose a sorting and classification system on a silicone tube production line, which detects the distance and position information of the silicone tube through ultrasonic sensors, infrared sensors and optical sensors, then uses piezoresistive sensors to detect the defect holes of the silicone tube, and then uses photoelectric sensors to detect the shape of the silicone tube. Finally, the temperature sensor is used to detect the temperature of the silicone tube, and the detailed parameter information of the silicone tube is obtained, and the silicone tube is classified and sorted. Although this patent improves the accuracy of silicone tube classification detection by using multiple sensors to obtain detailed and complete parameter information, the use of multiple sensor modules in series makes the entire system complicated and cumbersome, and has high requirements for infrared equipment. If hardware failure occurs, the corresponding maintenance and replacement costs are also high.

[0005] Another example is the patent application with publication number CN118080374A, which proposes a feeding line for multiple materials including a loading device, a sorting device and a unloading device. The type of small packages of snacks is identified by a visual identifier, and the identification results are handed over to a robotic arm for sorting and packaging. This patent requires the collection and labeling of a large amount of data for each type of item to be identified, which greatly increases the complexity of the work and labor costs. In addition, a large amount of data needs to be collected and labeled or complex and tedious calculations need to be performed, which has certain limitations in terms of real-time performance and computing resources. In addition, there are problems such as the need for supporting hardware devices and low environmental generalization.

[0006] It can be seen that although there are many types of technologies currently used in assembly line cargo sorting, and many solutions have introduced intelligent and automated technologies, there are still many problems. Summary of the invention

[0007] The purpose of the present invention is to provide a method for sorting goods on an assembly line based on imitation learning, taking into account the convenience of data collection and the generalization of the task environment.

[0008] To achieve the above object, the technical solution adopted by the present invention is: A method for sorting goods on an assembly line based on imitation learning, the method comprising: Data collection: Use a handheld gripper equipped with multimodal information collection equipment to record mapping videos, gripper motion videos, and task data videos; Data cleaning: Select high-quality mapping videos from the recorded mapping videos, perform 3D reconstruction based on the high-quality mapping videos to obtain feature maps, extract the initial task trajectory in a unified coordinate system by combining the feature maps and task data videos, optimize and select the initial task trajectory, and output the final task trajectory; Model training: The task RGB image is extracted according to the task data video, the clamping force of the handheld clamp is collected according to the force sensor on the handheld clamp, and the finger width of the handheld clamp is extracted according to the task RGB image. The task RGB image, clamping force, finger width of the handheld clamp, and the handheld clamp position and handheld clamp angle contained in the final task trajectory are used as training data sets. The imitation learning method is used to train the visual model. The trained visual model is used to deploy to the robotic arm to complete the assembly line cargo sorting task.

[0009] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.

[0010] Preferably, the mapping video is a video of the multimodal information collection device scanning the mapping scene, the gripper movement video is a video of the multimodal information collection device scanning the handheld clamp to control the hand-designated point opening and closing movements, and the task data video is a video of the multimodal information collection device scanning the handheld clamp to perform an assembly line cargo sorting task.

[0011] Preferably, there are multiple recorded mapping videos, and the step of selecting high-quality mapping videos from the recorded mapping videos includes: In the formula, is the variance value of the image in the mapping video, is the total number of pixels in the image, For the image pixels, For the The coordinates of the pixels, For the The gray value of a pixel, is the average gray value of the entire image; The single-frame variance value is calculated for each frame in the mapping video, and the variance values ​​of all frames are accumulated and averaged to obtain the average total variance of a single mapping video. The mapping video with the smallest average total variance is selected as the screened high-quality mapping video.

[0012] Preferably, a Savitzky-Golay filter is used to optimize the initial task trajectory.

[0013] Preferably, the initial task trajectory includes multiple task trajectories, and the screening of the initial task trajectory includes: if the height difference between two lowest points in a task trajectory is greater than a threshold, the current task trajectory is discarded, otherwise the current task trajectory is retained.

[0014] Preferably, the step of extracting the finger width of the handheld fixture according to the task RGB image comprises: Detect the two QR codes on the left and right fingers of the handheld fixture in the RGB image and the four vertices corresponding to the QR codes; Based on the detected vertices and side lengths of the QR code, the rotation matrix and translation matrix of the QR code on the left and right fingers are obtained; The difference between the translation matrices of the left and right fingers is calculated as the finger width of the handheld gripper.

[0015] Preferably, the method of training the visual model by using an imitation learning method comprises: Obtain encodable reasoning features based on the training dataset; Adding Gaussian noise to the codable inference feature and inputting it into the visual model to obtain a predicted action increment, wherein the action increment includes a handheld clamp posture increment, a handheld clamp finger width increment, and a clamping force feedback value; A mean square error loss value is calculated using Gaussian noise as a label and an action increment as an estimated value, and the visual model is updated according to the mean square error loss value.

[0016] Preferably, the step of obtaining the encodable inference feature based on the training data set comprises: Input the task RGB image into the visual encoder to obtain the task features; The task features, gripping force, hand-held gripper position, hand-held gripper angle, and hand-held gripper finger width are spliced ​​into encodable reasoning features.

[0017] The present invention provides an assembly line cargo sorting method based on imitation learning, which improves the efficiency and effect of task completion under the premise of simplifying the data collection and model training processes, and takes into account the convenience of data collection and the generalization of task objectives: the trained model can focus on a specific area in the image, that is, the target object, rather than the background, and has a certain generalization ability for objects of similar shapes and different sizes, and different assembly lines. After adding a new category, it only needs to be fine-tuned by supplementing new data (without labeling); compared with the traditional method based on mechanical structure - the slot needs to be replaced according to different sizes, the method based on specific sensors - the hardware equipment needs to be replaced after the same size, shape and material change, and the method based on traditional neural network training - new data needs to be collected and labeled after adding a new category, it has more generalization ability and convenience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a method for sorting goods on an assembly line based on imitation learning of the present invention; Figure 2 It is a data collection flow chart of the present invention; Figure 3It is a data cleaning flow chart of the present invention; Figure 4 This is a flow chart of model training of the present invention; Figure 5 Schematic diagram of the hardware structure built for the model deployment stage of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0021] The present invention aims to solve the problem of complicated and cumbersome existing methods for sorting goods on assembly lines. Specifically, it aims to solve the problems of overly cumbersome and complicated data collection and model training processes, task environment restrictions, and target object limitations in sorting goods on assembly lines. The present invention adopts an improved imitation learning model to train a deep learning model suitable for sorting goods on assembly lines. On the premise of simplifying the data collection process, the task completion accuracy and model training effect are improved by adding a data cleaning process, taking into account the convenience of data collection and the generalization of the task environment.

[0022] like Figure 1 As shown, this embodiment provides an assembly line cargo sorting method based on imitation learning, which is mainly divided into three stages: data collection, data cleaning and model training.

[0023] (I) During the data collection phase, a handheld fixture equipped with multimodal information collection equipment is used to record mapping videos, gripper motion videos, and task data videos. The multimodal information collection equipment includes but is not limited to RGBD cameras, binocular cameras, force sensors, monocular cameras, IMUs, laser radars, and other equipment; the handheld fixture is used as a task executor. Figure 2 As shown, the specific steps include:

[0024] Recording mapping video: The mapping video is a video of the image acquisition device in the multimodal information collection device scanning the mapping scene. The recording process provided in this embodiment is: after connecting the RGBD camera to the computer, turn on the recording function, and use the handheld clamp to perform the specified action to scan and map. There are three task actions to be performed. First, move horizontally to scan the entire mapping scene, then move vertically forward and backward several times, and finally the handheld clamp moves in a circle around the mapping scene. The entire acquisition process needs to scan the mapping scene as comprehensively and carefully as possible to obtain a complete scene effect. In the assembly line cargo sorting training scenario, it is necessary to scan the assembly line conveying device involved to obtain the best mapping effect. For example, a conveyor belt on the production line is mapped. The recording time for each recording is about 2 minutes, and the recording is repeated many times to ensure the recording effect, and the task data video is recorded on the conveyor belt.

[0025] Recording gripper motion video: The gripper motion video is a video of the image acquisition device in the multimodal information collection device scanning the handheld gripper to control the finger to open and close at a fixed point. The recording process provided in this embodiment is: after connecting the RGBD camera to the computer, turn on the recording function, and during recording, the handheld gripper is used to control the finger to open and close at a fixed point five times.

[0026] Recording task data video (task video): The task data video is a video of the image acquisition device in the multimodal information collection device scanning the handheld clamp to perform the assembly line cargo sorting task. The recording process provided in this embodiment is: after connecting the RGBD camera to the computer, turn on the recording function. During the recording process, the clamp should be kept running smoothly as much as possible, and more than 200 videos are recorded for a single task. For the assembly line cargo sorting task in industrial production, the target task is set to sort three different items on the production line and organize them into their corresponding classification boxes. In the process of recording the task video, it is necessary to complete the action of "using the collection handheld clamp to grab a certain part into the corresponding box" in each round. When grabbing, the grabbing point should also be made to fall as much as possible at the middle point of the object to improve the stability of the grabbing and the accuracy of the subsequent task screening. In order to improve the adaptability of the model and the success rate of the task, it is also necessary to make each clamping task start from different positions and different initial angles as much as possible. After completing the recording of two hundred valid videos, the target task object is replaced and the next round of recording is performed. A total of 600 task videos were recorded in three rounds for three types of objects, and a task trajectory was extracted from each video.

[0027] In the data collection stage, this embodiment provides a simple and easy-to-operate data collection method, which only requires recording three types of videos with concise instructions: mapping video, grippers video, and task video, thereby lowering the threshold for hardware use and data collection.

[0028] (II) Data cleaning stage: High-quality mapping videos are selected from the recorded mapping videos, and 3D reconstruction is performed based on the high-quality mapping videos to obtain feature maps. The initial task trajectory in a unified coordinate system is extracted by combining the feature maps and the task data videos, and the initial task trajectory is optimized and selected to output the final task trajectory. Figure 3 As shown, the specific steps include:

[0029] (1) Mapping video screening: The mapping quality is quantitatively evaluated based on the image clarity index, so as to select the best one from multiple mapping videos to obtain the best mapping effect in the future. In the quantitative evaluation process, a clearly focused image has a greater grayscale difference than a blurred image, so this embodiment uses the variance function As a clarity evaluation index, the smaller the function is, the higher the image clarity is. The calculation formula is as follows:

[0030] In the formula, is the variance value of the image in the mapping video, is the total number of pixels in the image, For the image pixels, For the The coordinates of the pixels, For the The gray value of a pixel, is the average gray value of the entire image.

[0031] The single-frame variance value is calculated for each frame in the mapping video, and the variance values ​​of all frames are accumulated and averaged to obtain the average total variance of a single mapping video. The mapping video with the smallest average total variance is selected as the screened high-quality mapping video.

[0032] (2) Mapping based on mapping video: Use the selected high-quality mapping video to extract the RGB image and depth data, perform 3D reconstruction based on the ORB-SLAM3 system, and save the feature map.

[0033] (3) Task trajectory optimization: Process the task data video into a task RGB image, and combine the feature map to run the trajectory estimation algorithm to obtain the initial task trajectory of the handheld fixture in a unified coordinate system for visualization and model training. The trajectory estimation algorithm includes but is not limited to the visual SLAM algorithm, the laser SLAM algorithm, etc. This embodiment uses the ORB-SLAM3 algorithm. In view of abnormal conditions such as glitches and noise interference that may appear in the trajectory, the Savitzky-Golay filter is used to smooth the trajectory curve. The SG filter is actually a weighted average algorithm of a moving window. The weighted coefficient obtained by least squares fitting a given high-order polynomial in a sliding window is smoothed to optimize the trajectory curve quality.

[0034] (4) Task data screening: After completing the task trajectory quality optimization, it is also necessary to eliminate the invalid data that may exist. Specifically, consider the two lowest points in each task trajectory (the lowest position height of the first time the object is clamped on the conveyor belt) and the lowest position height when placing objects in the sorting box on the same plane as the conveyor belt ) height difference It should be less than a threshold, otherwise the estimated error of the task trajectory is considered too large and invalid, and the final task trajectory is obtained.

[0035] This embodiment records the motion information of the end of the handheld clamp used for collection, visualizes the motion trajectory through a trajectory estimation algorithm, and estimates its offset, continuity, whether there are singular points and other properties, and eliminates invalid data to ensure the accuracy, completeness, consistency and reliability of the task data.

[0036] (III) Model training phase: Extract the task RGB image based on the task data video, collect the gripping force of the handheld gripper based on the force sensor on the handheld gripper, extract the finger width of the handheld gripper based on the task RGB image, use the task RGB image, gripping force, finger width of the handheld gripper, and the handheld gripper position and handheld gripper angle contained in the final task trajectory as the training data set, use the imitation learning method to train the visual model, and the trained visual model is used to deploy to the robotic arm to complete the assembly line cargo sorting task. Figure 4 As shown, the specific steps include:

[0037] (1) Generate a training data set: The training data set should contain data in five dimensions: task RGB image, force magnitude (gripping force fed back by the force sensor), eef_pose (handheld gripper position), eef_rot_axis_angle (handheld gripper angle), and gripper_width (handheld gripper finger width).

[0038] The task RGB image is extracted from the task data video and is a frame in the task data video.

[0039] The finger width of the handheld gripper is extracted based on the task RGB image. Specifically, the task RGB image is input into OpenCV In the function, The function will detect the two QR codes on the left and right fingers in the image and the four vertices corresponding to the QR codes, and input the detected vertices and the length of the QR code into OpenCV respectively. In the function, the rotation matrix and translation matrix of the QR code on the left and right fingers are obtained. The difference between the translation matrices of the two fingers is the finger width of the handheld fixture.

[0040] Similarly, the gripper motion video is processed into a gripper motion RGB image, and the finger width is extracted from the gripper motion RGB image in the same way as the handheld gripper finger width extraction method as the maximum and minimum widths for subsequent standardization of the handheld gripper finger width.

[0041] The hand-held gripper position and hand-held gripper angle are included in the extracted final task trajectory.

[0042] The data of five dimensions are taken as a training sample, the collection time of the data of five dimensions in the same training sample corresponds, and the training data set contains multiple training samples.

[0043] (2) Feature extraction and integration: After standardizing the data (gripping force, hand-held gripper position, hand-held gripper angle, and hand-held gripper finger width), the task RGB image data is fed into vit_base_patch16_clip_224.openai visual encoder to extract task feature features, and the task feature features, gripping force, hand-held gripper position, hand-held gripper angle, and hand-held gripper finger width are concatenated and integrated into encodable reasoning features. Since 1) the clip-vit model is pre-trained on a large number of images, it can capture a wide range of patterns and knowledge and has better generalization capabilities when dealing with diverse tasks; 2) the clip-vit architecture has powerful modeling capabilities, especially when dealing with complex and fine-grained visual tasks. By using the Transformer architecture, clip-vit can better capture long-term dependencies and contextual information in the image, adjust the attention weights during training, and gradually learn to focus on areas related to the target object while ignoring unimportant background information, thereby improving the generalization ability of the scene.

[0044] (3) Generate predicted action increment action: Add the codable inference feature to Gaussian noise noise_trajectory and then input it into the ConditionalUnet1D model to obtain the predicted action increment action required to control the movement of the robot arm, including the handheld fixture posture increment (handheld fixture position increment and handheld fixture angle increment), handheld fixture finger width increment and gripping force feedback value. The training method includes but is not limited to diffusion strategy, reinforcement learning (DDPG, etc.), VLA, etc. This embodiment adopts diffusion strategy as the imitation learning method.

[0045] (4) Model update: Use Gaussian noise as the label and the predicted action increment as the estimated value to calculate the mean square error loss value, and update the ConditionalUnet1D model based on the mean square error loss value.

[0046] The present invention solves the problem of robot arm task execution for assembly line work. The advantages of using imitation learning method to train visual models include being not restricted by robot arm hardware. The trained visual model can be deployed to any robot arm to complete specified tasks (such as gripping, moving, cutting, etc.). During the experiment, the present invention used a self-developed robot arm to successfully complete the test of sorting and arranging tasks for three different items. I will not go into details and limit the hardware here.

[0047] like Figure 5 As shown, after the visual model training is completed, the hardware structure built in the model deployment phase may include a robotic arm, a gripper, a gripper base, gripper fingers, a personal computer, a communication device, and a multimodal information collection device.

[0048] The robotic arm is used to complete task movement and is applicable to all six-degree-of-freedom robotic arms. The posture increment in the motion increment will be superimposed on the current posture of the robotic arm to obtain the absolute posture, that is, the target point that the robotic arm will reach at the next moment; it communicates remotely with the personal computer through the TCP / IP protocol, receives the posture increment in the motion increment, and uploads the robotic arm posture to the personal computer.

[0049] The gripper is used to control the gripper fingers to complete the gripping action according to the finger width increment and the gripping force feedback value. It is applicable to all grippers whose gripping width is greater than the width of the target object.

[0050] Gripper base, used to connect the robot arm to the gripper and install the vision sensor.

[0051] The gripper finger is used to complete the gripping task. Other tools can also be installed on the finger to adapt to different task requirements. For example, a blade can be installed on the finger to perform cutting tasks. The handheld gripper in the model training is equivalent to the sum of the gripper, gripper base, and gripper finger in the deployment stage. Combined with the camera that collects image data, it is collectively called a camera gripper. It communicates remotely with the personal computer through the TCP / IP protocol, receives the finger width increment and gripping force feedback value in the action increment, and uploads the camera image, finger width, and gripping force collected by the force sensor to the personal computer.

[0052] A personal computer (PC) is used to deploy and run the imitation learning model framework obtained through training to complete the task; in other embodiments, the personal computer can also be replaced by a development board, which is used to run the relevant models in the method through the basic code framework and output action increments.

[0053] Communication equipment, used to establish remote communication between the personal computer and the robotic arm. Remote communication methods include but are not limited to: hardware: WIFI, Bluetooth, wired, serial port, etc.; software protocol: Socket, I2C, UDP / TCP, etc.

[0054] Multimodal information collection equipment, including but not limited to RGBD cameras, force sensors and other equipment.

[0055] Some of the existing assembly line cargo sorting solutions are based on direct sorting based on mechanical structures. They can only perform unconscious sorting for specific assembly lines, with a narrow scope of application and low fault tolerance. Some solutions are based on specific sensor detection equipment for classification and identification, which has high hardware requirements, and also leads to rising costs and thus compressing profits. Some solutions are based on traditional image processing and neural network training methods to achieve target detection and key point positioning. These methods often require the collection of a large number of samples for labeling. If the target object is changed, it needs to be re-labeled, which brings a lot of extra workload. In addition, the identification, judgment, control and other links lack relevance, and it is often necessary to adjust multiple links one by one when the task changes.

[0056] The present invention proposes to use the imitation learning method to train the model, which improves the efficiency and effect of task completion under the premise of simplifying the data collection and model training process, and takes into account the convenience of data collection and the generalization of task objectives: the trained model can focus on a specific area in the image, that is, the target object, rather than the background, and has a certain generalization ability for objects of similar shapes and different sizes, and different assembly lines. After adding new categories, it only needs to be fine-tuned by adding new data (without labeling); compared with the traditional method based on mechanical structure - the slot needs to be replaced according to different sizes, the method based on specific sensors - the hardware equipment needs to be replaced after the same size, shape and material change, and the method based on traditional neural network training - new data needs to be collected and labeled after adding new categories, it has more generalization ability and convenience.

[0057] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0058] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A method for sorting goods on an assembly line based on imitation learning, characterized in that: The assembly line cargo sorting method based on imitation learning comprises: Data collection: Use a handheld gripper equipped with multimodal information collection equipment to record mapping videos, gripper motion videos, and task data videos; Data cleaning: Select high-quality mapping videos from the recorded mapping videos, perform 3D reconstruction based on the high-quality mapping videos to obtain feature maps, extract the initial task trajectory in a unified coordinate system by combining the feature maps and task data videos, optimize and select the initial task trajectory, and output the final task trajectory; Model training: The task RGB image is extracted according to the task data video, the clamping force of the handheld clamp is collected according to the force sensor on the handheld clamp, and the finger width of the handheld clamp is extracted according to the task RGB image. The task RGB image, clamping force, finger width of the handheld clamp, and the handheld clamp position and handheld clamp angle contained in the final task trajectory are used as training data sets. The imitation learning method is used to train the visual model. The trained visual model is used to deploy to the robotic arm to complete the assembly line cargo sorting task.

2. The assembly line cargo sorting method based on imitation learning according to claim 1 is characterized in that: The mapping video is a video of the multimodal information collection device scanning the mapping scene, the gripper movement video is a video of the multimodal information collection device scanning the handheld clamp to control the hand-specified point opening and closing movements, and the task data video is a video of the multimodal information collection device scanning the handheld clamp to perform the assembly line cargo sorting task.

3. The assembly line cargo sorting method based on imitation learning according to claim 1 is characterized in that: There are multiple recorded map building videos, and selecting high-quality map building videos from the recorded map building videos includes: In the formula, is the variance value of the image in the mapping video, is the total number of pixels in the image, For the image pixels, For the The coordinates of the pixels, For the The gray value of a pixel, is the average gray value of the entire image; The single-frame variance value is calculated for each frame in the mapping video, and the variance values ​​of all frames are accumulated and averaged to obtain the average total variance of a single mapping video. The mapping video with the smallest average total variance is selected as the screened high-quality mapping video.

4. The assembly line cargo sorting method based on imitation learning according to claim 1 is characterized in that: The Savitzky-Golay filter is used to optimize the initial mission trajectory.

5. The assembly line cargo sorting method based on imitation learning according to claim 1 is characterized in that: The initial task trajectory includes multiple task trajectories, and the screening of the initial task trajectory includes: if the height difference between two lowest points in a task trajectory is greater than a threshold, the current task trajectory is discarded, otherwise the current task trajectory is retained.

6. The assembly line cargo sorting method based on imitation learning according to claim 1 is characterized in that: The method of extracting the finger width of the handheld fixture according to the task RGB image includes: Detect the two QR codes on the left and right fingers of the handheld fixture in the RGB image and the four vertices corresponding to the QR codes; Based on the detected vertices and side lengths of the QR code, the rotation matrix and translation matrix of the QR code on the left and right fingers are obtained; The difference between the translation matrices of the left and right fingers is calculated as the finger width of the handheld gripper.

7. The assembly line cargo sorting method based on imitation learning according to claim 1 is characterized in that: The imitation learning method is used to train the visual model, including: Obtain encodable reasoning features based on the training dataset; Adding Gaussian noise to the codable inference feature and inputting it into the visual model to obtain a predicted action increment, wherein the action increment includes a handheld clamp posture increment, a handheld clamp finger width increment, and a clamping force feedback value; A mean square error loss value is calculated using Gaussian noise as a label and an action increment as an estimated value, and the visual model is updated according to the mean square error loss value.

8. The assembly line cargo sorting method based on imitation learning according to claim 7 is characterized in that: The method of obtaining the encodable reasoning features based on the training data set includes: Input the task RGB image into the visual encoder to obtain the task features; The task features, gripping force, hand-held gripper position, hand-held gripper angle, and hand-held gripper finger width are spliced ​​into encodable reasoning features.

Images