Multi-sensor robot sensing and positioning system for sorting precise fine-grained objects

Abstract

The system of the present invention provides a multi-sensor robotic sensing and positioning system for sorting precise fine-grained objects. According to the system, multiple sensors are used for carrying out area proposal, instance segmentation, fine-grained classification and 3D pickup position calculation on shot images containing objects to be sorted, the objects are recognized and classified through the artificial intelligence technology, and the objects are accurately placed at proper positions, so that the sorting accuracy is improved, and the sorting efficiency is improved. The automatic sorting device is suitable for being applied to the environment where various kinds of objects need to be sorted, the sorting efficiency is greatly improved, the time needed for completing tasks is shortened, and the number of workers needed for work is also reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to an intelligent sorting system, in particular to a robot perception and positioning system for sorting precision fine-grained objects. BACKGROUND

[0002] The sorting application of robots has been widely and maturely applied in traditional controlled spaces, such as pipelines with basically unchanged environment and target objects. However, with the increasingly wide application of HMLV (High-mix low-volume manufacturing) production mode, the traditional robot sorting application cannot meet the needs of HMLV. HMLV usually produces small batches of products with a wide variety of products, and is used to manufacture unique and more complex products with specific quality requirements. Compared with traditional mass production (low variety, high yield), HMLV manufacturing is not a constant or long-time maintaining environment, but an environment full of many variables. HMLV needs to change articles, materials and machines more frequently, which makes the operating personnel have to constantly adjust the work and material scheduling, and the traditional sorting robot cannot adapt to so many changes, cannot preset so many different products in advance, and the sorting performance is significantly reduced. In addition, a variety of raw materials may also block each other, so that the traditional robot grasping posture may be misplaced due to the proximity of objects, and therefore cannot meet the needs of HMLV. SUMMARY

[0003] To solve the above technical problems, the present application provides a robot perception and positioning system for sorting precision fine-grained objects, which can use artificial intelligence technology to identify and classify articles and accurately place the articles in the appropriate position, suitable for application in an environment with a wide variety of articles that need to be sorted, greatly improving the sorting efficiency, reducing the time required to complete the task, and also reducing the number of workers required for the work.

[0004] Specifically, the present application provides a robot perception and positioning system for sorting precision fine-grained objects, characterized in that it comprises an identification module and a placement module, wherein:

[0005] The identification module comprises a first camera module and a second camera module, the first camera module is used for identifying, classifying and positioning the articles to be sorted, and the second camera module is used for confirming the position of the container, and the placement module is used for placing the articles to be sorted into the container according to the positioning information of the first camera and the second camera;

[0006] The system of the present application utilizes multiple sensors to process the images taken containing the objects to be sorted through region proposal, instance segmentation, fine-grained classification and 3D picking position calculation, which can utilize artificial intelligence technology to identify and classify the objects and accurately place the objects in the appropriate position, suitable for application in environments where a variety of objects need to be sorted, greatly improving the sorting efficiency, reducing the time required to complete the task, and also reducing the number of workers required for the work. BRIEF DESCRIPTION OF DRAWINGS

[0007] One or more embodiments are illustrated by way of example in the figures that form a part of this disclosure and which are shown by way of example and not limitation in the figures.

[0008] Figure 1 is an architecture diagram of the system provided by the present application.

[0009] Figure 2 is a processing schematic diagram of the first camera module of the system provided by the present application.

[0010] Figure 3 is a processing schematic diagram of the system in the fine-grained classification stage provided by the present application.

[0011] Figure 4 is a process of selecting a corresponding option in the user interface by a user using the user selection system in the system provided by the present application.

[0012] Figure 5 is a schematic diagram of the user interface of the system. DETAILED DESCRIPTION

[0013] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings. However, those skilled in the art can understand that in the embodiments of the present application, many technical details are proposed in order to make the reader better understand the present application. However, the technical solutions claimed by the present application can be realized even without these technical details and various changes and modifications based on the following embodiments.

[0014] The robot perception and positioning system for sorting fine-grained objects provided by the embodiments of the present application can not only be applied to traditional large-scale production sorting, but also can be applied to HMLV production mode, which can greatly improve the sorting efficiency in HMLV.

[0015] Specifically, as shown in Figure 1 The system includes an identification module and a placement module. The identification module mainly identifies, classifies and locates the target object and the container. The placement module is used to place the target object in a suitable position.

[0016] In this system, the recognition module processes image information acquired by the camera module. The camera module in this system includes a first camera module and a second camera module.

[0017] The first camera module includes at least one RGBD camera and one NIR camera. An RGBD camera is a device capable of simultaneously capturing color (RGB) and depth (D) information of a scene. An NIR camera operates in the near-infrared band (approximately 900 nm to 1700 nm), providing better imaging capabilities for different materials. In the field of industrial machine vision, NIR hyperspectral imaging technology is used in many areas, such as sorting plastics and textiles, quality control of food and pharmaceuticals, and film thickness and moisture measurement. NIR cameras can capture the spectral characteristics of each pixel, providing unique feature information for different materials, thereby enabling accurate classification and analysis. Therefore, this system uses NIR cameras to better acquire and utilize the image features of different items for better classification.

[0018] The first camera module of this application is mainly used for identifying, classifying, and locating items to be sorted. To achieve rapid identification, classification, and accurate location among a wide variety of items, this system employs target recognition technologies, including image segmentation and classification, to locate target objects in the image. Specifically, this system segments the image obtained by the first camera module into different parts, representing each part as a feature vector. These feature vectors are trained and classified using a learning method, with the category determined by the highest score given by the trained model. After determining the category, the system calculates the grasping direction and position by fitting the segmented mask to a rectangle. The details are as follows:

[0019] like Figure 1 and Figure 2 As shown, the first camera module of this system integrates detection, segmentation, and classification models. The processing of the acquired image by this module includes four stages: region proposal, instance segmentation, fine-grained classification, and 3D picking location calculation.

[0020] (1) Region proposal stage

[0021] The purpose of the "Region Proposal" stage is to quickly and efficiently extract regions from the entire image that may contain target objects. The proposed regions generated in this stage will be used for subsequent object detection tasks. The efficiency and accuracy of this stage are crucial to the overall performance of the object detection system. This system significantly improves the speed and efficiency of object detection by reducing the number of candidate regions that require further processing.

[0022] During the region proposal stage, this system uses a pre-trained open-set detector to detect all objects in the image, including objects of unknown and complex shapes. This model requires no additional training, enabling the system to quickly adapt to new objects and environments. At this stage, the model outputs bounding boxes for all detected objects, which can then be used to locate and segment objects of interest in the image.

[0023] At this stage, the system can use Non-Maximum Suppression (NMS) to eliminate some duplicate or highly overlapping bounding boxes, and prioritize regions most likely to contain the detected object by setting a high confidence threshold. Some regions with low confidence are ignored, but the system will re-detect the image after each detection and capture until there are no detection regions in the image with a confidence level higher than the target value. Through the above processing, the system improves both accuracy and efficiency while avoiding missing objects.

[0024] (2) Instance segmentation stage

[0025] In the instance segmentation stage, this system obtains a segmentation mask using a zero-shot visual baseline model. This model can accurately segment any object in an image based on different types of cues (including bounding boxes). The segmentation results will be used in the subsequent fine-grained classification stage.

[0026] In traditional supervised learning, models require a large amount of labeled data to learn how to recognize different categories. However, using zero-shot vision-based models allows models to learn without directly seeing samples of certain categories.

[0027] The system's fundamental visual model is trained on a large, high-quality dataset and can be applied to various tasks through cue engineering, thus possessing strong versatility. Specifically, the model consists of an image encoder, a cue encoder, and a segmentation mask decoder. The system uses a vision transformer model pre-trained with a mask autoencoder as the image encoder. A position embedding layer is used to encode point or bounding box cues, which are then input into the segmentation mask decoder along with the image encoding to output the segmentation result. Therefore, this system uses a zero-shot visual fundamental model, reducing its dependence on the original template or preset template type in object recognition and making the recognition of items outside the preset template class more reliable.

[0028] (3) Fine-grained classification stage

[0029] Fine-grained classification requires models to be able to distinguish between subcategories that belong to the same broad category but have subtle differences. This type of classification typically involves the precise identification of similar objects that may look very similar but have clear distinctions in their categories.

[0030] Generally, fine-grained classification can be achieved using the following methods:

[0031] Deep learning: uses convolutional neural networks (CNNs) to extract deep features from images, which are crucial for distinguishing subtle differences.

[0032] Attention mechanism: Through the attention mechanism, the model can focus on the part of the image that is most helpful for classification.

[0033] Multi-task learning: Combining classification tasks with other tasks (such as keypoint detection or partial annotation) to provide additional information to aid in classification.

[0034] Data augmentation: Data augmentation techniques are used to generate more training samples to improve the model's generalization ability.

[0035] In the fine-grained classification stage, such as Figure 3As shown, this system uses a multi-attention convolutional neural network. The system takes the acquired RGBD and near-infrared (NIR) signal images as input to the neural network, extracts depth features from different regions of the image, and uses these depth features as local descriptors. Next, the neural network generates partial attention from the feature channels. These partial attentions help the model focus on the most informative parts of the image for the classification task. Finally, a custom loss function is used to iteratively optimize the classification results. This loss function includes a classification loss component and a channel grouping loss component. The classification loss ensures accurate classification, while the channel grouping loss encourages the model to learn consistent feature representations. This iterative optimization of the loss function helps the model converge to a robust and discriminative feature set for the current classification task.

[0036] For detecting unknown samples in the input image, a confidence threshold and a range of expected object sizes can be set to reject or exclude unknown samples. If the classification confidence is below the threshold or the segmentation mask size deviates significantly from the expected object size, it is identified as an unknown object. Compared with traditional end-to-end object detection models, this system's modules are more flexible.

[0037] (4) 3D picking location calculation

[0038] This system calculates the 2D coordinates of the minimum bounding rectangle of the object segmentation mask to obtain the gripping angle of the gripper. Then, using depth information obtained from an RGBD camera and hand-eye calibration, it calculates the 3D coordinates of the object being gripped by the gripper. Specifically, this can be achieved through the following methods:

[0039] Acquiring Depth Information: The scene is captured using an RGBD camera, which can simultaneously provide a color image (RGB) and a depth image. Depth information is extracted from the depth image, which is typically a single-channel image with the same resolution as the color image, where the value of each pixel represents the distance from the camera to the corresponding point in the scene.

[0040] Hand-eye calibration: Hand-eye calibration is performed to determine the spatial relationship between the camera coordinate system (defined by the RGBD camera) and the robot coordinate system (defined by the robot's base or tool coordinate system). This typically involves the following steps: determining the position and orientation of the camera relative to the robot's end effector (gripper); and calculating the transformation matrix using a series of calibration objects at known positions or using an algorithm (such as the Park-Martin method).

[0041] 3D reconstruction: Combining color images and depth information, the 3D coordinates of each point in the scene are calculated using the pixel coordinates in the color image and the corresponding depth values. This process typically involves converting each pixel value in the depth image into an actual distance, and then using the camera's intrinsic parameters (such as focal length and principal point coordinates) to convert the 2D pixel coordinates into 3D spatial coordinates.

[0042] Object localization: Detecting or segmenting target objects in a color image. This can be achieved using deep learning methods (such as convolutional neural networks). Based on the detected object's position in the color image, the corresponding depth information is found, thereby obtaining a 3D point cloud of the object's surface.

[0043] Coordinate transformation: This involves transforming the detected 3D point cloud of the object from the camera coordinate system to the robot coordinate system. This requires using a transformation matrix obtained from hand-eye calibration. Applying the transformation matrix, the coordinates of each 3D point are transformed from the camera coordinate system to the robot coordinate system.

[0044] Through the aforementioned first camera module, this system can achieve the identification, classification, and location of items. Especially for raw materials like HMLV, which are diverse and many items are very similar, this system can efficiently, quickly, and accurately identify, classify, and locate these items with relatively low training costs.

[0045] In the second camera module of this system, an RGBD camera is configured to monitor the container and confirm its position. To obtain the container's position and orientation, the system uses a hybrid object extraction and object matching algorithm. A general background removal neural network is applied to the acquired image, combined with a keypoint extraction and alignment system, enabling accurate object placement without training the neural network. This approach eliminates the need for dedicated alignment hardware, increasing flexibility for various container types and shapes.

[0046] This system also includes a user selection system, which can be presented as a web application or in other ways. Users can use this system to select parameters for the pick-and-place process, such as... Figure 4 The system displays the object placement location, the number of repeated trials, and other information. Users can also view the status of the pick-and-place operation through status text and a real-time view from the camera. Additionally, the system provides a training user interface where users can upload new object datasets to retrain the recognition model, allowing technicians without AI knowledge to easily update the model as needed.

[0047] like Figure 1As shown, the placement module of this system includes a robotic arm. Based on the position of the target item confirmed by the first camera and the corresponding container position confirmed by the second camera, the system controls the robotic arm to grasp the target item and place it on the corresponding container, thus achieving sorting. Besides the robotic arm, other mechanical devices capable of achieving similar effects can serve as the placement module of this system. In the robotic arm module, the system controls the robotic arm by providing pick-up and place-down positions. These positions are encapsulated in Robot Operating System (ROS) commands and transmitted via ROS message subscription and publication. Furthermore, the pick-up and place-down positions must be in world coordinates, which are derived from 2D coordinates using parameters obtained through hand-eye calibration (i.e., obtained by the second camera module).

[0048] In summary, this system provides a robotic perception and positioning system for sorting fine-grained objects. It can use artificial intelligence technology to identify and classify items and accurately place them in the appropriate locations. It is suitable for applications in environments where there are many different types of items to be sorted, greatly improving sorting efficiency, reducing the time required to complete the task, and also reducing the number of workers required.

[0049] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0050] Those skilled in the art will understand that the above embodiments are specific examples of implementing the present invention, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of the present invention.

Claims

1. A robotic perception and localization system for sorting precision fine-grained objects, characterized by, The system comprises an identification module and a placement module, wherein: The identification module comprises a first camera module and a second camera module, the first camera module is used for identifying, classifying and positioning the items to be sorted, and the second camera module is used for confirming the position of the container, and the placement module is used for placing the items to be sorted into the container according to the positioning information of the first camera and the second camera. The first camera module comprises at least one RGBD camera, and the first camera module performs region proposal, instance segmentation, fine-grained classification and 3D picking position calculation on the image containing the items to be sorted captured by the RGBD camera.

2. The system of claim 1, wherein, The first camera module further comprises an NIR camera.

3. The system of claim 1, wherein, In the instance segmentation stage, the system obtains a segmentation mask by using a zero-shot visual base model, which can accurately segment any object in the image according to different types of prompts (including bounding boxes).

4. The system of claim 1, wherein, In the fine-grained classification stage, the system processes by the following steps: Taking the obtained image as the input of a multi-attention convolutional neural network, deep features are extracted from different regions of the image, and the deep features of different regions are taken as local descriptors; Partial attentions are generated from the feature channels, which help the model focus on the most informative parts of the image for the classification task; The classification results are iteratively optimized using a customized loss function, which includes a classification loss component and a channel grouping loss component, the classification loss ensures accurate classification, and the channel grouping loss encourages the model to learn consistent feature representations.

5. The system of claim 1, wherein, The system pre-sets a confidence threshold and an expected object size range, and when identifying the items to be sorted, if the classification confidence of the identification result is lower than the threshold or the size of the segmentation mask deviates significantly from the expected object size, it is identified as an unknown object.

6. The system of claim 1, wherein, In the 3D picking position calculation stage, the system processes by the following steps: Calculate the 2D coordinates of the minimum bounding rectangle of the object segmentation mask to obtain the grabbing angle of the gripper; Calculate the 3D coordinates of the gripper grabbing the object through the depth information obtained by the RGBD camera and the hand-eye calibration.

7. The system of claim 1, wherein, The second camera module of the system comprises at least one RGBD camera.

8. The system of claim 7, wherein, The system uses a hybrid of object extraction algorithms and object matching algorithms, applies a general background removal neural network to the image obtained by the second camera module, and combines a key point extraction and alignment system.

9. The system of claims 1-8, wherein, The system further comprises a user selection system, which can select the classification preferences or categories of the first camera module and the positioning preferences of the second camera according to the user's selection.

10. The system of claim 9, wherein, The user selection system can train the built-in identification model according to the data of new sorting items uploaded by the user.

Images