Material sorting control method, device and equipment and storage medium

Abstract

The invention provides a material sorting control method and device, equipment and a storage medium, and the method comprises the steps: carrying out the target detection of a collected color image of a material scene, and obtaining the segmentation region, position information and category information of each material; processing the depth image according to the segmented region to obtain three-dimensional space coordinates and size parameters of each material; analyzing a spatial relationship between the segmented regions, and judging a shielding state between the materials according to the depth image to obtain a relative position relationship between the materials; and according to the relative position relation, grabbing sequence planning is conducted on all the materials, the three-dimensional space coordinates and the size parameters of the current grabbable materials are sent to the mechanical arm to execute grabbing operation, the relative position relation is updated after grabbing is completed, and grabbing sequence planning is repeatedly executed till sorting of all the materials is completed. According to the method, the shielding relation between the materials is judged, the grabbing sequence is planned according to the shielding relation, collision in the grabbing process is avoided, and the grabbing success rate and the sorting efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a material sorting control method, apparatus, equipment, and storage medium. Background Technology

[0002] In industrial production, materials are typically supplied to the production line in batches to containers such as crates and baskets. Due to differences in material shape and weight, as well as vibrations during handling, materials are often piled up randomly in the containers, resulting in obstruction and overlap between them. Traditional manual sorting methods are labor-intensive, inefficient, and prone to errors, making them unsuitable for the demands of modern production.

[0003] Existing robotic arm automated grasping systems typically employ a simple first-come, first-served strategy, sequentially grasping identified materials without considering occlusion relationships between them. This approach is prone to collisions with materials above when grasping materials that are obscured at lower levels, leading to grasping failures, material damage, or equipment malfunctions. This impacts production efficiency and equipment safety, and makes it unsuitable for material sorting needs in complex stacking scenarios. Summary of the Invention

[0004] The main objective of this invention is to solve the technical problem that existing robotic arm grasping systems do not consider the occlusion relationship between materials, resulting in a high grasping failure rate and a high likelihood of collisions. This invention provides a material sorting control method, the material sorting control method comprising: Target detection is performed on the collected color images of the material scene to obtain the segmentation region, location information and category information of each material; The depth image of the material scene is processed according to the segmented region to obtain the three-dimensional spatial coordinates and size parameters of each material; The spatial relationship between the segmented regions is analyzed, and the occlusion state between materials is determined based on the depth image to obtain the relative positional relationship between the materials. Based on the relative positional relationship, the grabbing sequence of each material is planned. The three-dimensional spatial coordinates and size parameters of the currently grabbable material are sent to the robotic arm to perform the grabbing operation. After the grabbing is completed, the relative positional relationship is updated. The grabbing sequence planning is repeated until all materials are sorted.

[0005] The present invention also provides a material sorting control device, the material sorting control device comprising: The target detection module is used to perform target detection on the acquired color images of the material scene to obtain the segmentation region, location information and category information of each material; The three-dimensional positioning module is used to process the depth image of the material scene based on the segmented region to obtain the three-dimensional spatial coordinates and size parameters of each material. The occlusion judgment module is used to analyze the spatial relationship between the segmented regions, determine the occlusion state between materials based on the depth image, and obtain the relative positional relationship between the materials. The gripping control module is used to plan the gripping sequence of each material according to the relative position relationship, send the three-dimensional spatial coordinates and size parameters of the currently grippable material to the robotic arm to perform the gripping operation, and update the relative position relationship after the gripping is completed. The gripping sequence planning is repeated until all materials are sorted.

[0006] The present invention also provides a material sorting control device, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the material sorting control device to perform the steps of the material sorting control method described above.

[0007] The present invention also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the material sorting control method described above.

[0008] The aforementioned material sorting control method, apparatus, equipment, and storage medium perform target detection on the acquired color images of the material scene to obtain the segmented regions, location information, and category information of each material; process the depth images based on the segmented regions to obtain the three-dimensional spatial coordinates and size parameters of each material; analyze the spatial relationships between the segmented regions, determine the occlusion status between materials based on the depth images, and obtain the relative positional relationships between materials; plan the grasping sequence for each material based on the relative positional relationships, send the three-dimensional spatial coordinates and size parameters of the currently graspable materials to the robotic arm to execute the grasping operation, update the relative positional relationships after grasping, and repeat the grasping sequence planning until all materials are sorted. This invention, by determining the occlusion relationships between materials and planning the grasping sequence accordingly, avoids collisions during the grasping process, improving the grasping success rate and sorting efficiency.

[0009] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0010] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0011] Figure 1This is a schematic diagram of the first embodiment of the material sorting control method in this invention; Figure 2 This is a schematic diagram of a second embodiment of the material sorting control method in this invention; Figure 3 This is a schematic diagram of one embodiment of the material sorting control device in this invention; Figure 4 This is a schematic diagram of one embodiment of the material sorting control equipment in this invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The terms "comprising" and "having," and any variations thereof, used in the embodiments of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0014] To facilitate understanding of this embodiment, a material sorting control method disclosed in this invention will first be described in detail. For example... Figure 1 As shown, this method includes the following steps: 101. Perform target detection on the collected color images of the material scene to obtain the segmentation region, location information and category information of each material; In this embodiment, the step of performing target detection on the acquired color image of the material scene to obtain the segmentation region, location information, and category information of each material includes: performing convolutional feature extraction on the color image to obtain a multi-layer feature map; performing linear attention calculation on the multi-layer feature map, obtaining attention weights by applying nonlinear activation functions to query features and key features respectively and then performing matrix operations, and weighting and aggregating value features according to the attention weights to obtain attention-enhanced features; performing channel-dimensional weight filtering on different levels of features of the attention-enhanced features, and then upsampling high-level features through transposed convolution and fusing them with low-level features to obtain multi-scale fusion features; performing target detection and instance segmentation on the multi-scale fusion features to obtain the bounding box coordinates, category labels, and pixel-level segmentation masks of each material; determining the segmentation region of each material according to the segmentation mask, determining the location information of each material according to the bounding box coordinates, and determining the category information of each material according to the category labels.

[0015] Specifically, the material sorting control system acquires color images of the material handling scene using industrial cameras, with an image resolution of 640×640 pixels. These color images are then input into a target detection model for processing, which employs a lightweight deep convolutional neural network architecture based on VisionTransformer.

[0016] In one embodiment, the feature extraction module uses MobileNet or EfficientNet series networks as the backbone network to perform multi-scale convolutional feature extraction on the color image. The first convolutional layer uses a 3×3 convolutional kernel with a stride of 1 and 64 output channels to obtain the first layer feature map; the second convolutional layer uses depthwise separable convolution with a stride of 2 and 128 output channels to obtain the second layer feature map. Through layer-by-layer convolutional operations, multi-layer feature maps from edge texture to material shape structure are extracted, resulting in feature maps with resolutions of 320×320, 160×160, and 80×80, respectively.

[0017] In one embodiment, linear attention computation based on EfficientViT is performed on the multi-layer feature maps. For any feature map layer, a query feature matrix Q, a key feature matrix K, and a value feature matrix V are generated through three 1×1 convolutional layers, each with a dimension of N×d, where N represents the total number of pixels in the feature map and d represents the feature dimension. The ReLU activation function is applied to Q and K respectively to obtain ReLU(Q) and ReLU(K).

[0018] Furthermore, a linear attention mechanism is adopted instead of the traditional Softmax attention mechanism. The transpose of ReLU(K) is multiplied by V to obtain the key-value interaction feature, with a dimension of d×d. The transpose of ReLU(K) is then summed along the feature dimension to obtain a normalization factor. ReLU(Q) is multiplied by the key-value interaction feature and divided by the normalization factor to obtain the attention output feature. This method reduces the computational complexity from O(N²) to O(N), making it suitable for deployment on industrial equipment with limited computing power. The attention output feature is then processed using depthwise separable convolution to obtain the attention-enhanced feature.

[0019] In one embodiment, a channel attention mechanism based on CBAM (Convolutional Block Attention Module) is used to filter high-level features. Global average pooling and global max pooling are performed on the high-level features, compressing the features from H×W×C to 1×1×C. The two pooling results are concatenated along the channel dimension to obtain a 1×1×2C statistical feature. This feature is then dimensionality-reduced to C channels using a 1×1 convolution, and weight coefficients are calculated using the sigmoid function. These weight coefficients are multiplied channel-by-channel by the original high-level feature to obtain the filtered high-level features.

[0020] Transposed convolution is used to upsample the filtered high-level features. The kernel size is 3×3, the stride is 2, and the padding is 1, which increases the feature resolution by a factor of 2. For example, an 80×80 feature is upsampled to a 160×160 feature. The parameters of the transposed convolution are learnable and can adaptively adjust the upsampling strategy according to task requirements, offering stronger expressive power compared to bilinear interpolation. The upsampled high-level features are then pixel-wise added to the corresponding low-level features using an FPN structure to obtain multi-scale fused features.

[0021] Multi-scale fusion features are processed through a Mask R-CNN-based detection head module, including classification, regression, and segmentation branches. The classification branch uses a Softmax classifier to predict class labels, the regression branch uses a CIoU (Complete IoU) loss function to predict bounding box coordinates, and the segmentation branch uses an FCN (Fully Convolutional Network) to generate pixel-level segmentation masks. During training, a multi-task learning strategy is employed, using a weighted sum of cross-entropy loss, CIoU loss, and binary cross-entropy loss as the total loss.

[0022] During inference, the Non-Maximum Suppression (NMS) algorithm is used to remove duplicate bounding boxes, obtaining the final bounding box coordinates, category labels, and segmentation masks for each material. The set of coordinates with a pixel value of 1 is extracted from the segmentation mask to determine the segmented region of the material. Location information is determined based on the bounding box coordinates, and category information is determined based on the category labels, providing foundational data for subsequent 3D localization and occlusion detection.

[0023] 102. Process the depth image of the material scene according to the segmented region to obtain the three-dimensional spatial coordinates and size parameters of each material; In this embodiment, processing the depth image of the material scene according to the segmented region to obtain the three-dimensional spatial coordinates and size parameters of each material includes: extracting corresponding depth data from the depth image according to the segmented region to obtain the depth value of each material; calculating the conversion ratio between pixels and actual distance based on the pixel size and actual size of a pre-calibrated standard object; calculating the coordinate position of each material in three-dimensional space based on the depth value and the conversion ratio; and calculating the actual length, width, and height of each material based on the pixel size of the segmented region and the conversion ratio to obtain the size parameters of each material.

[0024] Specifically, after obtaining the segmented areas of each material, the material sorting control system uses depth images to perform three-dimensional spatial positioning of each material. In this embodiment, the depth images are acquired using an RGB-D camera or a structured light depth camera. The resolution of the depth images is consistent with that of the color images, and each pixel records the actual distance from that point to the camera, in millimeters.

[0025] In one embodiment, depth data is extracted from the depth image based on the segmented region. Specifically, the segmented region is defined by a pixel-level segmentation mask, where a pixel with a value of 1 indicates that the pixel belongs to the foreground region of the material. All pixels with a value of 1 in the segmentation mask are traversed, and the depth values ​​at their corresponding positions in the depth image are obtained. These depth values ​​are then stored as a depth data set. For example, if the segmented region of a material contains 5000 pixels, the depth values ​​at the corresponding 5000 positions in the depth image are extracted to form the depth data set for that material.

[0026] It should be noted that some pixels in the depth image may have missing or abnormal depth values, usually due to surface reflection, occlusion, or camera measurement errors. Pixels with a depth value of zero or outside a reasonable range, such as those with a depth value less than 100 mm or greater than 3000 mm, are marked as invalid depth values ​​and discarded. The remaining valid depth values ​​are processed using a median filtering algorithm to remove the influence of outliers. This algorithm replaces outliers by statistically analyzing the median depth values ​​within the neighborhood, effectively preserving edge information while removing noise.

[0027] In one embodiment, statistical analysis is performed on the effective depth data to calculate the mean, median, and standard deviation of the depth values. If the standard deviation of the depth values ​​is less than a preset threshold, such as 10 mm, it indicates that the material surface is relatively flat, and the mean depth value is used as the characteristic depth value of the material. If the standard deviation is greater than the preset threshold, it indicates that the material surface has large undulations or stacking, and the median is used as the characteristic depth value to avoid interference from extreme values. This statistical analysis method can adapt to materials with different shapes and surface characteristics, improving the robustness of depth estimation.

[0028] In one embodiment, the conversion ratio between pixels and actual distance is calculated through a calibration process. Specifically, before the material sorting control system is started, a standard object is placed at a fixed position in the material container for calibration. The standard object is a rectangular block of known dimensions, such as a square standard piece with a length of 100 mm and a width of 100 mm. The standard object is made of a non-reflective matte material, and its color is clearly distinguishable from the material to be detected for accurate identification. The standard object should be in close contact with the reference plane of the material container, ensuring that its depth is similar to that of the material to be detected.

[0029] The Zhang Zhengyou calibration method was used to calibrate the camera's intrinsic parameters. This method involves photographing a checkerboard calibration board at different angles and using multi-view geometric constraints to solve for the camera's intrinsic parameter matrices, including focal length, principal point coordinates, and distortion coefficients. After obtaining the camera's intrinsic parameters, the aforementioned object detection model was used to identify a standard object and obtain the bounding box coordinates of the standard object in the image. The pixel length and pixel width of the standard object in the image were calculated, where the pixel length is the horizontal pixel span of the bounding box, and the pixel width is the vertical pixel span of the bounding box.

[0030] The conversion ratio between pixels and actual distances is calculated based on the actual length of the standard object and the pixel length in the image. For example, if the standard object has a pixel length of 200 pixels and an actual length of 100 millimeters, the conversion ratio is 0.5 millimeters per pixel. This conversion ratio reflects the actual physical size corresponding to each pixel in the image under the current camera mounting position and focal length setting. This ratio will serve as the benchmark parameter for all subsequent material size calculations.

[0031] It should be noted that the conversion ratio will vary slightly across different depth planes, and the farther an object is from the camera, the larger the actual size corresponding to each pixel. To simplify calculations, this embodiment assumes that all materials are located in a depth plane similar to the standard object and uses a uniform conversion ratio for calculation. For applications with significant depth differences, a perspective transformation algorithm can be used for depth-related ratio correction. This algorithm dynamically adjusts the conversion ratio based on the actual depth of the object to ensure measurement accuracy.

[0032] In one embodiment, three-dimensional spatial coordinates are calculated based on the material's characteristic depth values ​​and camera intrinsic parameters. Specifically, for the segmented region of the material, the geometric center point of the segmentation mask is calculated, i.e., the average of the horizontal and vertical coordinates of all foreground pixels, to obtain the pixel coordinates of the center point. A pinhole camera imaging model is used to convert the two-dimensional pixel coordinates into three-dimensional spatial coordinates. This model, based on the camera's optical principles, establishes a mapping relationship between pixel coordinates and three-dimensional spatial coordinates.

[0033] Based on the principal point coordinates, focal length parameters, and characteristic depth values ​​of the material in the camera's intrinsic parameter matrix, the spatial coordinates of the material's center point in the three-dimensional camera coordinate system are calculated through coordinate transformation. This coordinate system has the camera's optical center as the origin, the optical axis as the Z-axis, and the horizontal and vertical directions of the image plane as the X-axis and Y-axis, respectively. Through this calculation process, the material is transformed from a two-dimensional image coordinate system to a three-dimensional spatial coordinate system, obtaining the material's precise position in three-dimensional space.

[0034] In one embodiment, the actual size parameters of the material are calculated based on the pixel dimensions of the segmented region. Specifically, a minimum bounding rectangle fitting is performed on the segmentation mask. This algorithm uses a rotating caliper method to find the smallest rectangle that can tightly enclose the material's outline. This function can be implemented using the minimum bounding rectangle function in the OpenCV computer vision library, which returns parameters such as the center coordinates, length, width, and rotation angle of the rectangle. This rectangle accurately reflects the true length and width dimensions of the material even if it is tilted in the image.

[0035] Multiply the length and width pixel values ​​of the rectangle by the previously calculated conversion ratio to obtain the actual length and width of the material, in millimeters. The height parameter of the material is estimated using the range of depth data. Within the segmented area of ​​the material, all valid depth values ​​are traversed to find the maximum and minimum depth values; the difference between these two values ​​represents the height span of the material along the camera's line of sight, reflecting the material's actual height.

[0036] 103. Analyze the spatial relationship between the segmented regions, determine the occlusion state between materials based on the depth image, and obtain the relative positional relationship between the materials; In this embodiment, after obtaining the segmented areas and three-dimensional coordinates of each material, the material sorting control system needs to analyze the occlusion relationships between the materials. In industrial scenarios, materials are often stacked randomly in the material box, with upper and lower layers occluding each other. If the robotic arm directly grabs the lower layer of occluded material, it may collide with the upper layer of material, resulting in grabbing failure or material damage.

[0037] In one embodiment, the boundary contours of the segmented regions of each material are extracted using the Canny edge detection algorithm or the contour finding algorithm to obtain the boundary pixel coordinate sequence. For any two materials, the Euclidean distance between their boundary points is calculated, the minimum distance value and the corresponding nearest point pair are found, and the midpoint of the nearest point pair is calculated as the interaction position. When the minimum distance is less than a preset distance threshold, such as 20 pixels, it is determined that the two materials have a spatial interaction relationship.

[0038] Construct a square sampling region at the interaction location, with a side length of 150 pixels. Within this region, extract the depth values ​​of the pixels belonging to each material using the segmentation mask of the two materials. Calculate the arithmetic mean of the extracted depth data to obtain the average depth value of each material at the interaction location.

[0039] Compare the average depth values ​​of two materials; a smaller depth value indicates a closer distance to the camera and a higher layer. If the average depth value of the first material is less than that of the second material, then the first material is determined to be occluding the second material. Perform pairwise traversal on all materials to ensure that the occlusion relationship of each pair is detected.

[0040] The occlusion relationships are recorded as an N x N matrix. A value of 1 indicates that a material in a row is above a material in a column, -1 indicates that the material is below the material in a column, and 0 indicates no occlusion relationship. This matrix clearly expresses the occlusion relationships of all materials, providing a data foundation for subsequent crawling order planning.

[0041] 104. Based on the relative positional relationship, plan the grabbing sequence for each material, send the three-dimensional spatial coordinates and size parameters of the currently grabbable material to the robotic arm to perform the grabbing operation, and update the relative positional relationship after the grabbing is completed. Repeat the grabbing sequence planning until all materials are sorted.

[0042] In this embodiment, the step of planning the grasping order of each material according to the relative positional relationship, sending the three-dimensional spatial coordinates and size parameters of the currently graspable material to the robotic arm to perform the grasping operation, updating the relative positional relationship after grasping, and repeating the grasping order planning until all materials are sorted includes: scanning the row of the relationship matrix of the relative positional relationship, finding materials in the matrix rows that only contain upper-level relationship identifiers and no-relationship identifiers, and determining the found materials as the current graspable material set; selecting a material from the current graspable material set as the current grasping target, obtaining the three-dimensional spatial coordinates and size parameters corresponding to the current grasping target, and sending the three-dimensional spatial coordinates and size parameters to the robotic arm control system; after receiving the grasping completion signal from the robotic arm control system, setting the row element and column element corresponding to the current grasping target in the relationship matrix as no-relationship identifiers; determining whether there are still elements in the relationship matrix that have not been set as no-relationship identifiers, if so, returning to continue executing the row scanning step, if not, determining that all materials have been sorted.

[0043] Specifically, in one embodiment, the relationship matrix is ​​scanned row by row to identify currently graspable materials. The relationship matrix is ​​an N x N square matrix, where N is the total number of materials. A row-by-row traversal algorithm is used to scan the matrix, checking all elements in each row. If a row contains only upper-level relationship identifiers (element value 1) and no-relationship identifiers (element value 0), but does not contain lower-level relationship identifiers (element value -1), it means that the material is not obscured by other materials and is located in an upper-level position that can be directly grasped.

[0044] It should be noted that the "upper-level relationship" indicator means that the material is located above other materials, while the "no relationship" indicator means that the material is independent and has no interaction with other materials; neither of these states affects the grasping process. The "lower-level relationship" indicator indicates that the material is obscured and must wait for the obscuring material to be removed before it can be grasped. This rule filters out all directly graspable materials, forming the current set of graspable materials.

[0045] In one embodiment, a material is selected from the set of graspable materials as the current grasping target. The selection strategy can be a first-come, first-served strategy, where the first graspable material is selected in order of its material number. This strategy is simple to implement and has low computational overhead. Alternatively, a priority scheduling algorithm can be used, which formulates an optimization strategy based on material attributes. For example, it can prioritize grasping smaller materials to reserve space for larger materials; or prioritize grasping materials of specific categories to achieve category-based sorting; or prioritize grasping materials closer to the robotic arm to reduce movement time and improve efficiency.

[0046] After identifying the target material, the three-dimensional spatial coordinates and dimensional parameters of the material are obtained. The three-dimensional spatial coordinates include the X, Y, and Z coordinates of the material's center point, which guide the robotic arm to move to the correct position. The dimensional parameters include length, width, and height, which are used to adjust the gripper opening and gripping posture to ensure stable material holding.

[0047] The three-dimensional coordinates and dimensional parameters are encapsulated into control commands and sent to the robotic arm control system via communication interfaces such as industrial Ethernet, serial communication, or fieldbus. Upon receiving the commands, the robotic arm control system uses inverse kinematics algorithms to calculate the angle values ​​of each joint and controls the end effector to move to the target position. Based on the dimensional parameters, the gripper opening and closing degree is adjusted, the gripper closes to complete the grasping, and the material is transported to the designated sorting location.

[0048] In one embodiment, after the robotic arm completes the grasping and placement, it sends a grasping completion signal via a communication interface. This signal can be a Boolean value indicating task completion, or it can contain detailed status information. Upon receiving the signal, the material sorting control system confirms that the material has been removed and updates the relationship matrix to reflect the current scene status.

[0049] Specifically, the row and column corresponding to the currently captured target are located in the relationship matrix, and all elements in that row and column are set to a "no relationship" flag, i.e., their values ​​are set to 0. Setting a row element to 0 indicates that the material no longer has a relationship with other materials, and setting a column element to 0 indicates that other materials are no longer occluded by this material. Through this dynamic update mechanism, the relationship matrix can reflect the occlusion relationships of the remaining materials in real time, eliminating the need for re-image acquisition and occlusion judgment, thus greatly improving computational efficiency.

[0050] After updating the relation matrix, determine if there are any non-zero elements in the matrix. Iterate through all elements and check if there are any elements with non-zero values. If there are non-zero elements, it means that there are still materials in the scene that have not been grabbed, and the grabbing task needs to continue. Return to the row scanning step, re-identify the set of grabbable materials, select a new grabbing target, and form a loop execution process.

[0051] If all elements in the relation matrix are 0, it means that all materials have been successfully grabbed and removed, and the sorting task is complete. Output a flag indicating that all materials have been sorted, end the current batch operation, and wait for the next batch of materials to arrive.

[0052] Understandably, through the aforementioned grasping sequence planning and dynamic update mechanism, the material sorting control system can automatically complete the entire process from identification, positioning, occlusion judgment to grasping execution. This method, through dynamic updating of the relationship matrix, avoids the overhead of re-processing images and re-judging occlusion after each grasp, significantly improving sorting efficiency, ensuring the safety of the robotic arm operation, and is suitable for the automated material sorting needs in industrial production.

[0053] In this embodiment, target detection is performed on the acquired color images of the material scene to obtain the segmented regions, location information, and category information of each material. The depth image is processed based on the segmented regions to obtain the three-dimensional spatial coordinates and size parameters of each material. The spatial relationship between the segmented regions is analyzed, and the occlusion state between materials is determined based on the depth image to obtain their relative positional relationship. Based on the relative positional relationship, a grasping sequence is planned for each material. The three-dimensional spatial coordinates and size parameters of the currently graspable material are sent to the robotic arm to execute the grasping operation. After grasping, the relative positional relationship is updated, and the grasping sequence planning is repeated until all materials are sorted. This invention avoids collisions during the grasping process by judging the occlusion relationship between materials and planning the grasping sequence accordingly, thus improving the grasping success rate and sorting efficiency.

[0054] Please see Figure 2 Another embodiment of the material sorting control method in this application includes: 201. Perform target detection on the collected color images of the material scene to obtain the segmentation region, location information and category information of each material; 202. Process the depth image of the material scene according to the segmented region to obtain the three-dimensional spatial coordinates and size parameters of each material; In this embodiment, steps 201-202 are similar to steps 101-102, and will not be described again here.

[0055] 203. Extract the boundaries of the segmented region and calculate the distance relationship between the boundaries of different materials to obtain the material pairs with spatial interaction and their interaction positions; In this embodiment, the step of extracting the boundaries of the segmented region and calculating the distance relationship between the boundaries of different materials to obtain material pairs with spatial interaction and their interaction positions includes: extracting the boundary contours of the segmented region and sampling the boundary contours to obtain a set of boundary sampling points; calculating the pairwise distances of the boundary sampling point sets of different materials and using a nearest neighbor search algorithm to determine the minimum distance value and the coordinates of the corresponding nearest point pair; converting a preset physical distance threshold into a pixel distance threshold according to a pre-calibrated conversion ratio between pixels and actual distances; comparing the minimum distance value with the pixel distance threshold, and when the minimum distance value is less than the pixel distance threshold, determining that the material pair has spatial interaction, and taking the midpoint of the coordinates of the nearest point pair as the interaction position.

[0056] Specifically, after obtaining the segmented areas for each material, the material sorting control system needs to analyze the spatial relationships between the materials to identify which materials are spatially close to each other or obstruct each other. Boundary distance calculation is a key step in determining the spatial interaction relationships between materials. By extracting material boundaries and calculating the shortest distance between boundaries, material pairs that are in contact or overlapping can be accurately identified.

[0057] In one embodiment, boundary contour extraction is performed on the segmented region. Specifically, the segmented region is represented by a pixel-level segmentation mask, where areas with a pixel value of 1 represent the foreground material and areas with a pixel value of 0 represent the background. A contour finding algorithm is used to process the segmentation mask to extract the boundary line between the material and the background, thus obtaining the boundary contour of the material. Contour extraction can be implemented using the findContours function in the OpenCV computer vision library. This function tracks boundary pixels based on the Suzuki algorithm and outputs the complete contour of the material as a sequence of pixel coordinates.

[0058] The resulting boundary contour typically contains hundreds or even thousands of pixels. Directly calculating pairwise distances between all boundary points results in a computational complexity of O(N²), where N is the total number of boundary points. This leads to excessively long computation times and impacts real-time performance. Therefore, it is necessary to sample the boundary contour to reduce computational load while maintaining accuracy.

[0059] In one embodiment, the boundary contour is uniformly sampled. Specifically, the total length of the boundary contour is calculated, which is the sum of the distances between all adjacent points. Sampling points are extracted along the contour curve at fixed intervals according to a preset sampling interval, for example, sampling one point every 5 pixels. Alternatively, an adaptive sampling strategy based on curvature can be used, increasing the sampling point density at locations with high contour curvature, i.e., corners or abrupt changes, and reducing the sampling points in smooth areas with low curvature. Through this sampling process, the number of boundary points can be reduced to one-tenth or even less of the original number, resulting in a boundary sampling point set while preserving the main geometric features of the contour.

[0060] In one embodiment, pairwise distance calculations are performed on the boundary sampling point sets of different materials. Specifically, for any two materials in the scene, their boundary sampling point sets are obtained, denoted as set A and set B. Each sampling point in set A is traversed, and the Euclidean distance between that point and all sampling points in set B is calculated. The Euclidean distance is obtained by taking the square root of the sum of the squares of the differences in the x-coordinates and y-coordinates of the two points, representing the straight-line distance between the two points on the two-dimensional image plane.

[0061] For large-scale point sets, direct pairwise traversal is computationally inefficient. Employing a nearest neighbor search algorithm can significantly improve computational efficiency. Specifically, a KD-tree data structure can be used to spatially index the sampled points in set B. A KD-tree is a spatial partitioning tree that recursively partitions the point set according to different dimensions, constructing a hierarchical search structure. For each query point in set A, a nearest neighbor search is performed in the KD-tree, reducing the time complexity to O(log N). This algorithm quickly finds the nearest neighbor of each point in set A in set B and records the shortest distance value.

[0062] Iterate through all query results and find the minimum distance value among all distance values. This minimum value is the shortest distance between the two material boundaries. Simultaneously, record the coordinates of the corresponding nearest point pair, i.e., the coordinates of the two closest sampling points on the two material boundaries. This point pair represents the closest spatial position of the two materials and is key information for determining the material interaction relationship.

[0063] In one embodiment, a physical distance threshold is converted into a pixel distance threshold based on a pre-calibrated conversion ratio between pixels and actual distances. Specifically, during the aforementioned calibration process, the conversion ratio between pixels and actual distances has been calculated, for example, 0.5 mm per pixel. The physical distance threshold is set according to actual application requirements; for example, spatial interaction is considered to exist when the boundary distance between two materials is less than 10 mm. This physical distance threshold is divided by the conversion ratio to obtain the pixel distance threshold; for example, 10 mm divided by 0.5 mm per pixel yields a pixel distance threshold of 20 pixels.

[0064] It's important to note that setting the pixel distance threshold requires comprehensive consideration of camera resolution, material size, and the accuracy requirements for occlusion detection. Setting the threshold too low may miss material pairs that actually interact, while setting it too high may misclassify spatially independent materials as interacting. Converting physical distance to pixel distance makes threshold setting more intuitive and easier to adjust based on the actual scenario.

[0065] In one embodiment, the calculated minimum distance value is compared with a pixel distance threshold. If the minimum distance value is less than the pixel distance threshold, it indicates that the boundaries of the two materials are very close in space, and it is determined that the pair of materials have a spatial interaction relationship, requiring further occlusion relationship determination. The midpoint position of the nearest point pair is calculated by adding the x-coordinates of the two nearest points and dividing by 2, and adding the y-coordinates and dividing by 2 to obtain the midpoint coordinates. This midpoint position is used as the interaction position of the two materials, and a sampling area will be constructed at this position for depth data extraction.

[0066] If the minimum distance value is greater than or equal to the pixel distance threshold, it indicates that the two materials are spatially far apart and have no contact or occlusion relationship. The relationship between these material pairs is recorded as unrelated, and no further depth comparison is needed. This distance threshold filtering method can quickly eliminate a large number of unrelated material pairs, allowing for depth analysis only on material pairs that truly interact, significantly reducing computational load and improving processing efficiency.

[0067] 204. Construct a sampling area at the interaction location, and extract depth data corresponding to different materials in the sampling area according to the segmentation area to obtain the depth value of each material at the interaction location; In this embodiment, constructing a sampling region at the interaction location and extracting depth data corresponding to different materials within the sampling region based on the segmented region to obtain the depth value of each material at the interaction location includes: calculating the pixel length and pixel width of the segmented region, calculating a feature size, where the feature size is the smaller or average value of the pixel length and pixel width; multiplying the feature size by a preset scaling factor to obtain the side length of the sampling region, and constructing a square sampling region centered on the interaction location; extracting depth data belonging to different materials within the square sampling region based on the mask information of the segmented region; performing statistical analysis on the extracted depth data, calculating the median and standard deviation of the depth values, marking depth values ​​that deviate from the median by more than a preset multiple of the standard deviation as outliers and removing them; and calculating the average value of the valid depth data after removing outliers to obtain the depth value of each material at the interaction location.

[0068] Specifically, after determining the interaction location of material pairs, the material sorting control system needs to construct a sampling area at that location to extract depth information to determine the hierarchical relationship between the materials. The construction of the sampling area needs to comprehensively consider the size characteristics of the materials and the actual situation of the scenario, ensuring that the sampling area can cover the interaction part of the two materials, while avoiding an area that is too large and introduces too much irrelevant information.

[0069] In one embodiment, the pixel dimensions of the segmented regions are statistically analyzed to determine the size of the sampling region. Specifically, for two materials that have spatial interaction, their segmentation masks are obtained respectively. All pixels with a value of 1 in the mask are traversed, and the ranges of their horizontal and vertical coordinates are calculated. The pixel width is obtained by subtracting the minimum value from the maximum value of the horizontal coordinate, and the pixel length is obtained by subtracting the minimum value from the maximum value of the vertical coordinate. These two parameters reflect the geometric dimensions of the materials on the image plane.

[0070] Calculate the characteristic dimensions of two materials. Characteristic dimensions can be calculated in several ways, the most common being the smaller of the pixel length and pixel width. This value represents the minimum span of the material and reflects its typical dimensional characteristics well. Alternatively, the average of the pixel length and pixel width can be used, which combines dimensional information in both directions. For materials with a near-square shape, both calculation methods yield similar results; for elongated materials, the smaller value is more representative of their characteristic dimensions.

[0071] In one embodiment, the feature size is multiplied by a preset scaling factor to obtain the side length of the sampling area. The scaling factor is typically set between 0.3 and 0.8, and the specific value can be adjusted according to the actual application scenario. If the scaling factor is set too small, the sampling area may not fully cover the interactive part of the material, resulting in insufficient depth sampling; if the scaling factor is set too large, the sampling area will contain too much background or other material information, affecting the accuracy of depth judgment. In this embodiment, the scaling factor can be set to 0.5, that is, the side length of the sampling area is half the feature size of the material.

[0072] A square sampling area is constructed centered on the midpoint of the interaction location calculated above. Specifically, let the coordinates of the midpoint of the interaction location be x0 and y0, and the side length of the sampling area be L. The coordinates of the upper left corner of the sampling area are x0 minus L divided by 2, and y0 minus L divided by 2; the coordinates of the lower right corner are x0 plus L divided by 2, and y0 plus L divided by 2. This square area, with the interaction location as its geometric center, can simultaneously cover the areas of two materials near the interaction location.

[0073] It should be noted that boundary checks are required when constructing the sampling region to ensure that the region does not exceed the image boundaries. If the calculated sampling region boundary exceeds the image boundaries, cropping is necessary to confine the excess portion within the valid image area. This typically occurs when the material is located at the edge of the image.

[0074] In one embodiment, within a square sampling area, depth data belonging to different materials is extracted based on the mask information of the segmented area. Specifically, all pixels within the sampling area are traversed. For each pixel, its pixel value in the segmentation mask of the first material is checked. If the pixel value is 1, it indicates that the point belongs to the foreground region of the first material. The depth value is extracted from the corresponding position in the depth image and added to the depth data set of the first material. Similarly, the pixel value of the point in the segmentation mask of the second material is checked. If it is 1, the depth value is extracted and added to the depth data set of the second material.

[0075] This separation and extraction process allows for the precise differentiation of depth information belonging to different materials within the same sampling area. This is because the segmentation mask provides pixel-level material attribution information, enabling accurate differentiation even when two materials are spatially close or partially overlapping. Pixels within the sampling area that do not belong to either the first or second material (i.e., background pixels) are not extracted, thus avoiding background interference.

[0076] In one embodiment, statistical analysis and outlier handling are performed on the extracted depth data. Specifically, the median and standard deviation are calculated for the depth data set of the first material. The median is obtained by taking the middle value after sorting all depth values ​​by size. This statistic is insensitive to outliers and can robustly reflect the central tendency of the depth distribution. The standard deviation is obtained by calculating the square root of the sum of squares of the deviations of each depth value from the mean, reflecting the dispersion of the depth values.

[0077] Outlier detection is performed using the 3-sigma criterion, which is based on the normal distribution theory and considers data points that deviate from the median by more than three standard deviations as outliers. Specifically, the absolute value of the deviation of each depth value from the median is calculated. If this deviation exceeds a preset multiple multiplied by the standard deviation, such as three standard deviations, the depth value is marked as an outlier. Outliers may be caused by measurement noise from the depth camera, reflections from the material surface, or depth jumps due to obstructed edges. Marked outliers are removed from the depth dataset, retaining only valid depth data.

[0078] In one embodiment, an average value is calculated for the valid depth data after outlier removal. Specifically, all valid depth values ​​are summed and divided by the number of valid data points to obtain an arithmetic mean. This average value represents the characteristic depth value of the material at the interaction location and can accurately reflect the depth information of the material in that area. The same outlier removal and average value calculation process is performed on the depth data set of the second material to obtain the depth value of the second material at the interaction location.

[0079] It should be noted that if, after outlier removal, the number of valid depth data points for a certain material is too small (e.g., less than 5 pixels), it indicates that the material's coverage area within the sampling region is very small, and depth sampling is insufficient. In this case, the sampling area can be appropriately expanded or the scaling factor adjusted, and depth data extraction can be performed again. This mechanism ensures the reliability of depth value calculation and avoids judgment errors caused by insufficient sampling.

[0080] 205. Compare the depth values, determine the upper and lower layer positions of the materials based on the relationship between the depth values, and obtain the occlusion relationship between each pair of materials; In this embodiment, after obtaining the depth values ​​of two materials at their interaction location, the material sorting control system determines the upper and lower layer positions of the materials by comparing the magnitudes of these depth values. In the depth image, the depth value represents the distance from the object's surface to the camera. A smaller depth value indicates a closer distance to the camera, meaning the object is located in an upper layer closer to the camera; a larger depth value indicates a farther distance from the camera, meaning the object is located in a lower layer further away from the camera. This determination principle is based on the camera's imaging geometry and is fundamental to 3D scene understanding.

[0081] Specifically, for material pairs with spatial interaction, the depth value D1 of the first material and the depth value D2 of the second material at the interaction location are obtained respectively. The two depth values ​​are compared. If D1 is less than D2, it means that the first material is closer to the camera, and the first material is determined to be above the second material. The occlusion relationship is recorded as the first material occluding the second material. Conversely, if D1 is greater than D2, it means that the second material is determined to be above the first material. The occlusion relationship is recorded as the second material occluding the first material.

[0082] In one embodiment, to improve the robustness of the judgment, a depth difference threshold is introduced for auxiliary judgment. The absolute value of the difference between the depth values ​​of two materials is calculated, that is, the absolute value of D1 minus D2. If the difference is greater than a preset depth difference threshold, for example, 5 mm, it indicates that the two materials have a clear layer distinction in the depth direction, and the upper and lower layer relationship can be determined definitively. If the depth difference is less than the threshold, it indicates that the two materials are very close in the depth direction, possibly on the same plane or with only a slight height difference. In this case, the overlapping area of ​​the segmented region can be further combined for auxiliary judgment.

[0083] Specifically, a region overlap analysis algorithm is used to calculate the ratio of the intersection area to the union area of ​​the segmentation masks of two materials. This ratio is called the Intersection over Union (IoU). If the IoU is greater than a preset threshold, such as 0.3, it indicates that the two materials have a large overlapping area in the image plane. Even if the depth difference is small, the occlusion relationship can be determined based on the slight difference in depth values. If the IoU is less than the threshold and the depth difference is also less than the threshold, it is determined that the two materials are spatially independent and there is no obvious occlusion relationship, and they are recorded as unrelated.

[0084] 206. Perform pairwise traversal on all materials and record the resulting occlusion relationships as a relation matrix. The elements of the relation matrix represent the relative positional state between the corresponding materials, thus obtaining the relative positional relationship between the materials. In this embodiment, the material sorting control system needs to perform a comprehensive occlusion relationship analysis on all materials in the scene to construct a complete description of their relative positions. A relational matrix data structure is used to record the occlusion relationships between materials. This matrix can clearly express the relative positional state between any two materials, facilitating subsequent grabbing sequence planning and dynamic updates.

[0085] Specifically, assuming there are N materials in the scenario, an N x N matrix is ​​created as the relationship matrix. The row and column indices of the matrix correspond to different material numbers, and the element in the i-th row and j-th column represents the relationship between the i-th material and the j-th material. A nested loop traversal algorithm is used, with the outer loop traversing material i from 1 to N, and the inner loop traversing material j from i plus 1 to N. For each pair of materials, the aforementioned boundary distance calculation, sampling region construction, depth data extraction, and occlusion relationship determination processes are performed.

[0086] The identified occlusion relationships are recorded in a relationship matrix using numerical encoding. If material i is above material j, the matrix element M[i][j] is set to 1, indicating a higher-level relationship, while the symmetrical element M[j][i] is set to -1, indicating a lower-level relationship. If the two materials have no spatial interaction or occlusion relationship, both M[i][j] and M[j][i] are set to 0, indicating no relationship. The diagonal element M[i][i] of the matrix is ​​always 0, because the material does not have an occlusion relationship with itself.

[0087] Through this pairwise traversal and encoding process, a complete relation matrix can be obtained. This matrix is ​​antisymmetric, meaning that M[i][j] is equal to the negative of M[j][i]. Each row in the matrix reflects the relationship between the material and all other materials, and each column reflects the relationship between all other materials and the material.

[0088] In one embodiment, a consistency check is performed on the relationship matrix. If material A occludes material B, and material B occludes material C, then material A should also occlude material C; that is, the occlusion relationship should be transitive. The Floyd-Warshall transitive closure algorithm is used to correct the relationship matrix, ensuring the logical consistency of the occlusion relationships. This algorithm uses three nested loops to check and supplement all indirect occlusion relationships, making the relationship matrix more complete and accurate.

[0089] 207. Based on the relative positional relationship, plan the grabbing sequence for each material, send the three-dimensional spatial coordinates and size parameters of the currently grabbable material to the robotic arm to perform the grabbing operation, and update the relative positional relationship after the grabbing is completed. Repeat the grabbing sequence planning until all materials are sorted.

[0090] In this embodiment, step 207 is similar to step 104 in the first embodiment, and will not be described again here.

[0091] In this embodiment, target detection is performed on the acquired color images of the material scene to obtain the segmented regions, location information, and category information of each material. The depth image is processed based on the segmented regions to obtain the three-dimensional spatial coordinates and size parameters of each material. The spatial relationship between the segmented regions is analyzed, and the occlusion state between materials is determined based on the depth image to obtain their relative positional relationship. Based on the relative positional relationship, a grasping sequence is planned for each material. The three-dimensional spatial coordinates and size parameters of the currently graspable material are sent to the robotic arm to execute the grasping operation. After grasping, the relative positional relationship is updated, and the grasping sequence planning is repeated until all materials are sorted. This invention avoids collisions during the grasping process by judging the occlusion relationship between materials and planning the grasping sequence accordingly, thus improving the grasping success rate and sorting efficiency.

[0092] The material sorting control method in the embodiments of the present invention has been described above. The material sorting control device in the embodiments of the present invention will be described below. Please refer to [link to relevant documentation] for details. Figure 3 One embodiment of the material sorting control device in this invention includes: The target detection module 301 is used to perform target detection on the acquired color image of the material scene to obtain the segmentation region, location information and category information of each material; The three-dimensional positioning module 302 is used to process the depth image of the material scene according to the segmented region to obtain the three-dimensional spatial coordinates and size parameters of each material. The occlusion judgment module 303 is used to analyze the spatial relationship between the segmented regions, determine the occlusion state between materials based on the depth image, and obtain the relative positional relationship between the materials. The gripping control module 304 is used to plan the gripping sequence of each material according to the relative position relationship, send the three-dimensional spatial coordinates and size parameters of the currently grippable material to the robotic arm to perform the gripping operation, and update the relative position relationship after the gripping is completed. The gripping sequence planning is repeated until all materials are sorted.

[0093] In this embodiment of the invention, the material sorting control device operates the aforementioned material sorting control method. The device performs target detection on a collected color image of the material scene to obtain the segmented regions, location information, and category information of each material. It then processes the depth image based on the segmented regions to obtain the three-dimensional spatial coordinates and size parameters of each material. The device analyzes the spatial relationships between the segmented regions, determines the occlusion state between materials based on the depth image, and obtains the relative positional relationships between materials. Based on the relative positional relationships, it plans the grasping sequence for each material, sending the three-dimensional spatial coordinates and size parameters of the currently graspable material to the robotic arm to execute the grasping operation. After grasping, the relative positional relationships are updated, and the grasping sequence planning is repeated until all materials are sorted. This invention, by determining the occlusion relationships between materials and planning the grasping sequence accordingly, avoids collisions during the grasping process, improving the grasping success rate and sorting efficiency.

[0094] above Figure 3 The material sorting control device in the embodiments of the present invention will be described in detail from the perspective of unitized functional entities. The material sorting control equipment in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0095] Figure 4 This is a schematic diagram of a material sorting control device 300 provided in an embodiment of the present invention. The material sorting control device 300 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 410 (e.g., one or more processors) and a memory 420, and one or more storage media 430 (e.g., one or more mass storage devices) for storing application programs 333 or data 432. The memory 420 and storage media 430 can be temporary or persistent storage. The program stored in the storage media 430 may include one or more units (not shown in the diagram), each unit may include a series of instruction operations on the material sorting control device 400. Furthermore, the processor 410 may be configured to communicate with the storage media 430 and execute the series of instruction operations in the storage media 430 on the material sorting control device 400 to implement the steps of the above-described material sorting control method.

[0096] The material sorting control device 400 may also include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input / output interfaces 460, and / or one or more operating systems 431, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 4The illustrated material sorting control device structure does not constitute a limitation on the material sorting control device provided by the present invention. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0097] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the material sorting control method.

[0098] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0099] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A material sorting control method, characterized in that, The material sorting control method includes: Target detection is performed on the collected color images of the material scene to obtain the segmentation region, location information and category information of each material; The depth image of the material scene is processed according to the segmented region to obtain the three-dimensional spatial coordinates and size parameters of each material; The spatial relationship between the segmented regions is analyzed, and the occlusion state between materials is determined based on the depth image to obtain the relative positional relationship between the materials. Based on the relative positional relationship, the grabbing sequence of each material is planned. The three-dimensional spatial coordinates and size parameters of the currently grabbable material are sent to the robotic arm to perform the grabbing operation. After the grabbing is completed, the relative positional relationship is updated. The grabbing sequence planning is repeated until all materials are sorted.

2. The material sorting control method according to claim 1, characterized in that, The process of performing target detection on the acquired color images of the material scene to obtain the segmentation region, location information, and category information of each material includes: Convolutional feature extraction is performed on the color image to obtain multi-layer feature maps; Linear attention calculation is performed on the multi-layer feature map. After applying non-linear activation functions to the query features and key features respectively, matrix operations are performed to obtain attention weights. Then, the value features are weighted and aggregated according to the attention weights to obtain attention-enhanced features. The attention enhancement features are weighted according to the channel dimension, and the high-level features are upsampled by transposed convolution and then fused with the low-level features to obtain multi-scale fused features. The multi-scale fused features are used for target detection and instance segmentation to obtain the bounding box coordinates, category labels, and pixel-level segmentation masks for each material. The segmentation region of each material is determined based on the segmentation mask, the position information of each material is determined based on the bounding box coordinates, and the category information of each material is determined based on the category label.

3. The material sorting control method according to claim 1, characterized in that, The process of processing the depth image of the material scene based on the segmented region to obtain the three-dimensional spatial coordinates and size parameters of each material includes: Based on the segmented region, the corresponding depth data is extracted from the depth image to obtain the depth value of each material; Calculate the conversion ratio between pixels and actual distance based on the pixel size and actual size of a pre-calibrated standard object; Based on the depth value and the conversion ratio, calculate the coordinate position of each material in three-dimensional space; Based on the pixel size of the segmented region and the conversion ratio, the actual length, width, and height of each material are calculated to obtain the size parameters of each material.

4. The material sorting control method according to claim 1, characterized in that, The step of analyzing the spatial relationship between the segmented regions and determining the occlusion state between materials based on the depth image to obtain the relative positional relationship between materials includes: The boundaries of the segmented region are extracted, and the distance relationship between the boundaries of different materials is calculated to obtain the material pairs with spatial interaction and their interaction positions. A sampling region is constructed at the interaction location. Based on the segmentation region, depth data corresponding to different materials are extracted from the sampling region to obtain the depth value of each material at the interaction location. The depth values ​​are compared, and the upper and lower layer positions of the materials are determined based on the relationship between the depth values ​​to obtain the occlusion relationship between each pair of materials. All materials are traversed pairwise, and the resulting occlusion relationships are recorded as a relationship matrix. The elements of the relationship matrix represent the relative positional state between the corresponding materials, thus obtaining the relative positional relationship between the materials.

5. The material sorting control method according to claim 4, characterized in that, The step of extracting the boundaries of the segmented region and calculating the distance relationship between the boundaries of different materials to obtain material pairs with spatial interaction and their interaction locations includes: The boundary contours of the segmented region are extracted, and the boundary contours are sampled to obtain a set of boundary sampling points; For the boundary sampling point sets of different materials, pairwise distance calculations are performed, and the nearest neighbor search algorithm is used to determine the minimum distance value and the coordinates of the corresponding nearest point pair; Based on the pre-defined conversion ratio between pixels and actual distance, the preset physical distance threshold is converted into a pixel distance threshold. The minimum distance value is compared with the pixel distance threshold. When the minimum distance value is less than the pixel distance threshold, it is determined that the pair of materials have spatial interaction, and the midpoint of the coordinates of the nearest point pair is taken as the interaction position.

6. The material sorting control method according to claim 4, characterized in that, The step of constructing a sampling region at the interaction location, and extracting depth data corresponding to different materials within the sampling region based on the segmented region, yields the depth value of each material at the interaction location, including: The pixel length and pixel width of the segmented region are statistically analyzed, and the feature size is calculated. The feature size is the smaller or average value of the pixel length and pixel width. Multiply the feature size by a preset scaling factor to obtain the side length of the sampling area, and construct a square sampling area with the interaction position as the center. Within the square sampling area, depth data belonging to different materials are extracted according to the mask information of the segmented area; Perform statistical analysis on the extracted depth data, calculate the median and standard deviation of the depth values, and mark depth values ​​that deviate from the median by more than a preset multiple of the standard deviation as outliers and remove them; The average value of the effective depth data after removing outliers is calculated to obtain the depth value of each material at the interaction position.

7. The material sorting control method according to claim 1, characterized in that, The step of planning the gripping sequence for each material based on the relative positional relationship, sending the three-dimensional spatial coordinates and size parameters of the currently grippable material to the robotic arm to execute the gripping operation, updating the relative positional relationship after gripping, and repeating the gripping sequence planning until all materials are sorted includes: Perform a row scan on the relationship matrix of the relative positional relationships, and search for materials in the matrix rows that contain only the upper-level relationship identifier and no relationship identifier. The materials found are determined as the current set of materials that can be grabbed. Select a material from the current set of graspable materials as the current grasping target, obtain the three-dimensional spatial coordinates and size parameters corresponding to the current grasping target, and send the three-dimensional spatial coordinates and size parameters to the robotic arm control system; After receiving the grasping completion signal from the robotic arm control system, the row and column elements corresponding to the current grasping target are set to no relation identifiers in the relation matrix; Determine whether there are any elements in the relationship matrix that are not set to have no relationship. If so, return to continue executing the row scanning step. If not, determine that all materials have been sorted.

8. A material sorting control device, characterized in that, The material sorting control device includes: The target detection module is used to perform target detection on the acquired color images of the material scene to obtain the segmentation region, location information and category information of each material; The three-dimensional positioning module is used to process the depth image of the material scene based on the segmented region to obtain the three-dimensional spatial coordinates and size parameters of each material. The occlusion judgment module is used to analyze the spatial relationship between the segmented regions, determine the occlusion state between materials based on the depth image, and obtain the relative positional relationship between the materials. The gripping control module is used to plan the gripping sequence of each material according to the relative position relationship, send the three-dimensional spatial coordinates and size parameters of the currently grippable material to the robotic arm to perform the gripping operation, and update the relative position relationship after the gripping is completed. The gripping sequence planning is repeated until all materials are sorted.

9. A material sorting control device, characterized in that, The material sorting control device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the material sorting control device to perform the steps of the material sorting control method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instruction is executed by the processor, it implements the steps of the material sorting control method as described in any one of claims 1-7.

Images