A cargo handling method and system based on spatial relationship reasoning

Abstract

The present invention relates to the field of intelligent devices and discloses a cargo handling method and system based on spatial relationship reasoning, comprising: obtaining a color image and a depth image containing cargo, screening valid cargo, and obtaining spatial relationship information of the valid cargo, wherein the spatial relationship information includes depth information, height information, depth difference, vertical overlap degree, and horizontal overlap degree; identifying and obtaining an unconstrained cargo set based on the depth difference, vertical overlap degree, and horizontal overlap degree of the valid cargo; prioritizing the unconstrained cargo set based on the depth information and height information of the unconstrained cargo to generate an optimal handling sequence; and driving mechanical equipment to perform actual grasping operations according to the priority order of the optimal handling sequence to complete the cargo handling operation. The present invention innovatively constructs a constraint graph model based on spatial dependency relationships to intelligently identify unconstrained targets that can be safely grasped.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and robotic control, and specifically relates to a cargo handling method and system based on spatial relationship reasoning, which is mainly used for cargo identification and grasping in automated scenarios such as logistics sorting, warehouse management, and transportation scheduling. Background Art

[0002] With the rapid development of e-commerce and logistics industries, the demand for automated loading and unloading, sorting and warehousing management of goods is increasing.

[0003] In practical applications, the main challenges faced by automated unloading robots are: first, cargo is often stacked in an irregular manner, leading to severe occlusion; second, the robots need to determine a reasonable grasping sequence to avoid cargo collapse or damage during unloading. For example, in tobacco logistics scenarios, when a batch of tobacco packaging boxes are randomly stacked in a container or truck, traditional object detection algorithms can only identify the visible surface of the packaging boxes and cannot infer the shape and position of the obscured parts, making it even more difficult to determine a safe and feasible grasping sequence.

[0004] Existing methods lack effective modeling of the spatial dependencies between objects and are unable to determine which boxes can be grasped first without causing instability in other boxes. Summary of the Invention

[0005] The main purpose of the present invention is to solve the technical problems of insufficient target recognition accuracy and unreasonable grasping sequence planning in complex environments in the existing technology. A cargo handling method based on spatial relationship reasoning includes the following steps:

[0006] Acquire a color image and a depth image containing goods, screen valid goods, and obtain spatial relationship information of the valid goods, wherein the spatial relationship information includes depth information, height information, depth difference, vertical overlap degree, and horizontal overlap degree;

[0007] According to the depth difference, vertical overlap and horizontal overlap of valid cargoes, the unconstrained cargo set is identified and obtained;

[0008] Generate the optimal transport sequence based on the unconstrained cargo collection and priority sorting based on the depth and height information of the unconstrained cargo;

[0009] According to the priority order of the optimal handling sequence, the mechanical equipment is driven to perform the actual grabbing operation to complete the cargo handling operation.

[0010] The step of obtaining a color image and a depth image containing goods and screening spatial relationship information of valid goods includes:

[0011] An RGB-D camera is used to collect input data to obtain a color image and a depth image containing the goods. The color image and the depth image are aligned using a calibration matrix;

[0012] The ResNet-101 network is used as the backbone network of the RT-DETR model to extract multi-scale features of color images;

[0013] Build a high-quality cargo packaging dataset, collect cargo packaging images at multiple scales and poses, and perform professional annotation to distinguish front and side views, as well as stacking relationships;

[0014] Based on a high-quality cargo packaging dataset, we fine-tune the RT-DETR model to enable it to recognize the bounding boxes and spatial topological relationships of cargo packaging.

[0015] Use the fine-tuned RT-DETR model to perform target detection on color images and output the category and bounding box information of candidate targets;

[0016] Based on the bounding box information and depth image, the depth standard deviation, relative area ratio, and aspect ratio error of each bounding box are calculated, and valid goods are screened based on the depth standard deviation, relative area ratio, and aspect ratio error.

[0017] The method of calculating the depth standard deviation, relative area ratio, and aspect ratio error of each bounding box based on the bounding box information and the depth image, and screening valid goods based on the depth standard deviation, relative area ratio, and aspect ratio error, includes:

[0018] In each detected bounding box Randomly sample depth values ​​and calculate robust standard deviation , used to distinguish the front and side of the goods:

[0019]

[0020] Where D is the depth map, n is the number of sampling points, and SampleDepth is a function that randomly samples depth values ​​within the bounding box. RobustStd uses the interquartile range method to filter outliers and then calculates the standard deviation:

[0021]

[0022] Where Q1 and Q3 represent the first and third quartiles respectively, IQR=Q3-Q1; StdDev represents the standard deviation;

[0023] Set the depth standard deviation threshold , filter the detection boxes that exceed this threshold;

[0024] Calculate each bounding box The relative area ratio , remove boxes that are beyond a reasonable range:

[0025]

[0026] in, , H and W are the image height and width respectively, is the area ratio threshold;

[0027] Based on the preset physical dimensions of the cargo Calculating aspect ratio error :

[0028]

[0029] in, Indicates the physical size ratio, is the shape error threshold;

[0030] Apply the intersection-over-union threshold to remove overlapping frames and retain high-quality, low-duplication valid frames:

[0031]

[0032] in, is the IoU threshold, Represents two bounding boxes When multiple highly overlapping boxes are detected, the system retains the one with the highest confidence.

[0033] After the screening step, a valid detection frame is obtained;

[0034] The step of identifying and obtaining an unconstrained cargo set based on the depth difference, vertical overlap degree, and horizontal overlap degree of the valid cargo includes:

[0035] For each pair of goods (i, j), the bounding box , bounding box , judged by the following three conditions Is it right Forming support constraints:

[0036]

[0037] Condition 1 is the depth difference condition, which requires that the depth difference between the two cargoes does not exceed the threshold δ depth,j , the threshold is dynamically calculated based on the size of the packaging box, z i 、z j is the depth of goods i and j; condition 2 is the vertical overlap condition, which measures the degree of overlap between two goods in the vertical direction and requires that it does not exceed the threshold δ y , threshold δ yThe value ranges from 0.3 to 0.9. Condition 3 is the horizontal overlap condition, which measures the degree of overlap between two goods in the horizontal direction and is required to be no less than the threshold δ. x , threshold δ x The value ranges from 0.01 to 0.5. Only when the constraints of conditions 1, 2, and 3 are met at the same time, is it considered that the item i forms a support constraint on the item j.

[0038] Initially, the constraint degree deg[i] of all goods is 0. By traversing all possible pairs of goods, the number of support constraints imposed on each good is accumulated to obtain the constraint degree deg[i].

[0039] The method of generating an optimal transport sequence based on the unconstrained cargo set and prioritizing the unconstrained cargo based on depth information and height information includes:

[0040] The unconstrained goods set T is:

[0041]

[0042] In the formula, N represents the total number of valid goods detected, deg[i] represents the constraint degree of goods i, and a constraint degree of zero means that the valid goods are not supported or blocked by other valid goods and can be safely transported;

[0043] Construct a sorting function based on spatial location features:

[0044]

[0045] in the formula is the depth value of cargo i, is the height of cargo i; the sorting function ensures that valid cargo in front is moved first, and that valid cargo above is moved first when the depths are similar;

[0046] Generate the final transport sequence:

[0047]

[0048] In the formula, Sort represents the ascending sorting operation based on the sorting function f; key represents the sorting key function. As valid goods are moved one by one, the system needs to dynamically update the constraint relationship diagram; every time a valid goods is removed, the related constraint relationship is also eliminated.

[0049] According to the priority order of the optimal handling sequence, the mechanical equipment is driven to perform the actual grabbing operation to complete the cargo handling operation, specifically:

[0050] Perform dynamic temporal stability processing on the unconstrained cargo set to obtain a stable detection frame set;

[0051] Based on the stable set of detection boxes, two special lists are constructed: the pre-grab list and the grab list, which are used to track the stability of the target and select the final grab object;

[0052] The pre-grab list records all detected valid goods and their appearance history. When a valid goods appears a sufficient number of times within the time window and its position is stable, it is considered a stable target and is eligible to be added to the grab list.

[0053] The grab list contains at most two stable targets with the highest priority, sorted by depth. The first target in the list, TAKE, indicates the primary target to be grabbed at the moment, and the second target, READY, indicates the next target to be grabbed. The status of the targets in the grab list is dynamically monitored. If a target disappears for more than a set threshold, it is removed from the list.

[0054] According to the generated grab list, the robot arm is driven to perform the actual grab operation; for the first target TAKE in the grab list, the grab point position is first planned to complete the cargo handling operation.

[0055] A second aspect of the present invention provides a cargo handling system based on spatial relationship reasoning, comprising:

[0056] a spatial relationship information acquisition unit, configured to acquire a color image and a depth image containing goods, screen valid goods, and obtain spatial relationship information of the valid goods, wherein the spatial relationship information includes depth information, height information, depth difference, vertical overlap degree, and horizontal overlap degree;

[0057] An unconstrained cargo identification unit is used to identify and obtain an unconstrained cargo set based on the depth difference, vertical overlap degree, and horizontal overlap degree of valid cargo;

[0058] The handling sequence production unit is used to generate the optimal handling sequence based on the unconstrained cargo collection and priority sorting based on the depth information and height information of the unconstrained cargo;

[0059] The handling unit drives the mechanical equipment to perform the actual grabbing operation according to the priority order of the optimal handling sequence to complete the cargo handling operation.

[0060] The third aspect of the present invention provides an electronic device, comprising: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected via lines; the at least one processor calls the instructions in the memory so that the electronic device executes the above-mentioned cargo handling method based on spatial relationship reasoning.

[0061] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored therein, which, when executed on a computer, enables the computer to execute the cargo handling method based on spatial relationship reasoning as described above.

[0062] The present invention has the following beneficial effects:

[0063] 1. This invention utilizes a highly efficient target detection algorithm and multi-scale feature fusion technology, and fine-tunes the model specifically for tobacco packaging scenarios, achieving high-precision, real-time detection of standard targets. This method significantly improves processing speed while maintaining high detection accuracy, meeting the real-time requirements of logistics scenarios and enhancing the overall operational efficiency of the system.

[0064] 2. This invention combines depth information and geometric features to design a comprehensive bounding box filtering and stabilization mechanism. Through multiple screening criteria, including depth standard deviation analysis, area ratio verification, aspect ratio matching, and temporal consistency constraints, it effectively improves the accuracy and stability of object detection, particularly under complex lighting conditions and partially occluded scenes, and reduces false positives and missed detections.

[0065] 3. This invention innovatively constructs a constraint graph model based on spatial dependencies. By analyzing the support relationships and spatial topology between packaging boxes, it intelligently identifies unconstrained objects that can be safely grasped. This approach not only avoids the risk of cargo collapse caused by improper grasping sequence, but also optimizes the overall unloading process, improving system safety and operational efficiency.

[0066] 4. This invention incorporates a dynamic target tracking and capture queue management mechanism that monitors target state changes in real time and adjusts capture strategies accordingly. Through a two-tiered management structure consisting of a pre-capture list and a capture list, the system can flexibly respond to complex situations such as target appearance, disappearance, and position changes while ensuring decision stability, enhancing the system's adaptability in changing environments.

[0067] 5. This invention adopts a modular design approach, with clear interfaces and logical independence between functional modules, facilitating system maintenance and functional expansion. Furthermore, the entire system can be implemented using low-cost equipment such as common RGB-D cameras, eliminating the need for expensive specialized equipment. This significantly reduces deployment costs, improving the system's economic viability and promotional value. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a flow chart of the spatial relationship information of effective goods in the present invention.

[0069] Figure 2 This is a flow chart of the constraint degree calculation based on spatial relationship information of the present invention. DETAILED DESCRIPTION

[0070] The terms "first," "second," "third," "fourth," and so on (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar items and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that shown or described herein. In addition, the terms "including," "comprising," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus.

[0071] Depth Image

[0072] A depth image is a special type of image generated by an RGB-D camera. Each pixel represents the distance of an object from the camera (rather than its color). Depth images are typically single-channel (grayscale), with each pixel representing the vertical distance from the camera in three-dimensional space (usually in millimeters or meters). For example, if an item is 0.5 meters from the camera, the corresponding pixel might appear dark gray; if it is 1 meter away, the pixel might appear light gray (the specific mapping method depends on the camera configuration).

[0073] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 and 2 In the first embodiment of the cargo handling method based on spatial relationship reasoning in the embodiment of the present invention, the method includes:

[0074] Acquire a color image and a depth image containing goods, screen valid goods, and obtain spatial relationship information of the valid goods, wherein the spatial relationship information includes depth information, height information, depth difference, vertical overlap degree, and horizontal overlap degree;

[0075] According to the depth difference, vertical overlap and horizontal overlap of valid cargoes, the unconstrained cargo set is identified and obtained;

[0076] Generate the optimal transport sequence based on the unconstrained cargo collection and priority sorting based on the depth and height information of the unconstrained cargo;

[0077] According to the priority order of the optimal handling sequence, the mechanical equipment is driven to perform the actual grabbing operation to complete the cargo handling operation.

[0078] In the embodiment of the present invention, a total of five pieces of information are obtained: depth information, height information, depth difference, vertical overlap degree, and horizontal overlap degree.

[0079] Depth difference, vertical overlap, and horizontal overlap are used to determine which goods can be handled first, while depth information and height information are used to determine the handling order. Specifically, the following solutions can be adopted:

[0080] like Figure 2 , for each pair of goods (i, j), construct the constraint P(i, j):

[0081]

[0082] Among them, condition 1 represents the depth difference condition, that is, the depth difference between the two goods does not exceed the threshold δ depth,j Condition 2 represents the vertical overlap condition, which measures whether the degree of overlap between two goods in the vertical direction does not exceed the threshold δ x Condition 3 represents the horizontal overlap condition, which measures whether the overlap between two goods in the horizontal direction is less than the threshold δ x ;

[0083] Only when the constraints of conditions 1, 2, and 3 are met at the same time, is it considered that cargo i forms a support constraint on cargo j;

[0084] Initially, the constraint degree deg[i] of all goods is 0. By traversing all possible pairs of goods, the number of support constraints imposed on each good is accumulated to obtain the constraint degree deg[i].

[0085] To better explain the present invention, taking tobacco goods as an example, the embodiments of the present invention are as follows:

[0086] Obtain color and depth images of the tobacco cargo scene and detect the cargo using the Transformer-based real-time object detection model RT-DETR. The RT-DETR model includes a ResNet-101 backbone network, an efficient hybrid encoder, and a Transformer decoder.

[0087] Filter and optimize the bounding boxes of the detection results, calculate feature parameters such as depth standard deviation, area ratio, and aspect ratio error for each detection box, and select valid goods based on multi-feature threshold conditions;

[0088] Construct a spatial constraint relationship diagram between goods, calculate the degrees of freedom of each goods, prioritize unconstrained goods based on depth and height information, and generate the optimal handling sequence;

[0089] Perform dynamic temporal stability processing to achieve stable multi-target dynamic scheduling by comparing detection results between adjacent frames, retaining high-quality new detection frames and maintaining historical valid frames within the time window.

[0090] The detailed steps are:

[0091] Step 1: Use an image sensor to acquire color and depth images of tobacco products and extract multi-scale features using a pre-trained convolutional backbone network.

[0092] Step 2: Use the RT-DETR model to detect objects in the image, which includes an efficient hybrid encoder for feature fusion and global representation, a query selection strategy based on uncertainty minimization, and a Transformer decoder to iteratively update the object query and output the category and bounding box information of the candidate object;

[0093] Step 3: Build a high-quality tobacco packaging dataset, collect multi-scale and multi-pose tobacco packaging images, and perform professional annotation to distinguish the front, side, and stacking relationships;

[0094] Step 4: Based on the constructed dataset, a specific training strategy is used to fine-tune the RT-DETR model so that it can accurately identify the bounding boxes and spatial topological relationships of tobacco packaging boxes.

[0095] Step 5: Perform screening and optimization on the detection boxes, calculate features such as depth standard deviation, relative area ratio, and aspect ratio error for each bounding box, and select valid goods based on multi-feature threshold conditions;

[0096] Step 6: Construct a spatial constraint relationship diagram between goods, determine the constraint relationship between goods based on conditions such as depth difference, vertical overlap, and horizontal overlap, and cumulatively calculate the constraint degree of each goods;

[0097] Step 7: Identify the unconstrained cargo set with zero constraint degree, prioritize it based on depth and height information, and generate the optimal handling sequence;

[0098] Step 8: Drive the mechanical equipment to perform the actual grabbing operation according to the priority order to complete the handling of the tobacco goods.

[0099] Step 2 includes more details: efficient fusion and global representation of the multi-scale features extracted in step 1. The efficient hybrid encoder of the RT-DETR model includes:

[0100] Attention-based intra-scale feature interaction module (AIFI) only performs attention on the highest-level features. Perform the self-attention operation, which is calculated by the following steps:

[0101]

[0102] Among them, Q: query matrix; K: key matrix; V: value matrix; Flatten: matrix flattened into sequence.

[0103] The CNN-based cross-scale feature fusion module (CCFF) fuses multi-scale features S3, S4, and F5 step by step through convolutional layers and RepBlock components to obtain the final output feature O:

[0104]

[0105] Among them, the efficient hybrid encoder significantly reduces the overall computation and improves the inference speed by applying the self-attention operation only to the highest-level features instead of redundantly performing it across all scales;

[0106] Based on the query selection strategy of uncertainty minimization, the first K high-quality features are selected as the initialization of the target query. The uncertainty function is expressed as:

[0107]

[0108] Where P represents the position distribution and C represents the classification distribution. This strategy ensures that features with high classification and positioning quality are selected at the same time; Represents D-dimensional real number space.

[0109] The Transformer decoder is used to iteratively update the target query and output the category and bounding box information of the candidate target to further improve the positioning accuracy.

[0110] Step 3 includes more details: building a high-quality dataset suitable for the tobacco industry, including:

[0111] The data collection scenario was set up by using tobacco packaging boxes of different sizes in a simulated environment to simulate various placement methods, including neat placement, staggered placement, and scattered placement, and adjusting the ambient lighting conditions;

[0112] Data collection and screening: Collect tobacco packaging box images from multiple angles and lighting conditions, select high-quality data, and remove tilted, blurred, or duplicated images;

[0113] Data labeling: bounding box annotation of the filtered images according to unified rules. Labels include multiple categories such as front, side, and tape segmentation, and construct a dataset that conforms to the COCO format.

[0114] Step 4 in more detail involves adopting a specific training strategy for the RT-DETR model, including:

[0115] Based on a pre-trained model checkpoint on the COCO dataset, fine-tuned for a custom tobacco box object detection scenario. A batch size of 12 was used for training, with gradient clipping (maximum norm limited to 0.1) used to ensure stable training.

[0116] In terms of data enhancement, random color perturbation, random amplification and random cropping based on IoU are used;

[0117] The AdamW optimizer is used with an initial learning rate of 1e -4 , the learning rate of the backbone network layer is set to 1e -5 , combined with 2000 iterations of linear warm-up;

[0118] Adopting joint optimization objectives:

[0119]

[0120]

[0121] Step 5 includes more details: filtering the category information and target detection box returned in step 2, including:

[0122] For each bounding box B i Randomly sample depth values ​​and calculate robust standard deviation to distinguish the front and side of the goods:

[0123]

[0124] Where D represents the depth map, n represents the number of sampling points, and RobustStd represents the standard deviation calculated after filtering outliers using the interquartile range (IQR) method;

[0125] Calculate the relative area ratio of each bounding box and remove boxes that are out of a reasonable range:

[0126]

[0127] in , H and W are the image height and width respectively, is the area ratio threshold;

[0128] Based on the preset physical dimensions of the cargo Calculate the aspect ratio error:

[0129]

[0130] in is the aspect ratio error threshold, which is calculated to ensure that the size ratio of the detection box is consistent with the actual product;

[0131] The intersection-over-union (IoU) threshold is applied to remove overlapping frames, retaining high-quality, low-duplication valid frames, and further improving the accuracy and reliability of detection.

[0132] Step 6 includes more details: judging the constraint relationship of the goods selected in step 5:

[0133]

[0134] Condition 1 is the depth difference condition, which requires that the depth difference between the two cargoes does not exceed the threshold δ depth,j , the threshold is dynamically calculated based on the size of the packaging box, z i 、z j is the depth of goods i and j; condition 2 is the vertical overlap condition, which measures the degree of overlap between two goods in the vertical direction and requires that it does not exceed the threshold δ y , threshold δ y The value ranges from 0.3 to 0.9. Condition 3 is the horizontal overlap condition, which measures the degree of overlap between two goods in the horizontal direction and is required to be no less than the threshold δ. x , threshold δ x The value ranges from 0.01 to 0.5. Only when the constraints of conditions 1, 2, and 3 are met at the same time, is it considered that the item i forms a support constraint on the item j.

[0135] Initially, the constraint degree deg[i] of all goods is 0. By traversing all possible pairs of goods, the number of support constraints imposed on each good is accumulated to obtain the constraint degree deg[i].

[0136] Construct the final set of goods pairs C based on the constraint condition P:

[0137]

[0138] For each pair of goods (i, j), if , then execute deg[i]=deg[i]+1;

[0139] This constraint diagram accurately reflects the spatial dependencies between goods, providing a basis for subsequent safe and efficient handling sequence planning.

[0140] Step 7 includes in more detail: prioritizing the directed acyclic constraint relationships constructed in step 6:

[0141] Identify the unconstrained goods set T based on the constraint degree:

[0142]

[0143] Where N represents the total number of valid goods detected, and a constraint degree of zero means that the goods are not supported or blocked by other goods and can be safely transported;

[0144] Construct a sorting function f based on depth (ascending) and height (descending):

[0145]

[0146] where z i is the depth value of cargo i, yi is the height of cargo i. This sorting function ensures that cargo in the front (smaller depth) is moved first, and cargo above (larger height) is moved first when the depths are similar.

[0147] Generate the final transport sequence S:

[0148]

[0149] Sort represents an ascending sorting operation based on the sorting function f, and key represents the sorting key function. This sorting strategy prioritizes cargo that is shallow (small z value) and high (large y value) for transportation, while meeting safety constraints. This meets the safety and efficiency requirements of actual operations.

[0150] The above describes the cargo handling method based on spatial relationship reasoning in an embodiment of the present invention. The following describes the cargo handling device based on spatial relationship reasoning in an embodiment of the present invention:

[0151] a spatial relationship information acquisition unit, configured to acquire a color image and a depth image containing goods, screen valid goods, and obtain spatial relationship information of the valid goods, wherein the spatial relationship information includes depth information, height information, depth difference, vertical overlap degree, and horizontal overlap degree;

[0152] An unconstrained cargo identification unit is used to identify and obtain an unconstrained cargo set based on the depth difference, vertical overlap degree, and horizontal overlap degree of valid cargo;

[0153] The handling sequence production unit is used to generate the optimal handling sequence based on the unconstrained cargo collection and priority sorting based on the depth information and height information of the unconstrained cargo;

[0154] The handling unit drives the mechanical equipment to perform the actual grabbing operation according to the priority order of the optimal handling sequence to complete the cargo handling operation.

[0155] Four scenario cases are given below to more intuitively demonstrate the effects and advantages of the present invention:

[0156] Scenario 1: Single-layer stacking of standard tobacco packaging boxes

[0157] Using the aforementioned solution, in a single-layer stacking environment of standard tobacco packaging boxes, the system successfully classified the detection frame into three states: FREE (free to grasp), READY (next ready to grasp), and RESTRICTED (constrained and ungraspable). As shown in Table 2, in 10 grasping tests under difficult scenarios, the system's path selection contrast ratio reached 89.6%, indicating that the system was able to effectively identify the optimal grasping path and avoid the risk of cargo collapse caused by improper grasping sequence. At the same time, READY safety reached 100%, indicating that the system's judgment of the next target to be grasped fully complies with safety principles, ensuring the continuity of the grasping process.

[0158] Table 1: Evaluation indicators and descriptions of automatic unloading robot systems

[0159]

[0160] Table 2: Comparison of experimental results of automatic unloading robots based on dynamic spatial relationship reasoning

[0161]

[0162] Second scenario: Multi-layer staggered stacking of tobacco packaging boxes

[0163] Using the aforementioned solution, in a complex environment with multiple layers of staggered packaging, as shown in the second row of Table 2, the system achieved a path selection contrast ratio of 90% in the difficult test, while both the false box detection rate and the false box entry rate reached 100%, demonstrating a significant improvement in the system's ability to identify false targets in complex environments. Furthermore, the take accuracy rate reached 90%, demonstrating that the proposed spatial relationship reasoning algorithm can accurately construct support constraint relationships between packaging boxes.

[0164] Scenario 3: System adaptability testing in environments of varying difficulty

[0165] This scenario compares the system's performance in environments of varying difficulty. Data shows that as the scenario difficulty decreases from difficult to medium and then easy, the system's key performance indicators significantly improve. At the easy level, the false box detection rate reaches 70%, the fake box entry rate and READY safety both maintain 100%, and the take accuracy rate and orchestration accuracy rates both reach 100% and 80%. In particular, the no-take rate drops to 20%, indicating that the system rarely fails to identify a graspable object.

[0166] Scenario 4: Comprehensive performance testing in actual application environments

[0167] Using the aforementioned solution, in a moderately challenging environment close to real-world applications, as shown in the last row of Table 2, the system achieved a 90% false box detection rate across 10 picking tests, maintained a 100% false box entry rate, and achieved both take and arrangement accuracy rates exceeding 90%. These metrics demonstrate that when deployed in a real-world logistics environment, the system can effectively distinguish between RESTRICTED boxes and graspable objects, ensuring that the system makes grasping decisions based on reasonable spatial constraints.

[0168] An embodiment of the present invention further provides an electronic device that may vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) (e.g., one or more processors) and memory, and one or more storage media (e.g., one or more mass storage devices) for storing applications or data. The memory and storage media may be either transient or persistent storage. The program stored on the storage medium may include one or more modules, each of which may include a series of instruction operations on the electronic device. Furthermore, the processor may be configured to communicate with the storage medium and execute the series of instruction operations on the storage medium on the electronic device.

[0169] The electronic device may further include one or more power supplies, one or more wired or wireless network interfaces, one or more input and output interfaces, and / or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that the electronic device structure in this embodiment does not constitute a limitation to electronic devices, and may include more or fewer components, or combine certain components, or arrange the components differently.

[0170] An embodiment of the present invention provides a structure of an electronic device, which may vary significantly due to different configurations or performance, and may include one or more processors (central processing units, CPUs) (for example, one or more processors) and memories, and one or more storage media for storing applications or data (for example, one or more mass storage devices). The memories and storage media may be either transient or persistent storage. The program stored in the storage medium may include one or more modules, each of which may include a series of instruction operations on the electronic device. Furthermore, the processor may be configured to communicate with the storage medium to execute the series of instruction operations in the storage medium on the electronic device.

[0171] The electronic device may further include one or more power supplies, one or more wired or wireless network interfaces, one or more input and output interfaces, and / or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that the electronic device structure does not constitute a limitation on the electronic device, and may include more or fewer components than those described above, or combine certain components, or arrange the components differently.

[0172] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to execute the steps of the aforementioned method.

[0173] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0174] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0175] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cargo handling method based on spatial relationship reasoning, characterized in that: The following steps are involved: Acquire a color image and a depth image containing goods, screen valid goods, and obtain spatial relationship information of valid goods, wherein the spatial relationship information includes depth information, height information, depth difference, vertical overlap degree, and horizontal overlap degree, including: A fine-tuned RT-DETR model is used to detect objects in color images, outputting the categories and bounding box information of candidate objects. Based on the bounding box information and depth image, the depth standard deviation, relative area ratio, and aspect ratio error of each bounding box are calculated. Valid objects are then screened based on the depth standard deviation, relative area ratio, and aspect ratio errors. Based on the depth difference, vertical overlap, and horizontal overlap of valid cargoes, an unconstrained cargo set is identified, including: For each pair of goods (i, j), the bounding box , bounding box , determine whether i forms a support constraint on j through the following three conditions: Condition 1 is the depth difference condition, which requires that the depth difference between the two cargoes does not exceed the threshold δ depth,j , the threshold is dynamically calculated based on the size of the packaging box, z i 、z j is the depth of goods i and j; condition 2 is the vertical overlap condition, which measures the degree of overlap between two goods in the vertical direction and requires that it does not exceed the threshold δ y , threshold δ y The value ranges from 0.3 to 0.

9. Condition 3 is the horizontal overlap condition, which measures the degree of overlap between two goods in the horizontal direction and is required to be no less than the threshold δ. x , threshold δ x The value ranges from 0.01 to 0.

5. Only when the constraints of conditions 1, 2, and 3 are met at the same time, is it considered that the item i forms a support constraint on the item j. Initially, the constraint degree deg[i] of all goods is 0. By traversing all possible pairs of goods and accumulating the number of support constraints on each good, the constraint degree deg[i] of all goods is obtained, and then the unconstrained goods set of goods with constraint degree deg[i] of 0 is obtained; Generate the optimal transport sequence based on the unconstrained cargo collection and priority sorting based on the depth and height information of the unconstrained cargo; including: In each detected bounding box Randomly sample depth values ​​and calculate robust standard deviation , used to distinguish the front and side of the goods: Where D is the depth map, n is the number of sampling points, and SampleDepth is a function that randomly samples depth values ​​within the bounding box. RobustStd uses the interquartile range method to filter outliers and then calculates the standard deviation: Where Q1 and Q3 represent the first and third quartiles respectively, IQR=Q3-Q1; StdDev represents the standard deviation; Set the depth standard deviation threshold , filter the detection boxes that exceed this threshold; Calculate each bounding box The relative area ratio , remove boxes that are beyond a reasonable range: in, , H and W are the image height and width respectively, is the area ratio threshold; Based on the preset physical dimensions of the cargo Calculating aspect ratio error : in, Indicates the physical size ratio, is the shape error threshold; Apply the intersection-over-union threshold to remove overlapping frames and retain high-quality, low-duplication valid frames: in, is the IoU threshold, Represents two bounding boxes When multiple highly overlapping boxes are detected, the system retains the one with the highest confidence. After the screening step, a valid detection frame is obtained; According to the priority order of the optimal handling sequence, the mechanical equipment is driven to perform the actual grabbing operation to complete the cargo handling operation.

2. A cargo handling method based on spatial relationship reasoning according to claim 1, characterized in that: The screening of valid goods also includes: An RGB-D camera is used to collect input data to obtain a color image and a depth image containing the goods. The color image and the depth image are aligned using a calibration matrix; The ResNet-101 network is used as the backbone network of the RT-DETR model to extract multi-scale features of color images; Build a high-quality cargo packaging dataset, collect cargo packaging images at multiple scales and poses, and perform professional annotation to distinguish front and side views, as well as stacking relationships; Based on the constructed high-quality cargo packaging dataset, the RT-DETR model is fine-tuned to enable it to recognize the bounding boxes and spatial topological relationships of cargo packaging.

3. A cargo handling method based on spatial relationship reasoning according to claim 1, characterized in that: The method of generating an optimal transport sequence based on the unconstrained cargo set and prioritizing the unconstrained cargo based on depth information and height information includes: The unconstrained goods set T is: In the formula, N represents the total number of valid goods detected, deg[i] represents the constraint degree of goods i, and a constraint degree of zero means that the valid goods are not supported or blocked by other valid goods and can be safely transported; Construct a sorting function based on spatial location features: in the formula is the depth value of cargo i, is the height of cargo i; the sorting function ensures that valid cargo in front is moved first, and that valid cargo above is moved first when the depths are similar; Generate the final transport sequence: In the formula, Sort represents the ascending sorting operation based on the sorting function f; key represents the sorting key function. As valid goods are moved one by one, the system needs to dynamically update the constraint relationship diagram; every time a valid goods is removed, the related constraint relationship is also eliminated.

4. A cargo handling method based on spatial relationship reasoning according to claim 3, characterized in that: According to the priority order of the optimal handling sequence, the mechanical equipment is driven to perform the actual grabbing operation to complete the cargo handling operation, specifically: Perform dynamic temporal stability processing on the unconstrained cargo set to obtain a stable detection frame set; Based on the stable set of detection boxes, two special lists are constructed: the pre-grab list and the grab list, which are used to track the stability of the target and select the final grab object; The pre-grab list records all detected valid goods and their appearance history. When a valid goods appears a sufficient number of times within the time window and its position is stable, it is considered a stable target and is eligible to be added to the grab list. The grab list contains at most two stable targets with the highest priority, sorted by depth. The first target in the list, TAKE, indicates the primary target to be grabbed at the moment, and the second target, READY, indicates the next target to be grabbed. The status of the targets in the grab list is dynamically monitored. If a target disappears for more than a set threshold, it is removed from the list. According to the generated grab list, the robot arm is driven to perform the actual grab operation; for the first target TAKE in the grab list, the grab point position is first planned to complete the cargo handling operation.

5. A cargo transport system based on the cargo transport method according to claim 1, characterized in that: The system comprises: a spatial relationship information acquisition unit, configured to acquire a color image and a depth image containing goods, screen valid goods, and obtain spatial relationship information of the valid goods, wherein the spatial relationship information includes depth information, height information, depth difference, vertical overlap degree, and horizontal overlap degree; An unconstrained cargo identification unit is used to identify and obtain an unconstrained cargo set based on the depth difference, vertical overlap degree, and horizontal overlap degree of valid cargo; The handling sequence production unit is used to generate the optimal handling sequence based on the unconstrained cargo collection and priority sorting based on the depth information and height information of the unconstrained cargo; The handling unit drives the mechanical equipment to perform the actual grabbing operation according to the priority order of the optimal handling sequence to complete the cargo handling operation.

6. An electronic device comprising a memory and at least one processor, characterized in that: Instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the electronic device to execute each step of the cargo handling method based on spatial relationship reasoning as described in any one of claims 1 to 4.

7. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the steps of the cargo handling method based on spatial relationship reasoning as described in any one of claims 1 to 4 are implemented.

Images