Container depositing method, device, equipment, medium and program product

Abstract

The application provides a container stacking method, device, equipment, medium and program product. The method comprises the following steps: acquiring a first video stream, the first video stream being collected by a multi-view industrial camera array arranged at multiple positions of a spreader; processing the first video stream by using a bimodal perception network to obtain key points of multiple objects and pixel-level segmentation masks of the multiple objects in the first video stream; calculating a bottom corner point of the spreader and a bottom corner point of a container gripped by the spreader based on the key points of the spreader, the key points of the container gripped by the spreader and the pixel-level segmentation masks of the multiple objects; calculating a translation adjustment probability and a rotation adjustment probability of the spreader according to the bottom corner point of the spreader, a top corner point of a container to be stacked and the bottom corner point of the container gripped by the spreader; and performing a container stacking operation on the container gripped by the spreader according to the translation adjustment probability and the rotation adjustment probability of the spreader. The application can meet the accuracy requirement for alignment accuracy in a real stacking process.

Description

Technical Field

[0001] This invention relates to the field of visual inspection technology, and in particular to a container placement method, apparatus, equipment, medium, and program product. Background Technology

[0002] With the development of artificial intelligence technology, cost and efficiency have become the core competitiveness of enterprises. There is an urgent need for the unmanned transformation of a large number of repetitive and process-oriented operations in closed scenarios such as ports and terminals, especially in the container loading and stacking process. Currently, the manual operation of rubber-tired cranes is inefficient and requires dedicated personnel to operate. Intelligent upgrading is imperative.

[0003] Existing technical solutions for container handling using tire-mounted cranes in ports, such as the automated container stacking method using a vision system, involve installing and calibrating industrial cameras on the spreader, repeatedly acquiring images at different heights on the spreader, using template matching algorithms to detect container corners, calculating relative positional deviations, and then inputting the deviation data into the control system to achieve automatic alignment. However, this solution lacks sufficient optimization granularity, resulting in alignment accuracy failing to meet requirements during actual container stacking. Summary of the Invention

[0004] The purpose of this invention is to provide a container stacking method, apparatus, equipment, medium, and program product to solve the problem that the alignment accuracy cannot meet the accuracy requirements in the actual stacking process due to insufficient optimization granularity in existing container stacking methods.

[0005] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a container placement method, comprising: The first video stream is acquired by a multi-view industrial camera array set up at multiple locations on the lifting device. Using a dual-modal perception network, the first video stream is processed to obtain key points of multiple objects in the first video stream and pixel-level segmentation masks of the multiple objects; wherein, the multiple objects include a spreader, a container grasped by the spreader, and a container to be stacked; the key point of the container to be stacked is the top corner point of the container to be stacked. Based on the key points of the spreader, the key points of the container grabbed by the spreader, and the pixel-level segmentation mask of the multiple objects, the bottom corner points of the spreader and the bottom corner points of the container grabbed by the spreader are calculated. Based on the bottom corner of the spreader, the top corner of the container to be stacked, and the bottom corner of the container grabbed by the spreader, the translational adjustment probability and the rotational adjustment probability of the spreader are calculated; and the container grabbing by the spreader is placed into the container according to the translational adjustment probability and the rotational adjustment probability of the spreader.

[0006] In some embodiments, processing the first video stream using a dual-modal sensing network to obtain key points of multiple objects in the first video stream and pixel-level segmentation masks of the multiple objects includes: Using multiple convolutional layers in the dual-modal perception network, multi-scale feature extraction is performed on the first video stream to obtain feature maps at multiple scales, and the feature maps at multiple scales are fused to obtain fused features; Using the pose head in the dual-modal perception network, regression processing is performed on the fused features to obtain the key points of the multiple objects; Using the segmentation head in the dual-modal perception network, regression processing is performed on the fused features to obtain pixel-level segmentation masks for the multiple objects.

[0007] In some embodiments, calculating the bottom corner points of the spreader and the bottom corner points of the container gripped by the spreader based on the key points of the spreader, the key points of the container gripped by the spreader, and the pixel-level segmentation mask of the plurality of objects includes: Based on the pixel-level segmentation mask of the multiple objects, obtain the pixel points belonging to the pixel region of the spreader and the pixel points belonging to the pixel region of the container grabbed by the spreader; Edge detection processing is performed on the pixels belonging to the pixel area of ​​the spreader and the pixels belonging to the pixel area of ​​the container grasped by the spreader to obtain a set of contour points. The set of contour points includes the inner and outer contour point sets of the spreader and the inner and outer contour point sets of the container grasped by the spreader. Based on the set of outline points, the key points of the spreader, and the key points of the container gripped by the spreader, a target point set is determined, which includes outline points near the bottom corner of the spreader and outline points near the bottom corner of the container gripped by the spreader. Gradient change calculation is performed based on the target point set to obtain the bottom corner point of the spreader and the bottom corner point of the container grabbed by the spreader.

[0008] In some embodiments, determining the target point set based on the contour point set, the key points of the spreader, and the key points of the container gripped by the spreader includes: For each key point of the lifting device, the contour points within a circular frame centered on the key point of the lifting device and with a radius of a first preset radius, which are searched in the inner and outer contour point set of the lifting device, are determined as contour points near the bottom corner of the lifting device. For each key point of the container gripped by the spreader, the outline points within a circular frame with a radius of a second preset radius, centered on the key point of the container gripped by the spreader, are searched on the inner and outer contour points of the container gripped by the spreader and determined as the outline points near the bottom corner of the container gripped by the spreader.

[0009] In some embodiments, the step of performing gradient change calculation based on the target point set to obtain the bottom corner points of the spreader and the bottom corner points of the container gripped by the spreader includes: For the contour points near the bottom corner of the spreader, the angle change between two adjacent contour points in each segment is calculated, with every 2N points as a segment; and the gradient change corresponding to the segment is calculated based on the angle change between two adjacent contour points in the segment; the average of the two middle contour points in the segment with the largest gradient change is determined as the bottom corner of the spreader, where N is a positive integer greater than 1. For the contour points near the bottom corner of the container grabbed by the spreader, the angle change between two adjacent contour points within each segment is calculated, with each segment consisting of 2M points. Based on the angle change between two adjacent contour points within the segment, the gradient change corresponding to the segment is calculated. The average of the two middle contour points within the segment with the largest gradient change is determined as the bottom corner of the container grabbed by the spreader, where M is a positive integer greater than 1.

[0010] In some embodiments, the key points of the spreader are located within the pixel area belonging to the spreader, and the key points of the container grasped by the spreader are located within the pixel area belonging to the container grasped by the spreader.

[0011] In some embodiments, calculating the translational adjustment probability and rotational adjustment probability of the spreader based on the bottom corner of the spreader, the top corner of the container to be stacked, and the bottom corner of the container gripped by the spreader includes: Construct a ray from the bottom corner of the spreader to the top corner of the container to be stacked; The translational adjustment probability of the spreader is calculated based on the distance from the top corner of the container to be stacked to the ray. The diagonal angle of the container grabbed by the spreader is obtained based on the bottom corner point of the container; and the diagonal angle of the container to be stacked is obtained based on the size information of the container to be stacked. The rotation adjustment probability of the spreader is calculated based on the diagonal angle of the container grabbed by the spreader and the diagonal angle of the container to be stacked.

[0012] In some embodiments, the process of releasing the container gripped by the spreader based on the translational adjustment probability and the rotational adjustment probability of the spreader includes: The translational and rotational adjustment probabilities of the lifting device are weighted and summed to obtain the overall adjustment probability of the lifting device. If the overall adjustment probability of the spreader is greater than a preset threshold, the position of the container grasped by the spreader is adjusted. If the overall adjustment probability of the spreader is less than or equal to the preset threshold, the container grabbed by the spreader is lowered.

[0013] Secondly, embodiments of the present invention also provide a container placement device, comprising: The acquisition module is used to acquire a first video stream, which is obtained by a multi-view industrial camera array set up at multiple locations on the lifting device. The first processing module is used to process the first video stream using a dual-modal perception network to obtain key points of multiple objects in the first video stream and pixel-level segmentation masks of the multiple objects; wherein, the multiple objects include a lifting device, a container grasped by the lifting device, and a container to be stacked; the key point of the container to be stacked is the top corner point of the container to be stacked. The second processing module is used to calculate the bottom corner points of the spreader and the bottom corner points of the container being grabbed by the spreader based on the key points of the spreader, the key points of the container being grabbed by the spreader, and the pixel-level segmentation mask of the multiple objects. The third processing module is used to calculate the translational adjustment probability and rotational adjustment probability of the spreader based on the bottom corner point of the spreader, the top corner point of the container to be stacked, and the bottom corner point of the container grabbed by the spreader; and to perform the container placement operation of the container grabbed by the spreader based on the translational adjustment probability and rotational adjustment probability of the spreader.

[0014] Thirdly, embodiments of the present invention also provide a container placement device, including a processor and a transceiver, wherein the transceiver receives and transmits data under the control of the processor, and the processor is used to perform the following operations: The first video stream is acquired by a multi-view industrial camera array set up at multiple locations on the lifting device. Using a dual-modal perception network, the first video stream is processed to obtain key points of multiple objects in the first video stream and pixel-level segmentation masks of the multiple objects; wherein, the multiple objects include a spreader, a container grasped by the spreader, and a container to be stacked; the key point of the container to be stacked is the top corner point of the container to be stacked. Based on the key points of the spreader, the key points of the container grabbed by the spreader, and the pixel-level segmentation mask of the multiple objects, the bottom corner points of the spreader and the bottom corner points of the container grabbed by the spreader are calculated. Based on the bottom corner of the spreader, the top corner of the container to be stacked, and the bottom corner of the container grabbed by the spreader, the translational adjustment probability and the rotational adjustment probability of the spreader are calculated; and the container grabbing by the spreader is placed into the container according to the translational adjustment probability and the rotational adjustment probability of the spreader.

[0015] Fourthly, embodiments of the present invention also provide a container placement device, including a memory, a processor, and a program stored in the memory and executable on the processor; when the processor executes the program, it implements the container placement method as described in the first aspect.

[0016] Fifthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the container placement method as described in the first aspect.

[0017] In a sixth aspect, embodiments of the present invention also provide a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the container placement method as described in the first aspect.

[0018] The above-described technical solution of the present invention has at least the following beneficial effects: In this embodiment of the invention, a first video stream is acquired by a multi-view industrial camera array deployed at multiple locations on a lifting device. A dual-modal perception network is used to process the first video stream to obtain key points of multiple objects and pixel-level segmentation masks of these objects. The multiple objects include the lifting device, a container being gripped by the device, and a container to be stacked. The key point of the container to be stacked is its top corner. Based on the key points of the lifting device, the container being gripped, and the pixel-level segmentation masks of the multiple objects, the bottom corner points of the lifting device and the container being gripped are calculated. The bottom corner points of the lifting device and the container to be stacked are then used to calculate the key points of the lifting device and the container to be stacked. By analyzing the top corner of the container and the bottom corner of the container gripped by the spreader, the translational and rotational adjustment probabilities of the spreader are calculated. Based on these probabilities, the container is then placed. This dual-modal perception network determines the key points of the spreader and the container, roughly indicating the locations of their corners. It also determines pixel-level segmentation masks for multiple objects, representing their pixel positions. Combining these two methods enables refined corner point calculation, achieving the required corner point positioning accuracy for the container placement system. By placing the container based on the corner points of each object with the required positioning accuracy, the alignment accuracy requirements of real container stacking processes are met. Attached Figure Description

[0019] Figure 1 A flowchart illustrating the container placement method according to an embodiment of the present invention; Figure 2 This diagram shows the structure of the lifting device. Figure 3 A schematic diagram illustrating the contour point set and target point set in an embodiment of the present invention; Figure 4 A schematic diagram showing the rays and diagonal of the container in an embodiment of the present invention; Figure 5 A schematic diagram of the container loading device according to an embodiment of the present invention; Figure 6 This is a schematic diagram showing the hardware structure of the container placement equipment according to an embodiment of the present invention; Detailed Implementation

[0020] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0021] To facilitate understanding of the present invention, the relevant content involved in the present invention will be introduced first.

[0022] In port terminal operations, the container loading process currently relies on manually controlled rubber-tired cranes. These cranes use four locking mechanisms to lift the container from the truck, then move it horizontally to the stacking area, align it with its designated placement area, and slowly lower it while continuously adjusting its position so that the four locking cylinders at the bottom of the container engage with the four locking mechanisms below, completing the container loading and stacking operation. The entire process is entirely manual, relying on the crane's boom and four cameras for real-time monitoring and adjustment. This method is extremely inefficient and requires constant monitoring of the crane. If the container loading and stacking process could be automated and precisely completed, it would revolutionize port container handling operations.

[0023] Existing typical technical solutions for container loading operations using tire cranes in ports include: 1. Automated container loading and unloading technology Using a high-precision positioning map, initial alignment is achieved. Then, a visual AI algorithm is employed to establish a real-time alignment evaluation function for real-time attitude estimation and assessment of the box-grabbing action. Error quantification is performed to guide the position and orientation adjustment of the spreader for box placement and adjustment amount.

[0024] 2. Method for Dynamic Lowering of Container Spreader By using a lidar system installed below the lifting device and aligned with the box-laying area below, the lidar detection algorithm detects the positions of the keyhole and the box-laying area from the lidar point cloud data. Using the point cloud positioning information from the detection results, the box-laying operation can be accurately achieved.

[0025] 3. Automated Container Stacking Method Using Vision Systems (1) Install an industrial camera on the spreader to photograph the container; (2) calibrate the industrial camera to obtain the calibration positions of the four corners of the container in the corresponding images; (3) When the spreader is raised to the highest position, process the real-time video collected by the industrial camera, use the template matching algorithm to realize the automatic detection of the corners of the container, and obtain the real-time detection coordinate positions of the four corners of the container; (4) Calculate the relative position deviation between the spreader and the container based on the real-time detection coordinate positions of the four corners of the container and the corresponding calibration positions obtained in step (2); (5) Connect the relative position deviation to the spreader control system to realize automatic alignment when the spreader is raised to the highest position; (6) Move the spreader to the middle position and the lower position respectively, and repeat steps (3) to (5) to realize automatic alignment and grabbing of the container in the middle position and the lower position.

[0026] The shortcomings of existing technologies are as follows: The optimization approach focuses on the overall solution, but insufficient granularity limits the improvement in alignment accuracy. For example, the visual calibration method involves transforming corner points to the world coordinate system based on calibration parameters, combining this with the container corner feature points to obtain the positioning result and converting it back to the world coordinate system. However, this method suffers from the accumulation of errors from multiple processes. The most significant errors are those from corner feature point extraction (generally exceeding 3 pixels) and secondary errors from coordinate transformation to the world coordinate system. In real-world container stacking processes, alignment accuracy requirements are within 5cm, making this method difficult to meet practical accuracy demands.

[0027] Using lidar equipment has limitations due to its high cost and short lifespan. Because the lock hole detection accuracy required for box placement is high, lidar with 128 lines or even higher is needed, and the cost of a single lidar is tens of times that of a camera. Furthermore, the lifespan of lidar is generally between several thousand and tens of thousands of hours. In addition, detecting the box placement lines in the box placement area is impossible due to the limitations of lidar.

[0028] To address the aforementioned technical problems, this invention provides a container placement method, apparatus, equipment, medium, and procedure. The method and apparatus are based on the same concept, and since the principles underlying the problems solved are similar, their implementations can be mutually referenced; repeated details will not be elaborated further.

[0029] like Figure 1 The diagram shows a flowchart of a container placement method provided in an embodiment of the present invention. The method may include: Step 101: Obtain the first video stream, which is acquired by a multi-view industrial camera array set up at multiple locations on the lifting device; Optionally, fixed-angle industrial cameras can be installed at the four corners of the lifting device to construct a multi-view industrial camera array, i.e., a multi-view sensing array. Correspondingly, the first video stream includes four video streams. A schematic diagram of the lifting device's structure can be found here. Figure 2 As shown.

[0030] It should be noted that the multi-view industrial camera array design of this invention can effectively cope with partial occlusion, and the system can still work normally when a single industrial camera fails, thus improving the system's robustness.

[0031] Step 102: Using a dual-modal perception network, process the first video stream to obtain key points of multiple objects in the first video stream and pixel-level segmentation masks of the multiple objects; wherein, the multiple objects include a lifting device, a container grasped by the lifting device, and a container to be stacked; the key point of the container to be stacked is the top corner point of the container to be stacked. Here, the bimodal perception network is a bimodal perception network that performs keypoint detection and instance segmentation in parallel, specifically including keypoint detection and instance segmentation neural networks. It should be noted that this invention employs a bimodal perception network that performs keypoint detection and instance segmentation in parallel, directly performing quantization calculations and adjustments based on the pixel coordinate system. Specifically, after the first video stream is input into the instance segmentation neural network, pixel-level segmentation masks for multiple objects are obtained. Instance segmentation provides precise object contour regions to constrain the retrieval range. After the first video stream is input into the keypoint detection neural network, keypoints for multiple objects are obtained. Keypoint detection provides coarse positions of corner points to guide the search center.

[0032] It should be noted that the key points of this invention can be understood as key feature points. The corner points mentioned below can be understood as corner feature points.

[0033] It should be understood that the container being grabbed by the spreader refers to the container currently being lifted, while the container to be stacked refers to the container that has been placed on the ground or on a stack and is waiting to be covered or used as support.

[0034] Step 103: Based on the key points of the spreader, the key points of the container grabbed by the spreader, and the pixel-level segmentation mask of the multiple objects, calculate the bottom corner points of the spreader and the bottom corner points of the container grabbed by the spreader. Here, based on the key points of the spreader, the key points of the container grasped by the spreader, and the pixel-level segmentation mask of the multiple objects, the bottom corner points of the spreader and the container grasped by the spreader can be calculated through gradient and angle change analysis, achieving refined corner point solution. It should be noted that this invention achieves rapid localization through key point detection, followed by minimal computation for local refinement, with an overall processing latency of <100ms, demonstrating good real-time performance.

[0035] Step 104: Calculate the translational adjustment probability and rotational adjustment probability of the spreader based on the bottom corner point of the spreader, the top corner point of the container to be stacked, and the bottom corner point of the container grabbed by the spreader; and perform the container placement operation based on the translational adjustment probability and rotational adjustment probability of the spreader.

[0036] Here, based on the bottom corner of the spreader, the top corner of the container to be stacked, and the bottom corner of the container grabbed by the spreader, the distance deviation between the top corner of the container to be stacked and the side line of the spreader, and the rotation angle deviation between the diagonal angle of the container grabbed by the spreader and the diagonal angle of the container to be stacked can be calculated. Then, a quantitative evaluation model including translation adjustment probability and rotation adjustment probability is established, and finally, the container placement operation of the container grabbed by the spreader is realized based on this model.

[0037] The container placement method of this invention involves acquiring a first video stream, which is obtained by a multi-view industrial camera array deployed at multiple locations on a spreader; processing the first video stream using a dual-modal perception network to obtain key points of multiple objects in the first video stream and pixel-level segmentation masks of the multiple objects; wherein the multiple objects include a spreader, a container grasped by the spreader, and a container to be stacked; the key points of the container to be stacked are the top corner points of the container to be stacked; based on the key points of the spreader, the key points of the container grasped by the spreader, and the pixel-level segmentation masks of the multiple objects, the bottom corner points of the spreader and the container grasped by the spreader are calculated; based on the bottom corner points of the spreader and the container to be stacked, the bottom corner points of the container are calculated; and based on the bottom corner points of the spreader and the container to be stacked, the bottom corner points of the container are calculated. The top corner of the container and the bottom corner of the container gripped by the spreader are used to calculate the translational and rotational adjustment probabilities of the spreader. Based on these probabilities, the container is placed using the spreader. This dual-modal sensing network determines the key points of the spreader and the container, roughly indicating the locations of their corners. It also determines pixel-level segmentation masks for multiple objects, representing their pixel positions. Combining these two methods allows for refined corner point calculation, achieving the required corner point positioning accuracy for the container placement system. By placing the container based on the corner points of each object with the required positioning accuracy, the system meets the alignment accuracy requirements of real container stacking processes.

[0038] In some embodiments, step 102 above, using a dual-modal perception network to process the first video stream to obtain key points of multiple objects in the first video stream and pixel-level segmentation masks of the multiple objects, includes: Using multiple convolutional layers in the dual-modal perception network, multi-scale feature extraction is performed on the first video stream to obtain feature maps at multiple scales, and the feature maps at multiple scales are fused to obtain fused features; Here, the dual-modal perception network includes multiple convolutional layers, a feature fusion module, a pose head, and a segmentation head. Specifically, the multiple convolutional layers are connected to the feature fusion module, which is connected to both the pose head and the segmentation head. The feature fusion module is used to fuse feature maps at multiple scales. For example, there are 44 convolutional layers.

[0039] It should be understood that the aforementioned keypoint detection neural network includes multiple convolutional layers, a feature fusion module, and a pose head; the instance segmentation neural network includes the multiple convolutional layers, the feature fusion module, and a segmentation head.

[0040] Using the pose head in the dual-modal perception network, regression processing is performed on the fused features to obtain the key points of the multiple objects; Using the segmentation head in the dual-modal perception network, regression processing is performed on the fused features to obtain pixel-level segmentation masks for the multiple objects.

[0041] It should be noted that different heads are used to perform different tasks. The pose head in this invention is used to obtain key points of multiple objects. The segmentation head is used to obtain pixel-level segmentation masks of multiple objects.

[0042] In this embodiment, by processing the first video stream through a dual-modal network, key points of multiple objects in the first video stream and pixel-level segmentation masks of multiple objects are obtained, providing preliminary feature data for subsequent positioning of the bottom corner points of the spreader and the bottom corner points of the container grabbed by the spreader.

[0043] In some embodiments, in step 103 above, calculating the bottom corner points of the spreader and the bottom corner points of the container gripped by the spreader, based on the key points of the spreader, the key points of the container gripped by the spreader, and the pixel-level segmentation mask of the plurality of objects, includes: Step 1031: Based on the pixel-level segmentation mask of the multiple objects, obtain the pixel points belonging to the pixel region of the spreader and the pixel points belonging to the pixel region of the container grabbed by the spreader. Here, based on the pixel-level segmentation mask of multiple objects output by the instance segmentation neural network, the pixels belonging to the spreader and the pixels belonging to the container being grabbed by the spreader in the real-time container placement image can be obtained.

[0044] Step 1032: Perform edge detection processing on the pixels belonging to the pixel area of ​​the spreader and the pixels belonging to the pixel area of ​​the container grasped by the spreader to obtain a set of contour points. The set of contour points includes the inner and outer contour point sets of the spreader and the inner and outer contour point sets of the container grasped by the spreader. Here, the Canny edge detection algorithm can be used to detect the edges of pixels belonging to the pixel region of the spreader and pixels belonging to the pixel region of the container grasped by the spreader.

[0045] The set of contour points can be represented as β1, see [reference needed]. Figure 3 As shown.

[0046] Step 1033: Determine a target point set based on the set of outline points, the key points of the spreader, and the key points of the container gripped by the spreader. The target point set includes outline points near the bottom corner of the spreader and outline points near the bottom corner of the container gripped by the spreader. As an optional implementation, step 1033 specifically includes: For each key point of the lifting device, the contour points within a circular frame centered on the key point of the lifting device and with a radius of a first preset radius, which are searched in the inner and outer contour point set of the lifting device, are determined as contour points near the bottom corner of the lifting device. That is, iterate through each key point of the lifting device, and search within the set of inner and outer contour points of the lifting device for contour points within a circular frame centered on the key point of the lifting device and with a radius of a first preset radius (e.g., α pixels). Contour points within the circular frame that meet the above conditions are identified as contour points near the bottom corner of the lifting device. Figure 3 The outline points within the blue circular frame corresponding to the green outer contour line.

[0047] For each key point of the container gripped by the spreader, the outline points within a circular frame with a radius of a second preset radius, centered on the key point of the container gripped by the spreader, are searched on the inner and outer contour points of the container gripped by the spreader and determined as the outline points near the bottom corner of the container gripped by the spreader.

[0048] This involves iterating through each key point of the container gripped by the spreader, searching for contour points within a circular frame centered on the key point of the container's inner and outer contours, with a radius of a second preset radius. Contour points within this circular frame that meet the above conditions are identified as contour points near the bottom corner of the container gripped by the spreader. Figure 3 The outline points within the blue circular frame corresponding to the red outer contour line.

[0049] Here, the target point set includes the contour points near the bottom corner of the spreader and the contour points near the bottom corner of the container gripped by the spreader. The target point set can be represented as β2. (See [reference needed]). Figure 3 As shown.

[0050] Step 1034: Perform gradient change calculation based on the target point set to obtain the bottom corner point of the spreader and the bottom corner point of the container grabbed by the spreader.

[0051] As an optional implementation, step 1034 specifically includes: For the contour points near the bottom corner of the spreader, the angle change between two adjacent contour points in each segment is calculated, with every 2N points as a segment; and the gradient change corresponding to the segment is calculated based on the angle change between two adjacent contour points in the segment; the average of the two middle contour points in the segment with the largest gradient change is determined as the bottom corner of the spreader, where N is a positive integer greater than 1. It should be understood that the above processing is performed on the contour points near each of the four bottom corners of the spreader to achieve precise positioning of the four bottom corners of the spreader.

[0052] For example, each segment can be divided into 2N (e.g., 6) points, and the gradient (i.e., change) between two adjacent contour points (i.e., contour pixels) within the segment in the u (horizontal) and v (vertical) directions in the pixel coordinate system (i.e., uv coordinate system) can be calculated; specifically, it can be expressed as:

[0053]

[0054] in, This represents the gradient in the u-direction between two adjacent contour points. This represents the gradient in the v direction between two adjacent contour points.

[0055] Then, based on the gradients between two contour points in the u and v directions in the pixel coordinate system, the angular change between two adjacent contour points within the segment is calculated. Specifically, it can be expressed as:

[0056] Then, based on the angle change between two adjacent contour points within the segment, the gradient change corresponding to the segment is calculated. Specifically, this means calculating the average angle change based on the angle change between every two contour points within the segment; this average angle change is the gradient change corresponding to the segment. Specifically, it can be expressed as:

[0057] Finally, the average of the two middle contour points in the segment with the largest gradient change is determined as the bottom corner point of the lifting device.

[0058] For the contour points near the bottom corner of the container grabbed by the spreader, the angle change between two adjacent contour points within each segment is calculated, with each segment consisting of 2M points. Based on the angle change between two adjacent contour points within the segment, the gradient change corresponding to the segment is calculated. The average of the two middle contour points within the segment with the largest gradient change is determined as the bottom corner of the container grabbed by the spreader, where M is a positive integer greater than 1.

[0059] Similarly, the process of determining the bottom corner of the container grabbed by the spreader is the same as the process of determining the bottom corner of the spreader described above, and will not be repeated here.

[0060] In this embodiment, the pixel-level segmentation mask that is fused with the key points and instance segmentation output is used to achieve fine-grained corner point solving through gradient and angle change analysis, which can improve the accuracy and robustness of corner point extraction.

[0061] Optionally, the key points of the spreader are located within the pixel area belonging to the spreader, and the key points of the container grasped by the spreader are located within the pixel area belonging to the container grasped by the spreader.

[0062] This pertains to a corner verification scheme. Key points of the spreader output by the dual-modal sensing network must be within the pixel area belonging to the spreader, and key points of the container grasped by the spreader must be within the pixel area belonging to the container grasped by the spreader; otherwise, the frame is deemed invalid.

[0063] In some embodiments, in step 104 above, calculating the translational adjustment probability and rotational adjustment probability of the spreader based on the bottom corner of the spreader, the top corner of the container to be stacked, and the bottom corner of the container gripped by the spreader includes: Construct a ray from the bottom corner of the spreader to the top corner of the container to be stacked; It should be noted that the steps of this embodiment are performed based on the corner point information obtained from each frame in the first video stream (i.e., the bottom corner of the spreader, the top corner of the container to be stacked, and the bottom corner of the container grabbed by the spreader).

[0064] Here, the side line of the spreader mentioned earlier, that is, the ray L from the bottom corner of the spreader to the top corner of the container to be stacked, is as follows: Figure 4 As shown.

[0065] The translational adjustment probability of the spreader is calculated based on the distance from the top corner of the container to be stacked to the ray. Here, the distance from the top corner point P of the container to be stacked to ray L is defined as the distance deviation between the top corner point of the container to be stacked and the side edge line of the spreader, and is decomposed into the horizontal component in the pixel coordinate system. and vertical components .

[0066] If we assume: ① The top corner point P(i) of the containers to be stacked, i=1,2,3,4; where each value of i represents one of the four top corner points of the container.

[0067] ② Ray L(i), i=1,2,3,4; where each value of i represents the ray from each of the four bottom corners of the spreader to the top corner of the corresponding container to be stacked.

[0068] ③ The distance from P(i) to L(i) is disPL(i), i=1,2,3,4. Specifically:

[0069]

[0070] in, This represents the distance of corner point i in the u direction within the pixel coordinate system (i.e., the uv coordinate system). This represents the distance of corner point i in the v direction within the pixel coordinate system (i.e., the uv coordinate system).

[0071] Here, the translation adjustment probability P(XOY) of the spreader is calculated based on the distance disPL(i) from the top corner of the container to be stacked to the ray; specifically: ; in, This is the constant coefficient at that point, ranging between (0,1).

[0072] The diagonal angle of the container grabbed by the spreader is obtained based on the bottom corner point of the container; and the diagonal angle of the container to be stacked is obtained based on the size information of the container to be stacked. Here, based on the key point detection and instance segmentation results, i.e., the bottom corner points of the container gripped by the spreader, the diagonal angle of the container gripped by the spreader can be obtained. ( Figure 4 Q1 in the middle), such as Figure 4 As shown. The dimensions of the containers to be stacked can be obtained from the port operations system, and thus the diagonal angle of the containers to be stacked can be calculated. ( Figure 4 Q2 in the middle), such as Figure 4 As shown. It should be noted that the diagonal angle ranges from 0 to 360°, with true north as 0° and rotating 360° clockwise (using...). Figure 4 The red vertical line in the middle is the 0-degree line.

[0073] The rotation adjustment probability of the spreader is calculated based on the diagonal angle of the container grabbed by the spreader and the diagonal angle of the container to be stacked. .

[0074] Here, let's assume: ① The diagonal angle of the containers to be stacked , i=1,2,3,4; ② The diagonal angle of the container gripped by the spreader , i=1,2,3,4;

[0075] This embodiment establishes a quantitative evaluation model that includes translation adjustment probability and rotation adjustment probability.

[0076] In some embodiments, step 104 above, which involves performing the container placement operation based on the translational and rotational adjustment probabilities of the spreader, includes: The translational and rotational adjustment probabilities of the lifting device are weighted and summed to obtain the overall adjustment probability of the lifting device. It should be noted that during the container placement process, the possibility of adjustments will be constantly calculated. This refers to the overall adjustment probability of the spreading gear. Specifically, it consists of the translational adjustment probability and the rotational adjustment probability of the spreading gear, then:

[0077] in, It is the weight of the probability of the lifting device being translated and adjusted.

[0078] If the overall adjustment probability of the spreader is greater than a preset threshold, the position of the container grasped by the spreader is adjusted. Here, when the overall adjustment probability of the spreader... > If necessary, the orientation of the container being gripped by the spreader is adjusted. Specifically, in the pixel coordinate system, the horizontal translation is adjusted as PlanAdjustment(h(t))_x and the vertical translation as PlanAdjustment(h(t))_y, and the rotation is also adjusted. Where PlanAdjustment(h(t))_x = ,PlanAdjustment(h(t))_y= .

[0079] If the overall adjustment probability of the spreader is less than or equal to the preset threshold, the container grabbed by the spreader is lowered.

[0080] When the overall adjustment probability of the spreader ≤ If so, the container being grabbed by the lower spreader will continue to be handled.

[0081] It should be noted that the spreader's position is monitored and adjusted in real time during descent to ensure accurate alignment for container placement. Here, the overall adjustment probability of the spreader... > If this threshold is reached, an alarm can be triggered. These are empirical values ​​that are adjusted based on the actual project implementation and circumstances.

[0082] This invention eliminates the need for reinstalling equipment; existing port-side cameras can be reused, resulting in cost savings and making it more acceptable to terminal operators. Furthermore, the method can be used even if the original camera's internal parameters have not been calibrated, requiring no additional work. This invention uses pixel-based quantization and adjustment, eliminating the need for calibration and simulating actual process adjustments. The method offers strong real-time performance; image processing (including deep learning and traditional methods) utilizes GPUs, reducing computation time and maintaining real-time overall perception output even in high-resolution scenarios. Moreover, this invention, based on a pure vision solution, ensures container placement alignment and positioning without significantly increasing computational power.

[0083] like Figure 5 As shown, this embodiment of the invention also provides a container placement device, which includes: The acquisition module 501 is used to acquire a first video stream, which is acquired by a multi-view industrial camera array set up at multiple locations on the lifting device. The first processing module 502 is used to process the first video stream using a dual-modal perception network to obtain key points of multiple objects in the first video stream and pixel-level segmentation masks of the multiple objects; wherein, the multiple objects include a lifting device, a container grasped by the lifting device, and a container to be stacked; the key point of the container to be stacked is the top corner point of the container to be stacked. The second processing module 503 is used to calculate the bottom corner points of the spreader and the bottom corner points of the container grabbed by the spreader based on the key points of the spreader, the key points of the container grabbed by the spreader, and the pixel-level segmentation mask of the multiple objects. The third processing module 504 is used to calculate the translation adjustment probability and rotation adjustment probability of the spreader based on the bottom corner point of the spreader, the top corner point of the container to be stacked, and the bottom corner point of the container grabbed by the spreader; and to perform the container placement operation of the container grabbed by the spreader based on the translation adjustment probability and rotation adjustment probability of the spreader.

[0084] In some embodiments, the first processing module 502 includes: The first processing unit is used to extract features from the first video stream at multiple scales using multiple convolutional layers in the dual-modal perception network, to obtain feature maps at multiple scales, and to fuse the feature maps at multiple scales to obtain fused features. The second processing unit is used to perform regression processing on the fused features using the attitude head in the dual-modal perception network to obtain the key points of the multiple objects; The third processing unit is used to perform regression processing on the fused features using the segmentation head in the dual-modal perception network to obtain pixel-level segmentation masks for the multiple objects.

[0085] In some embodiments, the second processing module 503 includes: The acquisition unit is used to acquire pixels belonging to the pixel region of the spreader and pixels belonging to the pixel region of the container grasped by the spreader, based on the pixel-level segmentation mask of the plurality of objects. The fourth processing unit is used to perform edge detection processing on the pixels belonging to the pixel area of ​​the spreader and the pixels belonging to the pixel area of ​​the container grasped by the spreader, respectively, to obtain a set of contour points. The set of contour points includes the inner and outer contour point sets of the spreader and the inner and outer contour point sets of the container grasped by the spreader. The fifth processing unit is used to determine a target point set based on the set of contour points, the key points of the spreader, and the key points of the container gripped by the spreader. The target point set includes contour points near the bottom corner of the spreader and contour points near the bottom corner of the container gripped by the spreader. The first calculation unit is used to perform gradient change calculation based on the target point set to obtain the bottom corner point of the spreader and the bottom corner point of the container being gripped by the spreader.

[0086] In some embodiments, the fifth processing unit is specifically used for: For each key point of the lifting device, the contour points within a circular frame centered on the key point of the lifting device and with a radius of a first preset radius, which are searched in the inner and outer contour point set of the lifting device, are determined as contour points near the bottom corner of the lifting device. For each key point of the container gripped by the spreader, the outline points within a circular frame with a radius of a second preset radius, centered on the key point of the container gripped by the spreader, are searched on the inner and outer contour points of the container gripped by the spreader and determined as the outline points near the bottom corner of the container gripped by the spreader.

[0087] In some embodiments, the first computing unit is specifically used for: For the contour points near the bottom corner of the spreader, the angle change between two adjacent contour points in each segment is calculated, with every 2N points as a segment; and the gradient change corresponding to the segment is calculated based on the angle change between two adjacent contour points in the segment; the average of the two middle contour points in the segment with the largest gradient change is determined as the bottom corner of the spreader, where N is a positive integer greater than 1. For the contour points near the bottom corner of the container grabbed by the spreader, the angle change between two adjacent contour points within each segment is calculated, with each segment consisting of 2M points. Based on the angle change between two adjacent contour points within the segment, the gradient change corresponding to the segment is calculated. The average of the two middle contour points within the segment with the largest gradient change is determined as the bottom corner of the container grabbed by the spreader, where M is a positive integer greater than 1.

[0088] Optionally, the key points of the spreader are located within the pixel area belonging to the spreader, and the key points of the container grasped by the spreader are located within the pixel area belonging to the container grasped by the spreader.

[0089] In some embodiments, the third processing module 504 includes: A ray construction unit is used to construct a ray from the bottom corner of the spreader to the top corner of the container to be stacked; The second calculation unit is used to calculate the translation adjustment probability of the spreader based on the distance from the top corner of the container to be stacked to the ray. The sixth processing unit is used to obtain the diagonal angle of the container grabbed by the spreader based on the bottom corner point of the container grabbed by the spreader; and to obtain the diagonal angle of the container to be stacked based on the size information of the container to be stacked. The third calculation unit is used to calculate the rotation adjustment probability of the spreader based on the diagonal angle of the container grabbed by the spreader and the diagonal angle of the container to be stacked.

[0090] In some embodiments, the third processing module 504 includes: The fourth calculation unit is used to perform a weighted summation calculation on the translational adjustment probability and the rotational adjustment probability of the lifting device to obtain the comprehensive adjustment probability of the lifting device; The seventh processing unit is used to adjust the position of the container grasped by the spreader when the overall adjustment probability of the spreader is greater than a preset threshold. The eighth processing unit is used to lower the container grabbed by the spreader when the overall adjustment probability of the spreader is less than or equal to the preset threshold.

[0091] The container placement device of this invention acquires a first video stream, which is obtained by a multi-view industrial camera array deployed at multiple locations on a spreader. A dual-modal perception network is used to process the first video stream to obtain key points of multiple objects in the first video stream and pixel-level segmentation masks of the multiple objects. The multiple objects include a spreader, a container grasped by the spreader, and a container to be stacked. The key point of the container to be stacked is its top corner. Based on the key points of the spreader, the container grasped by the spreader, and the pixel-level segmentation masks of the multiple objects, the bottom corner points of the spreader and the container grasped by the spreader are calculated. The device then calculates the bottom corner points of the spreader and the container to be stacked based on the key points of the spreader and the container grasped by the spreader. The top corner of the container and the bottom corner of the container gripped by the spreader are used to calculate the translational and rotational adjustment probabilities of the spreader. Based on these probabilities, the container is placed using the spreader. This dual-modal sensing network determines the key points of the spreader and the container, roughly indicating the locations of their corners. It also determines pixel-level segmentation masks for multiple objects, representing their pixel positions. Combining these two methods allows for refined corner point calculation, achieving the required corner point positioning accuracy for the container placement system. By placing the container based on the corner points of each object with the required positioning accuracy, the system meets the alignment accuracy requirements of real container stacking processes.

[0092] To better achieve the above objectives, such as Figure 6 As shown, this embodiment of the invention also provides a container placement device, including a processor 600 and a transceiver 610. The transceiver 610 receives and transmits data under the control of the processor 600, and the processor 600 is used to perform the following processes: The first video stream is acquired by a multi-view industrial camera array set up at multiple locations on the lifting device. Using a dual-modal perception network, the first video stream is processed to obtain key points of multiple objects in the first video stream and pixel-level segmentation masks of the multiple objects; wherein, the multiple objects include a spreader, a container grasped by the spreader, and a container to be stacked; the key point of the container to be stacked is the top corner point of the container to be stacked. Based on the key points of the spreader, the key points of the container grabbed by the spreader, and the pixel-level segmentation mask of the multiple objects, the bottom corner points of the spreader and the bottom corner points of the container grabbed by the spreader are calculated. Based on the bottom corner of the spreader, the top corner of the container to be stacked, and the bottom corner of the container grabbed by the spreader, the translational adjustment probability and the rotational adjustment probability of the spreader are calculated; and the container grabbing by the spreader is placed into the container according to the translational adjustment probability and the rotational adjustment probability of the spreader.

[0093] In some embodiments, the processor 600 is further configured to: Using multiple convolutional layers in the dual-modal perception network, multi-scale feature extraction is performed on the first video stream to obtain feature maps at multiple scales, and the feature maps at multiple scales are fused to obtain fused features; Using the pose head in the dual-modal perception network, regression processing is performed on the fused features to obtain the key points of the multiple objects; Using the segmentation head in the dual-modal perception network, regression processing is performed on the fused features to obtain pixel-level segmentation masks for the multiple objects.

[0094] In some embodiments, the processor 600 is further configured to: Based on the pixel-level segmentation mask of the multiple objects, obtain the pixel points belonging to the pixel region of the spreader and the pixel points belonging to the pixel region of the container grabbed by the spreader; Edge detection processing is performed on the pixels belonging to the pixel area of ​​the spreader and the pixels belonging to the pixel area of ​​the container grasped by the spreader to obtain a set of contour points. The set of contour points includes the inner and outer contour point sets of the spreader and the inner and outer contour point sets of the container grasped by the spreader. Based on the set of outline points, the key points of the spreader, and the key points of the container gripped by the spreader, a target point set is determined, which includes outline points near the bottom corner of the spreader and outline points near the bottom corner of the container gripped by the spreader. Gradient change calculation is performed based on the target point set to obtain the bottom corner points of the spreader and the bottom corner points of the container being gripped by the spreader.

[0095] In some embodiments, the processor 600 is further configured to: For each key point of the lifting device, the contour points within a circular frame centered on the key point of the lifting device and with a radius of a first preset radius, which are searched in the inner and outer contour point set of the lifting device, are determined as contour points near the bottom corner of the lifting device. For each key point of the container gripped by the spreader, the outline points within a circular frame with a radius of a second preset radius, centered on the key point of the container gripped by the spreader, are searched on the inner and outer contour points of the container gripped by the spreader and determined as the outline points near the bottom corner of the container gripped by the spreader.

[0096] In some embodiments, the processor 600 is further configured to: For the contour points near the bottom corner of the spreader, the angle change between two adjacent contour points in each segment is calculated, with every 2N points as a segment; and the gradient change corresponding to the segment is calculated based on the angle change between two adjacent contour points in the segment; the average of the two middle contour points in the segment with the largest gradient change is determined as the bottom corner of the spreader, where N is a positive integer greater than 1. For the contour points near the bottom corner of the container grabbed by the spreader, the angle change between two adjacent contour points within each segment is calculated, with each segment consisting of 2M points. Based on the angle change between two adjacent contour points within the segment, the gradient change corresponding to the segment is calculated. The average of the two middle contour points within the segment with the largest gradient change is determined as the bottom corner of the container grabbed by the spreader, where M is a positive integer greater than 1.

[0097] Optionally, the key points of the spreader are located within the pixel area belonging to the spreader, and the key points of the container grasped by the spreader are located within the pixel area belonging to the container grasped by the spreader.

[0098] In some embodiments, the processor 600 is further configured to: Construct a ray from the bottom corner of the spreader to the top corner of the container to be stacked; The translational adjustment probability of the spreader is calculated based on the distance from the top corner of the container to be stacked to the ray. The diagonal angle of the container grabbed by the spreader is obtained based on the bottom corner point of the container; and the diagonal angle of the container to be stacked is obtained based on the size information of the container to be stacked. The rotation adjustment probability of the spreader is calculated based on the diagonal angle of the container grabbed by the spreader and the diagonal angle of the container to be stacked.

[0099] In some embodiments, the processor 600 is further configured to: The translational and rotational adjustment probabilities of the lifting device are weighted and summed to obtain the overall adjustment probability of the lifting device. If the overall adjustment probability of the spreader is greater than a preset threshold, the position of the container grasped by the spreader is adjusted. If the overall adjustment probability of the spreader is less than or equal to the preset threshold, the container grabbed by the spreader is lowered.

[0100] The container loading device of this invention acquires a first video stream, which is obtained by a multi-view industrial camera array deployed at multiple locations on a spreader. A dual-modal perception network is used to process the first video stream to obtain key points of multiple objects in the first video stream and pixel-level segmentation masks of the multiple objects. The multiple objects include a spreader, a container gripped by the spreader, and a container to be stacked. The key point of the container to be stacked is its top corner. Based on the key points of the spreader, the container gripped by the spreader, and the pixel-level segmentation masks of the multiple objects, the bottom corner points of the spreader and the container gripped by the spreader are calculated. The device then calculates the bottom corner points of the spreader and the container to be stacked based on the key points of the spreader and the container to be stacked. The top corner of the container and the bottom corner of the container gripped by the spreader are used to calculate the translational and rotational adjustment probabilities of the spreader. Based on these probabilities, the container is placed using the spreader. This dual-modal sensing network determines the key points of the spreader and the container, roughly indicating the locations of their corners. It also determines pixel-level segmentation masks for multiple objects, representing their pixel positions. Combining these two methods allows for refined corner point calculation, achieving the required corner point positioning accuracy for the container placement system. By placing the container based on the corner points of each object with the required positioning accuracy, the system meets the alignment accuracy requirements of real container stacking processes.

[0101] This invention also provides a container placement device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the various processes described in the above-described container placement method embodiments and achieves the same technical effect. To avoid repetition, these will not be repeated here.

[0102] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, this program implements the various processes described above in the container placement method embodiments, achieving the same technical effects. To avoid repetition, further details are omitted here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0103] This invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described functionality. Figure 1 The steps in the container placement method shown are as follows.

[0104] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0105] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 A device for one or more processes and / or the functions specified in one or more boxes.

[0106] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce a paper article including an instruction means, the instruction means being implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0107] These computer program instructions can also be loaded onto a computer or other programmable data processing equipment, causing the computer or other programmable equipment to perform a series of operational steps to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0108] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for placing a container, characterized in that, include: The first video stream is acquired by a multi-view industrial camera array set up at multiple locations on the lifting device. Using a dual-modal perception network, the first video stream is processed to obtain key points of multiple objects in the first video stream and pixel-level segmentation masks of the multiple objects; wherein, the multiple objects include a spreader, a container grasped by the spreader, and a container to be stacked; the key point of the container to be stacked is the top corner point of the container to be stacked. Based on the key points of the spreader, the key points of the container grabbed by the spreader, and the pixel-level segmentation mask of the multiple objects, the bottom corner points of the spreader and the bottom corner points of the container grabbed by the spreader are calculated. Based on the bottom corner of the spreader, the top corner of the container to be stacked, and the bottom corner of the container grabbed by the spreader, the translational adjustment probability and the rotational adjustment probability of the spreader are calculated; and the container grabbing by the spreader is placed into the container according to the translational adjustment probability and the rotational adjustment probability of the spreader.

2. The method according to claim 1, characterized in that, The step of processing the first video stream using a dual-modal perception network to obtain key points of multiple objects in the first video stream and pixel-level segmentation masks of the multiple objects includes: Using multiple convolutional layers in the dual-modal perception network, multi-scale feature extraction is performed on the first video stream to obtain feature maps at multiple scales, and the feature maps at multiple scales are fused to obtain fused features; Using the pose head in the dual-modal perception network, regression processing is performed on the fused features to obtain the key points of the multiple objects; Using the segmentation head in the dual-modal perception network, regression processing is performed on the fused features to obtain pixel-level segmentation masks for the multiple objects.

3. The method according to claim 1, characterized in that, The calculation of the bottom corner points of the spreader and the bottom corner points of the container gripped by the spreader, based on the key points of the spreader, the key points of the container gripped by the spreader, and the pixel-level segmentation mask of the multiple objects, includes: Based on the pixel-level segmentation mask of the multiple objects, obtain the pixel points belonging to the pixel region of the spreader and the pixel points belonging to the pixel region of the container grabbed by the spreader; Edge detection processing is performed on the pixels belonging to the pixel area of ​​the spreader and the pixels belonging to the pixel area of ​​the container grasped by the spreader to obtain a set of contour points. The set of contour points includes the inner and outer contour point sets of the spreader and the inner and outer contour point sets of the container grasped by the spreader. Based on the set of outline points, the key points of the spreader, and the key points of the container gripped by the spreader, a target point set is determined, which includes outline points near the bottom corner of the spreader and outline points near the bottom corner of the container gripped by the spreader. Gradient change calculation is performed based on the target point set to obtain the bottom corner points of the spreader and the bottom corner points of the container being gripped by the spreader.

4. The method according to claim 3, characterized in that, The step of determining the target point set based on the contour point set, the key points of the spreader, and the key points of the container grasped by the spreader includes: For each key point of the lifting device, the contour points within a circular frame centered on the key point of the lifting device and with a radius of a first preset radius, which are searched in the inner and outer contour point set of the lifting device, are determined as contour points near the bottom corner of the lifting device. For each key point of the container gripped by the spreader, the outline points within a circular frame with a radius of a second preset radius, centered on the key point of the container gripped by the spreader, are searched on the inner and outer contour points of the container gripped by the spreader and determined as the outline points near the bottom corner of the container gripped by the spreader.

5. The method according to claim 3, characterized in that, The step of calculating gradient changes based on the target point set to obtain the bottom corner points of the spreader and the container gripped by the spreader includes: For the contour points near the bottom corner of the spreader, the angle change between two adjacent contour points in each segment is calculated, with every 2N points as a segment; and the gradient change corresponding to the segment is calculated based on the angle change between two adjacent contour points in the segment; the average of the two middle contour points in the segment with the largest gradient change is determined as the bottom corner of the spreader, where N is a positive integer greater than 1. For the contour points near the bottom corner of the container grabbed by the spreader, the angle change between two adjacent contour points within each segment is calculated, with each segment consisting of 2M points. Based on the angle change between two adjacent contour points within the segment, the gradient change corresponding to the segment is calculated. The average of the two middle contour points within the segment with the largest gradient change is determined as the bottom corner of the container grabbed by the spreader, where M is a positive integer greater than 1.

6. The method according to claim 3, characterized in that, The key points of the spreader are located within the pixel area belonging to the spreader, and the key points of the container grasped by the spreader are located within the pixel area belonging to the container grasped by the spreader.

7. The method according to claim 1, characterized in that, The step of calculating the translational adjustment probability and rotational adjustment probability of the spreader based on the bottom corner point of the spreader, the top corner point of the container to be stacked, and the bottom corner point of the container gripped by the spreader includes: Construct a ray from the bottom corner of the spreader to the top corner of the container to be stacked; The translational adjustment probability of the spreader is calculated based on the distance from the top corner of the container to be stacked to the ray. The diagonal angle of the container grabbed by the spreader is obtained based on the bottom corner point of the container; and the diagonal angle of the container to be stacked is obtained based on the size information of the container to be stacked. The rotation adjustment probability of the spreader is calculated based on the diagonal angle of the container grabbed by the spreader and the diagonal angle of the container to be stacked.

8. The method according to claim 1, characterized in that, The process of releasing the container gripped by the spreader based on the translational and rotational adjustment probabilities of the spreader includes: The translational and rotational adjustment probabilities of the lifting device are weighted and summed to obtain the overall adjustment probability of the lifting device. If the overall adjustment probability of the spreader is greater than a preset threshold, the position of the container grasped by the spreader is adjusted. If the overall adjustment probability of the spreader is less than or equal to the preset threshold, the container grabbed by the spreader is lowered.

9. A container placement device, characterized in that, include: The acquisition module is used to acquire a first video stream, which is obtained by a multi-view industrial camera array set up at multiple locations on the lifting device. The first processing module is used to process the first video stream using a dual-modal perception network to obtain key points of multiple objects in the first video stream and pixel-level segmentation masks of the multiple objects; wherein, the multiple objects include a lifting device, a container grasped by the lifting device, and a container to be stacked; the key point of the container to be stacked is the top corner point of the container to be stacked. The second processing module is used to calculate the bottom corner points of the spreader and the bottom corner points of the container being grabbed by the spreader based on the key points of the spreader, the key points of the container being grabbed by the spreader, and the pixel-level segmentation mask of the multiple objects. The third processing module is used to calculate the translational adjustment probability and rotational adjustment probability of the spreader based on the bottom corner point of the spreader, the top corner point of the container to be stacked, and the bottom corner point of the container grabbed by the spreader; and to perform the container placement operation of the container grabbed by the spreader based on the translational adjustment probability and rotational adjustment probability of the spreader.

10. A container loading and unloading device, characterized in that, It includes a processor and a transceiver, the transceiver receiving and transmitting data under the control of the processor, characterized in that the processor is configured to perform the following operations: The first video stream is acquired by a multi-view industrial camera array set up at multiple locations on the lifting device. Using a dual-modal perception network, the first video stream is processed to obtain key points of multiple objects in the first video stream and pixel-level segmentation masks of the multiple objects; wherein, the multiple objects include a spreader, a container grasped by the spreader, and a container to be stacked; the key point of the container to be stacked is the top corner point of the container to be stacked. Based on the key points of the spreader, the key points of the container grabbed by the spreader, and the pixel-level segmentation mask of the multiple objects, the bottom corner points of the spreader and the bottom corner points of the container grabbed by the spreader are calculated. Based on the bottom corner of the spreader, the top corner of the container to be stacked, and the bottom corner of the container grabbed by the spreader, the translational adjustment probability and the rotational adjustment probability of the spreader are calculated; and the container grabbing by the spreader is placed into the container according to the translational adjustment probability and the rotational adjustment probability of the spreader.

11. A container loading / unloading device, comprising a memory, a processor, and a computer program stored in the memory and running thereon, characterized in that, When the processor executes the program, it implements the container placement method as described in any one of claims 1 to 8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the container placement method as described in any one of claims 1 to 8.

13. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps in the container placement method as described in any one of claims 1 to 8.

Images