Material cage stacking method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides a material cage stacking method and device, electronic equipment and a medium, and the method comprises the steps: collecting RGB images and depth images of an upper material cage and a lower material cage through a depth camera; wherein the depth camera is deployed on the carrying equipment; identifying a material cage corner area of the RGB image by adopting a pre-trained target detection model; obtaining a point cloud image of the material cage corner area according to the material cage corner area and the depth image; and according to the point cloud image, coordinate information and a required deviation angle of a driving position of the carrying equipment are determined, so that the carrying equipment is controlled to stack the material cages. According to the embodiment of the invention, the stacking of the material cages is completed by combining the target detection technology and the depth camera, the time required for processing is reduced, the real-time performance is improved, the method can be suitable for different types of material cages, a plurality of sensors are not required to be used for collecting data, and the cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent warehousing, and particularly to a method, device, electronic device and medium for stacking cages. Background Art

[0002] In order to make more efficient use of warehousing space, a lidar is usually deployed on an Automated Guided Vehicle (AGV), and then the cages are detected by the lidar to achieve automatic stacking of warehousing cages.

[0003] However, the solution using lidar for detection has a relatively high cost, and thus a method using a relatively inexpensive depth camera is derived to achieve cage stacking. In the existing methods using depth cameras, modeling is usually required, which consumes a lot of time and is not conducive to real-time performance. It is difficult to be applicable to different types of cages, and multiple sensors are also required to collect data, resulting in a high cost. Summary of the Invention

[0004] In view of the above problems, a method, device, electronic device and medium for stacking cages are provided to overcome or at least partially solve the above problems, including:

[0005] A method for stacking cages, including:

[0006] Collect RGB images and depth images of the upper cage and the lower cage through a depth camera; wherein, the depth camera is deployed on a handling device;

[0007] Use a pre-trained object detection model to identify the cage corner area of the RGB image;

[0008] Combine the cage corner area and the depth image to obtain a point cloud image of the cage corner area;

[0009] Determine the coordinate information of the position where the handling device needs to travel and the offset angle to be required according to the point cloud image, so as to control the handling device to stack cages.

[0010] Optionally, the determining the coordinate information of the position where the handling device needs to travel and the offset angle to be required according to the point cloud image, so as to control the handling device to stack cages includes:

[0011] Determine the coordinate information of the center points of the upper cage and the lower cage and the offset angle of the lower cage relative to the depth camera according to the point cloud image;

[0012] Determine the coordinate information of the position where the handling device needs to travel and the offset angle to be required according to the coordinate information of the center points and the offset angle, so as to control the handling device to stack cages.

[0013] Optionally, determining the coordinate information of the center points of the loading cage and the unloading cage and the offset angle of the unloading cage relative to the depth camera according to the point cloud image includes:

[0014] Determining the coordinate information of the cage angles of the loading cage and the unloading cage according to the point cloud image;

[0015] Determining the coordinate information of the center points of the loading cage and the unloading cage and the offset angle of the unloading cage relative to the depth camera according to the coordinate information of the cage angles.

[0016] Optionally, the coordinate information of the cage angle is the coordinate information of the centroid of the cage angle.

[0017] Optionally, determining the coordinate information of the position where the handling device needs to travel and the offset angle according to the coordinate information of the center point and the offset angle to control the handling device to stack cages includes:

[0018] Determining the coordinate information of the target point of the unloading cage according to the cage width of the unloading cage, the offset angle, and the coordinate information of the center point of the unloading cage;

[0019] Determining the coordinate information of the position where the handling device needs to travel and the offset angle according to the coordinate information of the center point of the loading cage, the coordinate information of the target point of the unloading cage, and the offset angle to control the handling device to stack cages.

[0020] Optionally, determining the coordinate information of the target point of the unloading cage according to the cage width of the unloading cage, the offset angle, and the coordinate information of the center point of the unloading cage includes:

[0021] Determining the offset distance according to the cage width of the unloading cage and the offset angle;

[0022] Determining the coordinate information of the target point of the unloading cage according to the coordinate information of the center point of the unloading cage and the offset distance.

[0023] Optionally, determining the coordinate information of the position where the handling device needs to travel and the offset angle according to the coordinate information of the center point of the loading cage, the coordinate information of the target point of the unloading cage, and the offset angle to control the handling device to stack cages includes:

[0024] Determining the coordinate information of the position where the handling device needs to travel according to the coordinate information of the center point of the loading cage and the coordinate information of the target point of the unloading cage;

[0025] Determine the angle by which the handling device needs to be offset according to the offset angle;

[0026] Control the handling device to stack the cages according to the coordinate information of the position where the handling device needs to travel and the angle by which it needs to be offset.

[0027] Optionally, the coordinate information of the position where the handling device needs to travel is the difference between the coordinate information of the center point of the upper cage and the coordinate information of the target point of the lower cage, and the angle by which the handling device needs to be offset is the same as the offset angle.

[0028] Optionally, the target point of the lower cage is the point corresponding to the center point of the upper cage during cage stacking.

[0029] Optionally, the point cloud image is an RGB image carrying coordinate information.

[0030] Optionally, the obtaining the point cloud image of the cage corner region by combining the cage corner region and the depth image includes:

[0031] Obtain the pixel point coordinates of the cage corner region in the RGB image;

[0032] Obtain the depth value in the depth image;

[0033] Generate the point cloud image of the cage corner region according to the pixel point coordinates and the depth value.

[0034] Optionally, the generating the point cloud image of the cage corner region according to the pixel point coordinates and the depth value includes:

[0035] Obtain the camera focal length and the optical center coordinates;

[0036] Generate the point cloud image of the cage corner region according to the pixel point coordinates, the depth value, the camera focal length, and the optical center coordinates.

[0037] Optionally, the target detection model has a backbone network, a neck network, and a head network, where:

[0038] The backbone network is used to extract features from the input image to obtain a feature map;

[0039] The neck network is used to perform multi-scale feature fusion on the feature map;

[0040] The head network is used to identify the fused feature map to obtain an identification result.

[0041] Optionally, the backbone network and the neck network are embedded with an efficient multi-scale attention module.

[0042] Optionally, the efficient multi-scale attention module is used for:

[0043] Performing global average pooling and convolution fusion on the input first feature to obtain a second feature;

[0044] Segmenting and activating the second feature, and weighting the segmented and activated second feature with the first feature to obtain a third feature;

[0045] Inputting the third feature into a cross-space learning module, activating the feature output by the cross-space learning module, and weighting it with the first feature to obtain an output fourth feature.

[0046] Optionally, the handling device includes: an automated guided vehicle.

[0047] Optionally, the depth camera is deployed at the middle position between two forks of the automated guided vehicle.

[0048] Optionally, the loading cage is a cage to be stacked onto the unloading cage.

[0049] A device for stacking cages, the device includes:

[0050] An image acquisition module, configured to acquire RGB images and depth images of a loading cage and an unloading cage through a depth camera; wherein, the depth camera is deployed on a handling device;

[0051] An image recognition module, configured to recognize the cage corner regions of the RGB images by using a pre-trained object detection model;

[0052] A point cloud image determination module, configured to combine the cage corner regions and the depth images to obtain point cloud images of the cage corner regions;

[0053] A handling device control module, configured to determine the coordinate information and the required offset angle of the position where the handling device needs to travel according to the point cloud images, so as to control the handling device to perform cage stacking.

[0054] An electronic device, including a processor, a memory, and a computer program stored on the memory and capable of running on the processor, where when the computer program is executed by the processor, the method for stacking cages as described above is implemented.

[0055] A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for stacking cages as described above is implemented.

[0056] The embodiments of the present invention have the following advantages:

[0057] In an embodiment of the present invention, RGB images and depth images of a loading cage and an unloading cage are collected through a depth camera; wherein, the depth camera is deployed on a handling device; a pre-trained object detection model is used to identify the cage corner area of the RGB image; combining the cage corner area and the depth image to obtain a point cloud image of the cage corner area; according to the point cloud image, the coordinate information of the position where the handling device needs to travel and the angle to be offset are determined, so as to control the handling device to stack the cages, realizing the completion of cage stacking by combining object detection technology and a depth camera, reducing the time required for processing, improving the real-time performance, being able to be applicable to different types of cages, and not requiring the use of multiple sensors to collect data, thus reducing the cost. Description of the Drawings

[0058] In order to more clearly illustrate the technical solutions of the present invention, the drawings required for the description of the present invention will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0059] Figure 1 is a flowchart of the steps of a method for stacking cages provided in some embodiments of the present invention;

[0060] Figure 2 is a top view schematic diagram of the structure of an automatic guided vehicle equipped with a depth camera provided in some embodiments of the present invention;

[0061] Figure 3 is a network architecture diagram of an object detection model provided in some embodiments of the present invention;

[0062] Figure 4 is a flowchart of an efficient multi-scale attention module provided in some embodiments of the present invention;

[0063] Figure 5 is a schematic diagram of offset angle calculation provided in some embodiments of the present invention;

[0064] Figure 6 is a schematic diagram of relative position calculation provided in some embodiments of the present invention;

[0065] Figure 7 is a flowchart of the steps of another method for stacking cages provided in some embodiments of the present invention;

[0066] Figure 8 is a structural block diagram of a device for stacking cages provided in some embodiments of the present invention. Detailed Embodiments

[0067] To make the above objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0068] In the related art, the methods for realizing cage stacking using a depth camera are as follows: Method 1: Establish a three-dimensional model of the cage by combining a depth camera and an RGB camera, match the three-dimensional models of the upper and lower cages, calculate the offset, and then control the driving displacement of the automatic guided vehicle to achieve complete matching of the three-dimensional models of the upper and lower cages, and then complete cage stacking. Method 2: Calculate the position deviation between the corners of the upper cage and the corners of the lower cage through the cooperation of multiple depth cameras, and then control the displacement of the automatic guided vehicle to complete stacking. Method 3: Calculate the depth information and angle offset information of the stacked cages by combining a depth camera and an inclination sensor, and then control the displacement of the automatic guided vehicle to complete stacking.

[0069] However, in this case, the three-dimensional modeling process of the cage consumes a lot of time and is not conducive to real-time performance; since there are many types of cages and the sizes of the stacked cage corners are different, the accuracy requirements are also different. Therefore, only cages of the same type can be modeled and analyzed, and in the prior art, multiple sensors are used to collect data, resulting in a high cost.

[0070] In the embodiments of the present invention, cage stacking is completed by combining target detection technology with a depth camera, reducing the processing time required, improving real-time performance, being applicable to different types of cages, and not requiring multiple sensors to collect data, reducing costs, as follows:

[0071] 1. Collect the RGB image and depth image of the cage through the depth camera, label the target area in the RGB image, use YOLOv8 (You Look Only Once version-8) as the target detection benchmark model, and use the Efficient Multi-scale Attention (EMA) module to improve the backbone network and neck network to enhance the detection performance of the target.

[0072] 2. Use the trained detection model to detect the target, combine the target area with the depth image to obtain the point cloud information of the target area, and calculate the coordinate information of the center point of the point cloud of the target area;

[0073] 3. Calculate the offset angle and position information of the upper and lower cages through the coordinate information of each area, and complete stacking by controlling the AGV.

[0074] In the embodiments of the present invention, an efficient multi-scale attention mechanism is used to improve the YOLOv8 object detection model, enhancing the extraction of information at different scales and the object detection ability. After performing object detection on the RGB image and then combining it with the corresponding depth map, the processing time is significantly reduced, which has great value in scenarios with high real-time requirements.

[0075] The present invention will be illustrated by the following examples with reference to the accompanying drawings:

[0076] Referring to Figure 1 , a flowchart of the steps of a method for stacking cages provided by some embodiments of the present invention is shown, which may specifically include the following steps:

[0077] Step 101, collect the RGB image and depth image of the loading cage and the unloading cage through a depth camera; wherein, the depth camera is deployed on the handling device.

[0078] In some examples, a depth camera refers to a type of camera that can capture and collect the distance from an object in space to the camera and generate a depth map.

[0079] In some examples, a cage refers to a type of logistics box used in warehousing and transportation, which is a new type of foldable product that replaces traditional wooden boxes and plastic boxes.

[0080] In some examples, an RGB image is an image composed of a combination of three primary colors: red (R), green (G), and blue (B).

[0081] In some examples, a depth image is a grayscale image, where each pixel point contains the distance information from the camera to the object. It is a commonly used image representation in computer vision for describing the three-dimensional structure of a scene.

[0082] In some embodiments of the present invention, the loading cage is the cage to be stacked on the unloading cage.

[0083] In practical applications, any cage to be stacked on another cage can be used as the loading cage, and the cage below the loading cage is the unloading cage.

[0084] In some embodiments of the present invention, the handling device includes: an automated guided vehicle. An automated guided vehicle is an industrial vehicle that loads goods automatically or manually, travels automatically along a set route, or tow a load carrier to a designated location, and then loads and unloads goods automatically or manually.

[0085] In some embodiments of the present invention, the depth camera is deployed at the middle position between the two forks of the automated guided vehicle.

[0086] In practical applications, a depth camera can be deployed on an automated guided vehicle (AGV) to collect RGB images and depth images of the blanking cage and the feeding cage that needs to be placed on the blanking cage in the warehouse.

[0087] As Figure 2 shown, a top-down schematic diagram of the structure of the AGV equipped with a depth camera, including: the upper left cage corner 1, the upper right cage corner 2, the lower left cage corner 3, the lower right cage corner 4, the AGV (i.e., the handling device) 5, the depth camera 6, the cage 7, and the AGV (i.e., the handling device) forklift 8. The camera is installed in the middle of the forklift on the AGV.

[0088] In practical applications, the AGV can trigger a collection task after reaching the designated position, call the depth camera to capture data, and the depth camera collects RGB images and depth images of the feeding cage and the blanking cage according to the collection task.

[0089] In some examples, an industrial control computer can also be deployed in the handling device. An industrial control computer is a computer device specifically used in the fields of industrial control and automation, also known as an industrial control computer or an embedded industrial control computer. Compared with a personal computer, an industrial control computer has higher reliability, stability, anti-interference ability, and security, and is suitable for long-term operation and stable control of industrial production processes in harsh environments.

[0090] Step 102: Use a pre-trained object detection model to identify the cage corner area in the RGB image.

[0091] In some examples, the object detection model can be YOLOv8. YOLOv8 is a state-of-the-art (SOTA) model that builds on the success of previous YOLO versions and introduces new features and improvements to further enhance performance and flexibility. Specific innovations include a new backbone network, a new Anchor-Free detection head, and a new loss function, and it can run on various hardware platforms from CPU to GPU.

[0092] In some examples, historical images (RGB images, depth images) collected by the depth camera can be used to create a cage stacking dataset based on the RGB images in the historical images, and the object detection model can be offline trained through the cage stacking dataset so that the object detection model can identify the cage corner area in the RGB image.

[0093] In some examples, the LabelImg annotation software can be used to create the dataset of stacked cages. LabelImg is an open-source image annotation tool that can be used to create and edit image annotation datasets. For example, a graphical image annotation tool (LabelImg) can be used to annotate the target area (cage corner) in the RGB image of the historical images collected by the depth camera, so as to obtain the dataset of stacked cages.

[0094] After annotation by the graphical image annotation tool, the annotated data can be divided into a training set and a test set, and then used for offline training of the object detection model.

[0095] In some examples, the depth camera can also collect the RGB images and depth images of the upper cage and the lower cage for corresponding storage as historical images for subsequent training of the object detection model.

[0096] In some examples, the trained object detection model can be deployed in an automated guided vehicle. After the depth camera collects the RGB images and depth images of the upper cage and the lower cage, the object detection model is used to identify the cage corner area in the RGB image.

[0097] In some examples, the trained object detection model can also be deployed on the server side. The automated guided vehicle can communicate with the server side. After the depth camera collects the RGB images and depth images of the upper cage and the lower cage, the data is transmitted to the automated guided vehicle and then sent to the server side by the automated guided vehicle, and then handed over to the object detection model in the server side to identify the cage corner area in the RGB image.

[0098] In some embodiments of the present invention, the object detection model has a backbone network, a neck network, and a head network, where:

[0099] The backbone network is used to extract features from the input image to obtain a feature map;

[0100] The neck network is used to perform multi-scale feature fusion on the feature map;

[0101] The head network is used to identify the fused feature map to obtain an identification result.

[0102] In practical applications, when the object detection model identifies the cage corner area of the input RGB image, it can first use the backbone network to extract features from the input RGB image to obtain a feature map, input the feature map into the neck network for multi-scale feature fusion, and then use the head network to identify the fused feature map to obtain an identification result, which is used as the cage corner area of the input RGB image.

[0103] In some examples, such as Figure 3As shown in the figure, the backbone network includes: Convolution (Conv), C2f module (Convolution to Feature), SPPF module, and Efficient Multi-scale Attention (EMA) module.

[0104] Among them, convolution is an operation. Similar to addition, subtraction, multiplication, division, dot product and cross product of matrices, etc., it has its own operation rules.

[0105] Among them, the C2f module is a module that combines a Convolutional Neural Network (CNN) and a Fully Connected Neural Network (FNN).

[0106] Among them, Spatial Pyramid Pooling (SPP) is a technique used for image recognition and object detection. Its role is to extract and encode features of images at different scales. It can resize any-sized input image to a fixed size and generate a feature vector of a fixed length. And SPPF changes the original parallel structure of SPP to a serial structure, reducing the computational amount (faster, F means Fast), and playing the same role as SPP.

[0107] In some examples, as Figure 3 shown, the neck network includes: Concat, Upsample, C2f module, EMA module, and Conv (convolution).

[0108] Among them, the Concat operation is often used to connect different feature tensors, thereby improving the performance of the model. In a Convolutional Neural Network (CNN), the feature maps of different convolutional layers can be concatenated to construct a more complex feature representation; in a Recurrent Neural Network (RNN), the outputs of different time steps can be concatenated to improve the model's ability to model sequential data.

[0109] Among them, Upsample is a signal processing technique used to increase the sampling rate of a signal or improve the resolution of a signal. In image processing, upsample can enlarge a low-resolution image to a high-resolution image through an interpolation algorithm.

[0110] In some examples, as Figure 3 shown, the head network includes three detection heads of different scales: Detect1, Detect2, and Detect3. Among them, the detection head (Detect) is used to output predictions and calculate the loss with the true annotations, and adjust the output predictions through gradient descent.

[0111] In some embodiments of the present invention, the backbone network and the neck network are embedded with an efficient multi-scale attention module. Since the angle of the lower material cage in the dataset is small (the distance between the lower material cage and the depth camera is relatively far, and it is small in the image), problems such as missed detection or poor detection effect may occur in the material cage angle area. To improve the detection efficiency of the model, the backbone network and the neck network are improved by adopting an efficient multi-scale attention module to enhance the detection performance of the material cage angle area.

[0112] In some embodiments of the present invention, the efficient multi-scale attention module is used to: perform global average pooling and convolutional fusion on the input first feature to obtain a second feature; segment and activate the second feature, and weight the segmented and activated second feature with the first feature to obtain a third feature; input the third feature into a cross-space learning module, activate the feature output by the cross-space learning module, and weight it with the first feature to obtain the output fourth feature.

[0113] As Figure 3 shown, an EMA (i.e., efficient multi-scale attention) module is embedded in the backbone network and the neck network of the object detection model to enhance the detection performance of the object.

[0114] In some examples, the structure of the efficient multi-scale attention module is as Figure 4 shown, where C is the number of channels, H is the height of the feature map, W is the width of the feature map, and G is the grouping parameter.

[0115] As shown in formula (1), the input feature X1 is grouped (Groups), and global average pooling is performed in the X and Y axis directions. The pooled data is concatenated (Concat), and then fused through a 1×1 convolution:

[0116] X2 = Conv 1×1 (Concat(XAvgPool(Groups(X1), YAvgPool(Groups(X1)))) (1)

[0117] As shown in formula (2), the feature X2 is segmented and activated (Sigmoid) respectively, and after activation, it is weighted (Re-weight) with the input feature:

[0118] X3 = Re-Weight(Sigmoid(Split(X2))) (2)

[0119] As shown in formulas (3) and (4), the weighted feature X3 is input into the Cross-spatial learning module, then the output feature is activated (Sigmoid), and finally weighted (Re-weight) attention is performed with the original input feature. Among them, the left and right paths of the cross-spatial learning module adopt similar calculation methods, both passing through global average pooling (AvgPool), Softmax function, Matmul function, completing interactive learning through cross-multiplication, and then adding the two-path features:

[0120] X4 = Matmul(Softmax(AvgPool(GroupNorm(X3))), Conv3×3(X1))

[0121] +Matmul(Softmax(AvgPool(Conv3×3(X1))), GroupNorm(X3))(3)

[0122] Output = Re-Weight(Sigmoid(X4)) (4)

[0123] In the embodiments of the present invention, by adopting an efficient multi-scale attention module to improve the YOLOv8 object detection model, the extraction of different-scale information is enhanced, and the object detection ability is enhanced; the calculation method of the offset position of the loading and unloading cage corner is improved.

[0124] Step 103, combining the cage corner area and the depth image to obtain a point cloud image of the cage corner area.

[0125] In some embodiments of the present invention, the point cloud image is an RGB image carrying coordinate information.

[0126] After the object detection model identifies the cage corner area of the input RGB image, the obtained cage corner area can be calculated to output the pixel coordinate information of the cage corner area, and then the depth image collected by the depth camera is calculated to generate point cloud information. Combining the pixel coordinate information of the cage corner area with the point cloud information, a point cloud image of the cage corner area carrying coordinate information is obtained.

[0127] In some embodiments of the present invention, the combining the cage corner area and the depth image to obtain a point cloud image of the cage corner area includes:

[0128] Obtain the pixel point coordinates of the cage corner area in the RGB image; obtain the depth value in the depth image; generate a point cloud image of the cage corner area according to the pixel point coordinates and the depth value.

[0129] In practical applications, before combining the hopper corner region and the depth image, the pixel coordinates in the RGB image of the hopper corner region can be obtained, and the depth value in the depth image can be obtained; by combining the pixel coordinates and the depth value, the point cloud image of the hopper corner region can be generated.

[0130] In some embodiments of the present invention, generating the point cloud image of the hopper corner region according to the pixel coordinates and the depth value includes:

[0131] Obtaining the camera focal length and the optical center coordinates; generating the point cloud image of the hopper corner region according to the pixel coordinates, the depth value, the camera focal length, and the optical center coordinates.

[0132] In practical applications, before generating the point cloud image of the hopper corner region, the camera focal length and the optical center coordinates can be obtained; an RGB image carrying coordinate information is calculated according to the pixel coordinates, the depth value, the camera focal length, and the optical center coordinates, and used as the point cloud image of the hopper corner region.

[0133] In practical applications, the accurate conversion of the depth image is a prerequisite for accurate position calculation. The calculation process of generating point cloud information by calculating the depth image collected by the depth camera can be as shown in formula (5):

[0134]

[0135] Among them, u and v represent the pixel coordinates of the RGB image; K represents the internal parameter matrix of the camera, fx and fy are the camera focal lengths, cx and cy represent the optical center; P represents the coordinates in the camera coordinate system. After sorting, the specific solution formula (6) is obtained:

[0136] X = Z(u - c x ) / f x

[0137] Y = Z(v - c y ) / f y

[0138] Z = d (6)

[0139] Among them, the Z value is the depth value d measured by the camera, that is, the distance from the real point to the camera plane.

[0140] In practical applications, after the target detection model combines the pixel coordinate information and the point cloud information of the hopper corner region, a point cloud image carrying coordinate information of the hopper corner region can be obtained.

[0141] In some examples, by performing target detection on the RGB image and then combining it with the corresponding depth map, the processing time required is greatly reduced, which is of great value for occasions with high real-time requirements.

[0142] Step 104: Based on the point cloud image, determine the coordinate information of the position where the handling device needs to travel and the offset angle required, so as to control the handling device to stack the cages.

[0143] After the target detection model obtains the point cloud image, the point cloud image can be calculated again, so as to determine the coordinate information of the position where the handling device needs to travel and the offset angle required, so as to control the handling device to stack the cages.

[0144] In some embodiments of the present invention, the determining the coordinate information of the position where the handling device needs to travel and the offset angle required based on the point cloud image to control the handling device to stack the cages includes:

[0145] Based on the point cloud image, determine the coordinate information of the center points of the loading cage and the unloading cage, and the offset angle of the unloading cage relative to the depth camera; based on the coordinate information of the center points and the offset angle, determine the coordinate information of the position where the handling device needs to travel and the offset angle required, so as to control the handling device to stack the cages.

[0146] After the target detection model obtains the point cloud image, the coordinate information of the cage angles of the loading cage and the unloading cage can be obtained from the point cloud image, and then the coordinate information of the center points of the loading cage and the unloading cage can be obtained through the coordinate information of the cage angles. For example, Figure 5 the coordinate information of the center point of the unloading cage in is a, and the coordinate information of the center point of the loading cage is b. The target detection model can also obtain the offset angle of the unloading cage relative to the depth camera through the coordinate information of the cage angles.

[0147] After the target detection model obtains the coordinate information of the center point and the offset angle, the coordinate information of the position where the handling device needs to travel and the offset angle required can be determined through calculation, so as to control the handling device to stack the cages according to the coordinate information of the position where the handling device needs to travel and the offset angle required.

[0148] In some embodiments of the present invention, the determining the coordinate information of the center points of the loading cage and the unloading cage and the offset angle of the unloading cage relative to the depth camera based on the point cloud image includes:

[0149] Based on the point cloud image, determine the coordinate information of the cage angles of the loading cage and the unloading cage; based on the coordinate information of the cage angles, determine the coordinate information of the center points of the loading cage and the unloading cage and the offset angle of the unloading cage relative to the depth camera.

[0150] In practical applications, the target detection model can obtain the coordinate information of the cage angles of the loading cage and the unloading cage from the point cloud image, such as formula (7):

[0151]

[0152] Among them, Pc represents the center point (centroid) of the point cloud image, and i = 0, 1, 2, …, n represents the coordinate information of each particle of the point cloud image. Through the calculation of formula (7), the coordinate information of the cage angles of the loading cage and the unloading cage can be obtained.

[0153] After the target detection model obtains the coordinate information of the cage angles of the loading cage and the unloading cage, the coordinate information of the center points of the loading cage and the unloading cage can be obtained through the coordinate information of the cage angles, and then the offset angle of the unloading cage relative to the depth camera can be obtained through the coordinate information of the cage angles.

[0154] In some embodiments of the present invention, the coordinate information of the cage angle is the coordinate information of the centroid of the cage angle, as Figure 5 shown, P1, P2, P3, and P4 are the coordinate information of the centroid of the cage angle.

[0155] As Figure 5 shown, the target detection model obtains the coordinate information of the cage angles of the loading cage and the unloading cage through formula (7). Among them, the coordinate information of the cage angle of the loading cage is respectively: P1(X1, Y1, Z1), P2(X2, Y2, Z2); the coordinate information of the cage angle of the unloading cage is respectively: P3(X3, Y3, Z3), P4(X4, Y4, Z4).

[0156] In some examples, by adding the coordinate information of the two cage angles of the loading cage and dividing by 2 (i.e., (P1 + P2) / 2), the coordinate information of the center point of the loading cage is obtained, as Figure 5 shown, point b is the coordinate information of the center point of the loading cage.

[0157] In some examples, by adding the coordinate information of the two cage angles of the unloading cage and dividing by 2 (i.e., (P3 + P4) / 2), the coordinate information of the center point of the loading cage is obtained, as Figure 5 shown, point a is the coordinate information of the center point of the unloading cage.

[0158] In some examples, the offset angle of the unloading cage relative to the depth camera can also be obtained through the coordinate information of the two cage angles of the loading cage. For example, the offset angle of the lower cage relative to the depth camera is obtained through P1 and P2. The specific formula (8) is as follows:

[0159] θπ = atan2(P2[2] - P1[2], P2[0] - P1[0])

[0160] θ = (θπ × 180) / π (8)

[0161] Among them, θπ represents the radian value, and θ represents the converted actual angle value.

[0162] In some embodiments of the present invention, determining the coordinate information of the position where the handling device needs to travel and the angle of deviation according to the coordinate information of the center point and the angle of deviation, so as to control the handling device to stack the cages, includes:

[0163] Determine the coordinate information of the target point of the lower cage according to the cage width of the lower cage, the angle of deviation, and the coordinate information of the center point of the lower cage; determine the coordinate information of the position where the handling device needs to travel and the angle of deviation according to the coordinate information of the center point of the upper cage, the coordinate information of the target point of the lower cage, and the angle of deviation, so as to control the handling device to stack the cages.

[0164] In some examples, in order to achieve cage stacking, cages of the same size can be used.

[0165] In practical applications, according to the cage width of the lower cage, the angle of deviation from the depth camera relative to the lower cage, and the coordinate information of the center point of the lower cage, the offset distance of the center point of another lower cage can be calculated, and the coordinate information of the target point of the lower cage can be obtained according to the offset distance of the center point of another lower cage. As Figure 6 shown, the coordinate information of point c is the coordinate information of the center point of the lower cage.

[0166] In some embodiments of the present invention, the target point of the lower cage is the point corresponding to the center point of the upper cage during cage stacking.

[0167] In practical applications, since cages of the same size are generally used, the target point of the lower cage calculated by the target detection model is the point corresponding to the center point of the upper cage during cage stacking.

[0168] In some embodiments of the present invention, determining the coordinate information of the target point of the lower cage according to the cage width of the lower cage, the angle of deviation, and the coordinate information of the center point of the lower cage includes:

[0169] Determine the offset distance according to the cage width of the lower cage and the angle of deviation;

[0170] Determine the coordinate information of the target point of the lower cage according to the coordinate information of the center point of the lower cage and the offset distance.

[0171] In practical applications, according to the cage width of the lower cage, the angle of deviation from the depth camera relative to the lower cage, and the coordinate information of the center point of the lower cage, the offset distance of the center point of another lower cage can be calculated, and the coordinate information of the target point of the lower cage can be obtained according to the offset distance of the center point of another lower cage. As Figure 6 shown, the coordinate information of point c is the coordinate information of the center point of the lower cage.

[0172] As shown Figure 6 in the figure, triangle △ace is similar to △efP3, so ∠eP3f = ∠ace can be obtained. Given that the width of the cage is the horizontal distance of ac, the offset distance of point c can be calculated as shown in formula (9):

[0173] ΔX = ac × sin(θ)

[0174] ΔZ = ac × cos(θ) (9)

[0175] After calculating the offset distance of point c through formula (9), the coordinate information of point c can be inferred and used as the coordinate information of the target point of the lower cage, as shown in formula (10):

[0176] c = ((X3 + X4) / 2 - ΔX, (Y3 + Y4) / 2, (Z3 + Z4) / 2 + ΔZ) (10)

[0177] In some embodiments of the present invention, determining the coordinate information of the position where the handling device needs to travel and the angle of deviation according to the coordinate information of the center point of the upper cage, the coordinate information of the target point of the lower cage, and the angle of deviation, so as to control the handling device to stack the cages, includes:

[0178] Determining the coordinate information of the position where the handling device needs to travel according to the coordinate information of the center point of the upper cage and the coordinate information of the target point of the lower cage; determining the angle of deviation that the handling device needs according to the angle of deviation; controlling the handling device to stack the cages according to the coordinate information of the position where the handling device needs to travel and the angle of deviation.

[0179] After obtaining the coordinate information of the center point of the upper cage, the coordinate information of the target point of the lower cage, and the angle of deviation of the lower cage relative to the depth camera, the coordinate information of the position where the handling device needs to travel and the angle of deviation can be calculated.

[0180] In some embodiments of the present invention, the coordinate information of the position where the handling device needs to travel is the difference between the coordinate information of the center point of the upper cage and the coordinate information of the target point of the lower cage, and the angle of deviation that the handling device needs is the same as the angle of deviation.

[0181] In practical applications, the difference between the coordinate information of the center point of the upper cage and the coordinate information of the target point of the lower cage can be used as the coordinate information of the position where the handling device needs to travel, and the angle of deviation of the lower cage relative to the depth camera can be used as the angle of deviation that the handling device needs.

[0182] For example, after calculating through formulas (1) to (6) to obtain a (coordinate information of the center point of the blanking cage), b (coordinate information of the center point of the feeding cage), c (coordinate information of the target point of the blanking cage), and θ (offset angle of the blanking cage relative to the depth camera), the target detection model then calculates the coordinate information of the position where the handling device needs to travel and the offset angle, as shown in formula (11):

[0183] AGV coord = c - b

[0184] AGV θ = θ (11)

[0185] After calculating the coordinate information of the position where the handling device needs to travel and the offset angle, it can be sent to the industrial control computer in the handling device. The industrial control computer controls the handling device to stack the cages according to the coordinate information of the position where the handling device needs to travel and the offset angle.

[0186] The following is an exemplary description in conjunction with the attached Figure 7 :

[0187] S1, Cage corner recognition training.

[0188] By using the historical images collected by the depth camera and using the graphic image annotation tool (LabelImg) to annotate the target areas (cage corners) in the RGB images of the historical images.

[0189] Among them, the historical images can include the RGB images and depth images of the feeding cage and the blanking cage.

[0190] After annotation by the graphic image annotation tool, the annotated data can also be divided into a training set and a test set, and then given to the target detection model for offline training.

[0191] S2, Recognition.

[0192] The depth camera collects the RGB images and depth images of the feeding cage and the blanking cage on-site, and inputs the RGB images into the trained model (target detection model) to perform the recognition of the cage corner area in the RGB images.

[0193] S3, Fusion.

[0194] After the target detection model recognizes the cage corner area of the input RGB image, it can calculate the obtained cage corner area, output the pixel coordinate information of the cage corner area, and then calculate the depth image collected by the depth camera to generate point cloud information. The pixel coordinate information of the cage corner area is image-fused with the point cloud information to obtain the point cloud image of the cage corner area.

[0195] S4, Position and angle calculation.

[0196] The target detection model can calculate the centroid of the region based on the point cloud image (calculate the coordinate information of the cage angles of the loading cage and the unloading cage, find the coordinate information of the center points of the loading cage and the unloading cage, and find the offset angle of the unloading cage relative to the depth camera), and then perform relative position calculation (calculate the offset distance of the center point of another unloading cage of the unloading cage, and find the coordinate information of the target point of the unloading cage based on the offset distance of the center point of another unloading cage).

[0197] After obtaining the coordinate information of the center point of the unloading cage, the coordinate information of the center point of the loading cage, the coordinate information of the target point of the c unloading cage, and the offset angle of the lower unloading cage relative to the depth camera, the target detection model then calculates the coordinate information and the required offset angle of the position where the handling device needs to travel, and obtains the coordinate position where the handling device needs to travel and the required offset angle.

[0198] The target detection model obtains the coordinate position where the handling device needs to travel and the required offset angle, and can send them to the industrial control computer in the handling device. The industrial control computer controls the handling device to stack the cages according to the coordinate position where it needs to travel and the required offset angle.

[0199] In some embodiments of the present invention, by embedding an efficient multi-scale attention module in the backbone network and the neck network of the target detection model, a comparative experiment is carried out with the number of parameters and the accuracy mAP(0.5) of the unimproved target detection model. The experiment further proves the effectiveness of the improved model. As shown in Table 1, it can be seen from the table that for the improved model, when simply improving the backbone network, the mAP(0.5) value reaches 98.0%, and the number of parameters remains basically unchanged; when adding improvements to both the backbone and the neck (Backbone+Neck), the mAP value is 98.6% while slightly increasing the number of parameters. The experiment shows that the improved model has higher accuracy.

[0200]

[0201] Table 1

[0202] In the embodiments of the present invention, through a depth camera, RGB images and depth images of the loading cage and the unloading cage are collected; wherein, the depth camera is deployed on the handling device; a pre-trained target detection model is used to identify the cage angle region in the RGB image; combining the cage angle region and the depth image, a point cloud image of the cage angle region is obtained; according to the point cloud image, the coordinate information and the required offset angle of the position where the handling device needs to travel are determined to control the handling device to stack the cages, realizing the completion of cage stacking by combining the target detection technology and the depth camera, reducing the time required for processing, improving the real-time performance, being able to be applicable to different types of cages, and not requiring multiple sensors to collect data, thus reducing the cost.

[0203] It should be noted that for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the described action sequence, because according to the embodiments of the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential for the embodiments of the present invention.

[0204] Referring to Figure 8 , a schematic structural diagram of a device for stacking material cages provided by some embodiments of the present invention is shown, which may specifically include the following modules:

[0205] An image acquisition module 801, configured to acquire RGB images and depth images of the upper material cage and the lower material cage through a depth camera; wherein, the depth camera is deployed on a handling device.

[0206] An image recognition module 802, configured to use a pre-trained object detection model to recognize the cage corner area of the RGB image.

[0207] A point cloud image determination module 803, configured to combine the cage corner area and the depth image to obtain a point cloud image of the cage corner area.

[0208] A handling device control module 804, configured to determine coordinate information of the position where the handling device needs to travel and the offset angle to be offset according to the point cloud image, so as to control the handling device to perform material cage stacking.

[0209] In some embodiments of the present invention, the handling device control module 804 includes:

[0210] A point cloud image calculation sub-module, configured to determine coordinate information of the center points of the upper material cage and the lower material cage and the offset angle of the lower material cage relative to the depth camera according to the point cloud image;

[0211] A handling device adjustment sub-module, configured to determine coordinate information of the position where the handling device needs to travel and the offset angle to be offset according to the coordinate information of the center point and the offset angle, so as to control the handling device to perform material cage stacking.

[0212] In some embodiments of the present invention, the point cloud image calculation sub-module includes:

[0213] A coordinate information determination unit of the cage corner, configured to determine coordinate information of the cage corners of the upper material cage and the lower material cage according to the point cloud image;

[0214] The coordinate information calculation unit of the cage corner is used to determine the coordinate information of the center points of the loading cage and the unloading cage and the offset angle of the unloading cage relative to the depth camera according to the coordinate information of the cage corner.

[0215] In some embodiments of the present invention, the coordinate information of the cage corner is the coordinate information of the centroid of the cage corner.

[0216] In some embodiments of the present invention, the handling equipment adjustment sub-module includes:

[0217] The coordinate information determination unit of the target point is used to determine the coordinate information of the target point of the unloading cage according to the cage width of the unloading cage, the offset angle, and the coordinate information of the center point of the unloading cage;

[0218] The position and angle determination unit of the handling equipment is used to determine the coordinate information of the position where the handling equipment needs to travel and the offset angle to be offset according to the coordinate information of the center point of the loading cage, the coordinate information of the target point of the unloading cage, and the offset angle, so as to control the handling equipment to stack the cages.

[0219] In some embodiments of the present invention, the coordinate information determination unit of the target point includes:

[0220] The offset distance determination sub-unit is used to determine the offset distance according to the cage width of the unloading cage and the offset angle;

[0221] The coordinate information determination sub-unit of the target point of the unloading cage is used to determine the coordinate information of the target point of the unloading cage according to the coordinate information of the center point of the unloading cage and the offset distance.

[0222] In some embodiments of the present invention, the position and angle determination unit of the handling equipment includes:

[0223] The position determination sub-unit of the handling equipment is used to determine the coordinate information of the position where the handling equipment needs to travel according to the coordinate information of the center point of the loading cage and the coordinate information of the target point of the unloading cage;

[0224] The offset angle determination sub-unit of the handling equipment is used to determine the offset angle that the handling equipment needs to be offset according to the offset angle;

[0225] The sub-unit for controlling the handling equipment according to relevant information is used to control the handling equipment to stack the cages according to the coordinate information of the position where the handling equipment needs to travel and the offset angle to be offset.

[0226] In some embodiments of the present invention, the coordinate information of the position where the handling device needs to travel is the difference between the coordinate information of the center point of the loading cage and the coordinate information of the target point of the unloading cage, and the deviation angle required by the handling device is the same as the deviation angle.

[0227] In some embodiments of the present invention, the target point of the unloading cage is the point corresponding to the center point of the loading cage when the cages are stacked.

[0228] In some embodiments of the present invention, the point cloud image is an RGB image carrying coordinate information.

[0229] In some embodiments of the present invention, the point cloud image determination module 803 includes:

[0230] The pixel point coordinate determination sub-module is used to obtain the pixel point coordinates in the RGB image of the cage corner area;

[0231] The depth value acquisition sub-module is used to obtain the depth value in the depth image;

[0232] The point cloud image generation sub-module is used to generate the point cloud image of the cage corner area according to the pixel point coordinates and the depth value.

[0233] In some embodiments of the present invention, the point cloud image generation sub-module includes:

[0234] The camera parameter acquisition unit is used to acquire the camera focal length and the optical center coordinates;

[0235] The point cloud generation unit according to relevant information is used to generate the point cloud image of the cage corner area according to the pixel point coordinates, the depth value, the camera focal length, and the optical center coordinates.

[0236] In some embodiments of the present invention, the target detection model has a backbone network, a neck network, and a head network, where:

[0237] The backbone network is used to extract features from the input image to obtain a feature map;

[0238] The neck network is used to perform multi-scale feature fusion on the feature map;

[0239] The head network is used to identify the fused feature map to obtain an identification result.

[0240] In some embodiments of the present invention, the backbone network and the neck network are embedded with an efficient multi-scale attention module.

[0241] In some embodiments of the present invention, the efficient multi-scale attention module is used for:

[0242] The first feature of the input is subjected to global average pooling and convolutional fusion to obtain a second feature;

[0243] The second feature is segmented and activated, and the segmented and activated second feature is weighted with the first feature to obtain a third feature;

[0244] The third feature is input into a cross-space learning module, the feature output by the cross-space learning module is activated, and is weighted with the first feature to obtain an output fourth feature.

[0245] In some embodiments of the present invention, the handling device includes: an automated guided vehicle.

[0246] In some embodiments of the present invention, the depth camera is deployed at the middle position between the two forks of the automated guided vehicle.

[0247] In some embodiments of the present invention, the loading cage is a cage to be stacked on the unloading cage.

[0248] Some embodiments of the present invention also provide an electronic device, which may include a processor, a memory, and a computer program stored on the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the method of stacking loading cages as described above.

[0249] Some embodiments of the present invention also provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the method of stacking loading cages as described above.

[0250] For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiments.

[0251] Each embodiment in this specification is described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, refer to each other.

[0252] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a device, or a computer program product. Therefore, the embodiments of the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0253] Embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the functions specified in one or more flows and / or blocks Figure 1 in one or more flows and / or blocks Figure 1 or in one or more blocks.

[0254] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in one or more flows and / or blocks Figure 1 in one or more flows and / or blocks Figure 1 or in one or more blocks.

[0255] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, such that a series of operation steps are executed on the computer or other programmable terminal device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more flows and / or blocks Figure 1 in one or more flows and / or blocks Figure 1 or in one or more blocks.

[0256] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0257] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the above elements.

[0258] The above provides a detailed introduction to a method, device, electronic device and medium for stacking material cages. In this text, specific examples are used to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for stacking feed cages, characterized in that, The method includes: Collecting RGB images and depth images of the loading cage and the unloading cage through a depth camera; wherein, the depth camera is deployed on the handling device; Using a pre-trained object detection model to identify the cage corner area of the RGB image; Combining the cage corner area and the depth image to obtain a point cloud image of the cage corner area; Determining the coordinate information of the position where the handling device needs to travel and the offset angle required based on the point cloud image, so as to control the handling device to stack the cages.

2. The method according to claim 1, characterized in that, The determining the coordinate information of the position where the handling device needs to travel and the offset angle required based on the point cloud image, so as to control the handling device to stack the cages includes: Determining the coordinate information of the center points of the loading cage and the unloading cage and the offset angle of the unloading cage relative to the depth camera according to the point cloud image; Determining the coordinate information of the position where the handling device needs to travel and the offset angle required based on the coordinate information of the center points and the offset angle, so as to control the handling device to stack the cages.

3. The method according to claim 2, wherein The determining the coordinate information of the center points of the loading cage and the unloading cage and the offset angle of the unloading cage relative to the depth camera according to the point cloud image includes: Determining the coordinate information of the cage corners of the loading cage and the unloading cage according to the point cloud image; Determining the coordinate information of the center points of the loading cage and the unloading cage and the offset angle of the unloading cage relative to the depth camera according to the coordinate information of the cage corners.

4. The method according to claim 3, characterized in that, The coordinate information of the cage corner is the coordinate information of the centroid of the cage corner.

5. The method according to claim 2, wherein The determining the coordinate information of the position where the handling device needs to travel and the offset angle required based on the coordinate information of the center points and the offset angle, so as to control the handling device to stack the cages includes: Determining the coordinate information of the target point of the unloading cage according to the cage width of the unloading cage, the offset angle, and the coordinate information of the center point of the unloading cage; Determining the coordinate information of the position where the handling device needs to travel and the offset angle required based on the coordinate information of the center point of the loading cage, the coordinate information of the target point of the unloading cage, and the offset angle, so as to control the handling device to stack the cages.

6. The method according to claim 5, characterized in that The determining the coordinate information of the target point of the unloading cage according to the cage width of the unloading cage, the offset angle, and the coordinate information of the center point of the unloading cage includes: Determining the offset distance according to the cage width of the unloading cage and the offset angle; Determining the coordinate information of the target point of the unloading cage according to the coordinate information of the center point of the unloading cage and the offset distance.

7. The method according to claim 5, characterized in that The determining the coordinate information of the position where the handling device needs to travel and the offset angle required based on the coordinate information of the center point of the loading cage, the coordinate information of the target point of the unloading cage, and the offset angle, so as to control the handling device to stack the cages includes: Determining the coordinate information of the position where the handling device needs to travel according to the coordinate information of the center point of the loading cage and the coordinate information of the target point of the unloading cage; Determining the offset angle required for the handling device according to the offset angle; Control the handling device to stack the cages according to the coordinate information of the position where the handling device needs to travel and the angle of deviation required.

8. The method according to claim 7, wherein The coordinate information of the position where the handling device needs to travel is the difference between the coordinate information of the center point of the upper cage and the coordinate information of the target point of the lower cage, and the angle of deviation required for the handling device is the same as the angle of deviation.

9. The method according to claim 5, characterized in that, The target point of the lower cage is the point corresponding to the center point of the upper cage during cage stacking.

10. The method according to claim 1, characterized in that, The point cloud image is an RGB image carrying coordinate information.

11. The method according to claim 10, characterized in that, Combining the cage corner region and the depth image to obtain the point cloud image of the cage corner region includes: Obtain the pixel point coordinates of the cage corner region in the RGB image; Obtain the depth value in the depth image; Generate the point cloud image of the cage corner region according to the pixel point coordinates and the depth value.

12. The method according to claim 11, wherein Generating the point cloud image of the cage corner region according to the pixel point coordinates and the depth value includes: Obtain the camera focal length and the optical center coordinates; Generate the point cloud image of the cage corner region according to the pixel point coordinates, the depth value, the camera focal length, and the optical center coordinates.

13. The method according to any one of claims 1 to 12, characterized in that The target detection model has a backbone network, a neck network, and a head network, where: The backbone network is used to extract features from the input image to obtain a feature map; The neck network is used to perform multi-scale feature fusion on the feature map; The head network is used to identify the fused feature map to obtain an identification result.

14. The method according to claim 13, wherein The backbone network and the neck network are embedded with an efficient multi-scale attention module.

15. The method according to claim 14, characterized in that, The efficient multi-scale attention module is used for: Perform global average pooling and convolutional fusion on the input first feature to obtain a second feature; Segment and activate the second feature, and weight the segmented and activated second feature with the first feature to obtain a third feature; Input the third feature into the cross-space learning module, activate the feature output by the cross-space learning module, and weight it with the first feature to obtain the output fourth feature.

16. The method according to claim 1, characterized in that, The handling device includes: An automated guided vehicle.

17. The method according to claim 15, wherein The depth camera is deployed at the middle position between the two forks of the automated guided vehicle.

18. The method according to claim 1, wherein The upper cage is the cage to be stacked on the lower cage.

19. A device for stacking material cages, characterized in that, The device includes: An image acquisition module, which is used to collect the RGB image and the depth image of the upper cage and the lower cage through a depth camera; wherein, the depth camera is deployed on the handling device; An image recognition module, which is used to identify the cage corner region of the RGB image by using a pre-trained target detection model; A point cloud image determination module, which is used to combine the cage corner region and the depth image to obtain the point cloud image of the cage corner region; A handling device control module, which is used to determine the coordinate information of the position where the handling device needs to travel and the angle of deviation required according to the point cloud image, so as to control the handling device to stack the cages.

20. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored on the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the method for cage stacking as described in any one of claims 1 to 18.

21. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the method for stacking cages as described in any one of claims 1 to 18.

Images