Box-type package rapid pose calculation method based on lightweight Yolov8-RTMPose model

Abstract

The invention provides a rapid position and posture calculation method for a box type package based on lightweight Yolov8-RTMPose. The method comprises the following steps: (1) acquiring RGB and depth images through a 3D camera and preprocessing the RGB and depth images; (2) a Yolov8 structure is improved, an Officient RepGFPN and BetterCIoU loss function is introduced, and model training is completed; (3) detecting a box-type packaging target in the image, and extracting a region-of-interest image with high confidence; (4) lightweight design is carried out on the RTMPose model, an MPC structure and an NATTEN attention mechanism are integrated, and the key point detection precision is improved; (5) inputting an ROI image, outputting 2D key points, and removing points with low confidence or geometric anomalies; (6) converting the 2D key points into three-dimensional coordinates in combination with the depth map and camera internal reference; and (7) estimating an initial pose through an EPnP algorithm, optimizing a re-projection error by using an LM algorithm, and outputting a six-degree-of-freedom pose parameter. The method is high in detection precision, high in operation speed and suitable for box type package positioning and grabbing tasks in complex scenes such as logistics and storage.

Description

TECHNICAL FIELD

[0001] The present application is suitable for the field of automatic grabbing of box packaging objects in logistics sorting, and particularly relates to an efficient grabbing method combining an improved Yolov8 target detection model, an RTMPose key point detection model, and an EPnP (Efficient Perspective-n-Point) and LM (Levenberg-Marquardt) pose solving algorithm. A lightweight neural network structure is adopted to significantly reduce the model parameter quantity and computational complexity while ensuring the accuracy of detection and key point positioning. A top-down multi-stage detection strategy is adopted to first realize object-level target positioning by the improved Yolov8 model, and then accurately regress the structural key points of the box packaging surface by the RTMPose model to realize an efficient calculation process from coarse to fine. Finally, the EPnP algorithm and the LM optimization method are fused to still stably calculate the pose of the target in the three-dimensional space when there is a slight error in the input key points, providing a high-robustness pose estimation result for accurate grabbing. Through the cooperative improvement of the above structure and algorithm, the present application can still realize high-precision and low-delay box packaging sorting and grabbing tasks under complex conditions such as multiple targets, occlusion, and light changes in the industrial field, and has good practicality and promotional value. BACKGROUND

[0002] With the continuous advancement of Industry 4.0 and intelligent manufacturing, the logistics automation sorting scene puts forward higher requirements for intelligent detection and accurate grabbing technology of packaging objects. Especially in large-scale warehousing and sorting applications in e-commerce, pharmaceutical, food and other industries, box packaging objects have characteristics such as various specifications (such as paper boxes, plastic boxes), complex surface printing (such as labels, barcodes and other information), and uncertain stacking methods, which cause problems in accuracy, speed and environmental adaptability of the detection system.

[0003] The current mainstream detection and recognition methods mainly rely on two types of perception schemes, a point cloud matching method based on a laser radar and a visual template matching method based on an RGB-D camera.

[0004] The point cloud matching scheme based on laser generally obtains three-dimensional point cloud data of the packaging object through a laser radar, and realizes recognition and registration combined with a preset geometric template. This method has strong anti-light interference ability and good stability, but has the following problems: (1) the point cloud processing algorithm is computationally complex and consumes a large amount of computing resources, making it difficult to meet the real-time requirements in a high-speed pipeline environment; (2) high-precision laser radar equipment is expensive and sensitive to reflective materials, and is prone to noise interference, affecting detection accuracy and reliability.

[0005] Camera-based 3D vision template matching scheme usually uses RGB-D camera to collect the depth image information of the object, and realizes size recognition and pose estimation by matching the pre-constructed template library. This method has relatively low cost and high initial detection accuracy, but has significant defects: it is highly dependent on the integrity and diversity of the template library, and a large number of templates need to be constructed for different specifications of box packaging, resulting in rising development and maintenance costs; at the same time, when dealing with complex environments such as stacked tilt, strong light reflection label, pattern occlusion, etc., the robustness is poor, and problems such as missed detection and false detection are prone to occur. SUMMARY

[0006] To solve the problems of large amount of calculation, poor real-time performance and weak adaptability in complex industrial scenes of existing box packaging pose calculation methods, the present application proposes a box packaging fast pose calculation method based on lightweight Yolov8-RTMPose model. This method comprehensively uses multi-stage deep learning detection strategy and three-dimensional pose solving algorithm to significantly improve the operation efficiency while ensuring the accuracy, and is suitable for high-speed sorting and other industrial application scenarios.

[0007] The method uses a lightweight improved Yolov8-RTMPose joint model, and the overall detection process uses a top-down multi-stage strategy. First, based on the optimized Yolov8 model, the input image is detected, the ROI (Region of Interest) region of the box packaging object is located, and the BetterCIoU regression loss and Efficient RepGFPN feature fusion structure are introduced to enhance the detection ability and robustness of multi-scale targets. Subsequently, the extracted ROI image is sent to the optimized RTMPose model for key point positioning. The model combines the MPC (Multimodal Perception Convolution) module and the NATTEN (Neighborhood Attention Transformer Encoder) mechanism to improve the fine perception ability of the edge structure and surface key points of the packaging box.

[0008] After completing the 2D key point detection, combined with the depth information obtained from the RGB-D image, the initial three-dimensional pose is estimated by the EPnP algorithm, and then the pose result is refined by the LM nonlinear least squares optimization algorithm, to finally obtain the accurate six-degree-of-freedom pose information of the packaging object.

[0009] Specific content:

[0010] A box packaging fast pose calculation method based on a lightweight Yolov8-RTMPose model, characterized in that the recognition method comprises the following steps:

[0011] Step 1) Obtain RGB and depth images of the box packaging using a 3D depth camera, and perform preprocessing such as image denoising, data alignment, etc., including the following steps:

[0012] Step 1.1) Obtain RGB and depth images of the box packaging using a 3D depth camera, where the 3D camera can be RealSense D435, Azure Kinect, etc., and perform denoising processing on the images using mean filtering, taking pixel (i, j) as the center, selecting a k x k neighborhood, and outputting the final pixel value by the formula:

[0013]

[0014] where I(i, j) is the original image gray value, I`(i, j) is the filtered pixel value, k controls the smoothing degree, and r is the distance of window expansion outward, r = (k-1) / 2, for example, when k = 7, r = 3;

[0015] Step 1.2) Normalize the RGB three-channel pixel values of the image from the range of 0-255 to the range of 0-1, and align the RGB normalized values with the depth data;

[0016] Step 2) Improve the Yolov8 model structure to make it lightweight, as shown in the attached Figure 2 , including the following steps:

[0017] Step 2.1) In the original Yolov8 feature fusion module, introduce Efficient RepGFPN, the specific process is as follows:

[0018] (a) After the input image is extracted by the backbone network, multiple scale candidate feature maps are generated. According to different input sizes, the spatial resolution of the feature map is 1 / 32, 1 / 16, 1 / 8 of the original image, which corresponds to the detection requirements of large, medium and small box targets respectively;

[0019] (b) Introduce the RepConv reparameterization structure. In the training stage, follow the multi-branch parallel convolution, and in the inference stage, fold the multi-branch structure into a single 3x3 convolution through parameter fusion;

[0020] (c) After extraction, use 1x1 convolution for channel mapping and adjust to 256 channels, and send it to the detection head module of Yolov8;

[0021] Step 2.2) On the basis of the original CIoU loss function, two balance factors are added, which are the quotient of the square of the height difference and the square of the height ratio, and the quotient of the square of the width difference and the square of the width ratio, and the improved BetterCIoU loss function is as follows:

[0022]

[0023] where IoU is the intersection over union of the predicted box and the ground truth box, av is a balancing factor, p 2 (b p ,b gt ) is the square of the distance between the two box centers, p 2 (h p ,h gt ) is the square of the height difference, p 2 (w p ,w gt ) is the square of the width difference, c is the ratio of the distance between the centers, c h is the height ratio, and c w is the width ratio.

[0024] Step 2.3) Collect multi-size, multi-angle box packaging image data, and make a box packaging target detection dataset consisting of the following categories: small-size desktop-level fine detection using Desktop Object Dataset, composed of camera acquisition; medium-size box packaging using Stacked Carton Dataset public dataset; large-size box object using Container Dataset, integrating network open source data, containing 300 groups of container images; and labeling, recording box packaging bounding box information and category information, and finally dividing the training set, validation set and test set according to 6:2:2;

[0025] Step 2.4) Use the improved Yolov8 network structure to train on the above box packaging dataset, and use the stochastic gradient descent method for the optimizer, and use the cosine annealing strategy to dynamically adjust the learning rate;

[0026] Step 3) Obtain the box packaging region of interest image through the improved lightweight Yolov8 model, including the following steps:

[0027] Step 3.1) Input the RGB image into the improved Yolov8 model to identify the target of the box packaging, and output the detection result of the box packaging, which includes the confidence and the coordinates of the top-left corner and the bottom-right corner of the box packaging in the RGB image;

[0028] Step 3.2) According to the confidence in the detection result, filter out the low-confidence detection boxes by setting the confidence threshold to 0.65;

[0029] Step 3.3) Extract the region of interest according to the detection box coordinates, and adaptively crop the original RGB image to obtain the region of interest image of the box packaging;

[0030] Step 4) Improve the RTMPose model to make it lightweight, as shown in the attached Figure 4, comprising the following steps:

[0031] Step 4.1) Designing a multi-modal perception convolution module MPC to optimize the 7x7Conv module in RTMPose, the specific process is as follows:

[0032] (a) Decompose the original 7x7 convolution into a three-way heterogeneous convolution flow, a DWConv3x3 high-resolution fine-grained flow, a DilatedConv5x5 medium-range spatial correlation flow, and a global context prior flow composed of a spatial compression excitation module SCSEBlock;

[0033] (b) Pack the multi-scale spatial context of the key point area through a mixed receptive field cover box;

[0034] (c) Use a gating fusion mechanism to dynamically integrate multi-branch features, and embed a motion perception channel attention MCA (Motion-Aware Channel Attention), and the fused features are reorganized across channels through 1x1 convolution;

[0035] Step 4.2) Replace the global attention unit GAU (Global Attention Unit) in RTMPose with the NATTEN module, which uses the neighborhood attention mechanism NAT (Neighborhood Attention) as its core, which limits the self-attention calculation to a kxk local area around each target pixel, the specific process is as follows:

[0036] (a) Through the Overlapping Tokenizer module, the image is cut and tokens are generated;

[0037] (b) Use the layer normalization structure LN (Layer Normalization) to normalize the input image, and also normalize the channel direction, the specific calculation formula is as follows:

[0038]

[0039] Where u and σ are the mean and standard deviation, and γ and β are learnable scaling and translation parameters;

[0040] (c) The neighborhood attention module NA (Neighborhood Attention) uses a local adaptive window attention mechanism, so that each token only pays attention to its local neighborhood;

[0041] (d) The Multilayer Perceptron module MLP (Multilayer Perceptron) is composed of two linear layers and a GELU activation function, which uses a nonlinear mapping to improve the expression ability of the box packaging boundary and detail features;

[0042] Step 4.3) Making a box packaging key point dataset, first defining box packaging surface preset feature points, the labeling rules are as shown in the attached Figure 3 According to the different shooting angles, it is divided into: (1) The angle captures one plane, sets four unoccluded key points and four occluded key points; (2) The angle captures two planes, sets six unoccluded key points and two occluded key points; (3) The angle captures three planes, sets seven unoccluded key points and one occluded key point; Using Labelme labeling tool, manually labeling the pixel coordinates of each key point, and recording the visibility label, 0 for occlusion, 1 for visibility; Finally, the labeled data is uniformly converted into the MMPose structured format supported by RTMPose;

[0043] Step 4.4) Using the optimized RTMPose key point detection network to train on the constructed box packaging key point labeling dataset;

[0044] Step 5) Obtain the box packaging 2D key points through the improved RTMPose model, including the following steps:

[0045] Step 5.1) Input the box packaging region of interest image into the improved RTMPose model, and the output result includes box packaging surface preset feature points and key point detection confidence;

[0046] Step 5.2) Set the key point confidence threshold to 0.7, and only keep the key points with confidence higher than the threshold for subsequent calculation;

[0047] Step 5.3) Use the geometric prior of the box packaging structure to verify the consistency of the detection results and eliminate abnormal points;

[0048] Step 6) Given the 2D image coordinates (u, v) of a pixel point and the depth information D(u, v) of the corresponding point, convert it to three-dimensional coordinates (X, Y, Z) in the depth camera coordinate system to obtain the 3D coordinates of the box packaging unoccluded key points, calculated as follows:

[0049]

[0050] Where f x and f y are the focal lengths in the pixel coordinate system, c x and c y are the positions of the image center;

[0051] Step 7) The 3D key point set is optimized and solved by using the pose estimation algorithm EPnP to obtain the 6-DOF pose parameters of the box packaging in the camera coordinate system, including the following steps:

[0052] Step 7.1) According to the 2D-3D correspondence of the key points in step 6) and the internal parameter matrix, distortion coefficient and initialization pose parameters of the 3D camera, the rotation matrix R and the translation matrix t of the pose parameters are output;

[0053] Step 7.2) Based on the pose parameters obtained in step 7.1), a nonlinear optimization algorithm LM is used to iteratively optimize the re-projection error function to further accurately the rotation matrix R and the translation matrix t, so as to minimize the re-projection error, and the following formula is the optimization process;

[0054]

[0055] Where p i is the 2D image point, P i is the 3D point in the world coordinate system, and K is the internal parameter matrix.

[0056] The present application has the following advantages and beneficial effects:

[0057] The improved Yolov8 and the improved RTMPose are used in the cascade model architecture to realize the cooperative work of rough detection and accurate positioning. In the target detection stage, based on the improved Yolov8 model, the multi-target box packaging recognition problem in the complex scene is effectively solved, and compared with the traditional deep point cloud template matching, the box packaging target recognition time is shortened. Secondly, in the key point positioning stage, the improved RTMPose model is aimed at the box packaging features, and in the box packaging motion process, it still maintains high pose accuracy, and meets the pose calculation speed of industrial demand. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 is the overall flowchart of the box packaging fast pose calculation method based on the lightweight Yolov8-RTMPose model of the present application;

[0059] Figure 2 is the improved Yolov8 model structure diagram;

[0060] Figure 3 is the three situations of the box packaging of the present application when self-occlusion occurs;

[0061] Figure 4 is the improved RTMPose model structure diagram; DETAILED DESCRIPTION

[0062] The advantages and objectives of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the description herein is only for explaining the present invention and is not intended to limit the present invention.

[0063] The present invention discloses a method for rapid pose calculation of box packaging based on a lightweight Yolov8-RTMPose model. This method can achieve efficient pose perception of box packaging objects in complex industrial scenarios while ensuring detection accuracy. The method mainly includes the following steps: first, using a RealSense D435 depth camera to synchronously acquire RGB images and depth images of the box packaging to be identified, and performing preprocessing operations on the acquired images, including image denoising and edge smoothing, to improve data quality and subsequent feature extraction. Subsequently, the processed RGB image is input into the lightweight improved Yolov8 model to quickly identify and locate the target box packaging in the image, and its ROI region in the image is extracted. The ROI region image is passed to the optimized RTMPose model to further realize the two-dimensional coordinate regression of key points on the box packaging surface. The RTMPose model structurally introduces MPC and NATTEN modules to enhance the modeling capability of edge features, thereby improving the robustness and accuracy of key point recognition. After obtaining the 2D image coordinates of the key points, the team combined the known RealSense D435 camera intrinsic parameter model to perform spatial geometric back-projection on the key points, calculating the 3D coordinates of the unobstructed key points in the camera coordinate system. The depth information of the corresponding pixels in the depth image was combined to further restore the precise spatial positions of the key points. Finally, based on the correspondence between the 3D key point coordinates and the 2D projection of the image, the EPnP algorithm was used to estimate the initial pose of the box packaging object in the camera coordinate system. The initial result was further optimized using the LM nonlinear optimization algorithm to obtain the complete six-degree-of-freedom pose parameters of the box packaging in the camera coordinate system.

[0064] Attachment Figure 1 The overall flow chart of the method for rapid pose calculation of box packaging of the present invention includes the following steps:

[0065] A method for fast pose calculation of box packaging based on a lightweight Yolov8-RTMPose model, characterized in that the recognition method comprises the following steps:

[0066] Step 1) Use a 3D depth camera to obtain RGB and depth images of the box packaging, and perform preprocessing such as image denoising and data alignment, including the following steps:

[0067] Step 1.1) Obtain the RGB and depth images of the box packaging using a 3D depth camera, which can be RealSense D435, Azure Kinect, etc., and use mean filtering to denoise the images. Select a k x k neighborhood centered on pixel (i, j) and output the final pixel value using the formula:

[0068]

[0069] where I(i, j) is the original image gray value, I'(i, j) is the filtered pixel value, k controls the smoothing degree, and r is the distance of window expansion, r = (k-1) / 2, for example, when k = 7, r = 3;

[0070] Step 1.2) Normalize the RGB three-channel pixel values of the image from the range of 0-255 to the range of 0-1, and align the RGB normalized values with the depth data;

[0071] Step 2) Improve the Yolov8 model structure to make it lightweight, as shown in the attached Figure 2 , which includes the following steps:

[0072] Step 2.1) In the original Yolov8 feature fusion module, introduce Efficient RepGFPN, the specific process is as follows:

[0073] (a) After the input image is extracted by the backbone network, multiple scale candidate feature maps are generated. According to different input sizes, the spatial resolution of the feature map is 1 / 32, 1 / 16, 1 / 8 of the original image, which corresponds to the detection requirements of large, medium and small box targets respectively;

[0074] (b) Introduce RepConv reparameterization structure. In the training stage, follow the multi-branch parallel convolution, and in the inference stage, fold the multi-branch structure into a single 3x3 convolution through parameter fusion;

[0075] (c) After extraction, use 1x1 convolution for channel mapping and adjust to 256 channels, and send it to the detection head module of Yolov8;

[0076] Step 2.2) On the basis of the original CIoU loss function, add two balance factors, which are the quotient of the square of the height difference and the square of the height ratio, and the quotient of the square of the width difference and the square of the width ratio, and improve it to BetterCIoU loss function, the calculation formula is as follows:

[0077]

[0078] where IoU is the intersection over union of the predicted box and the true box, αv is the balance factor, ρ 2 (b p , bgt ) is the square of the distance between the two box center points, p 2 (h p , h gt ) is the square of the height difference, p 2 (w p , w gt ) is the square of the width difference, c is the center point distance ratio, c h is the height ratio, c w is the width ratio;

[0079] Step 2.3) Collect multi-size, multi-angle box packaging image data, and make box packaging target detection dataset, which consists of the following categories: small size desktop level fine detection uses Desktop Object Dataset, which is collected by camera; medium size box packaging uses Stacked Carton Dataset public dataset; large size box object uses Container Dataset, which integrates network open source data and contains 300 groups of container images; and is labeled to record box packaging frame information and category information, and finally divides the training set, validation set and test set according to 6:2:2;

[0080] Step 2.4) Use the improved Yolov8 network structure to train on the above box packaging dataset, and use the stochastic gradient descent method for the optimizer, and use the cosine annealing strategy to dynamically adjust the learning rate;

[0081] Step 3) Obtain the box packaging region of interest image through the improved lightweight Yolov8 model, including the following steps:

[0082] Step 3.1) Input the RGB image into the improved Yolov8 model to identify the target of the box packaging, and output the detection result of the box packaging, which includes the confidence and the coordinates of the upper left corner and the lower right corner of the box packaging in the RGB image;

[0083] Step 3.2) According to the confidence in the detection result, all detection boxes are screened, and the confidence threshold is set to 0.65, and the low confidence detection boxes are filtered out;

[0084] Step 3.3) According to the detection box coordinates, the region of interest is extracted, and the original RGB image is adaptively sized and cropped to obtain the region of interest image of the box packaging;

[0085] Step 4) Improve the RTMPose model to make it lightweight, as shown in the attached Figure 4 , including the following steps:

[0086] Step 4.1) Design a multi-modal perception convolution module MPC to optimize the 7x7Conv module in RTMPose, the specific process is as follows:

[0087] (d) Decompose the original 7x7 convolution into three heterogeneous convolutional streams, DWConv3x3 high-resolution fine-grained stream, DilatedConv5x5 mid-range spatial correlation stream, and global context prior stream composed of spatial compression excitation block SCSEBlock;

[0088] (e) Mix receptive field coverage to pack multi-scale spatial context of keypoint regions in a box;

[0089] (f) Use a gating fusion mechanism to dynamically integrate multi-branch features, and embed a motion-aware channel attention MCA (Motion-Aware Channel Attention) channel attention, and the fused features are reorganized across channels by 1x1 convolution;

[0090] Step 4.2) Replace the global attention unit GAU (Global Attention Unit) in RTMPose with the NATTEN module, which uses the neighborhood attention mechanism NAT (Neighborhood Attention) as its core, which limits the self-attention calculation to a kxk local area around each target pixel. The specific process is as follows:

[0091] (e) Use the Overlapping Tokenizer module to split the image and generate tokens;

[0092] (f) Use the layer normalization structure LN (Layer Normalization) to normalize the input image, and also normalize the channel direction. The specific calculation formula is as follows:

[0093]

[0094] Where u and σ are the mean and standard deviation, and γ and β are learnable scaling and translation parameters;

[0095] (g) The neighborhood attention module NA (Neighborhood Attention) uses a local adaptive window attention mechanism, so that each token only pays attention to its local neighborhood;

[0096] (h) The multilayer perceptron module MLP (Multilayer Perceptron) is composed of two linear layers and a Gaussian error linear unit GELU activation function, which uses nonlinear mapping to improve the expression ability of box packaging boundary and detail features;

[0097] Step 4.3) Make a box packaging keypoint dataset. First, define the preset feature points on the surface of the box packaging, and the labeling rules are as follows: Figure 3, according to the shooting angle is different into: (1) view angle capture a plane, set four unobstructed key points and four obstructed key points; (2) view angle capture two planes, set six unobstructed key points and two obstructed key points; (3) view angle capture three planes, set seven unobstructed key points and one obstructed key point; Using Labelme marking tool, manually mark the pixel coordinates of each key point, and record the visibility label, 0 for obstructed, 1 for visible; Finally, the labeled data is uniformly converted into RTMPose support MMPose structured format;

[0098] Step 4.4) Using the optimized RTMPose key point detection network, training on the constructed box packaging key point annotation dataset;

[0099] Step 5) Obtain the box packaging 2D key points through the improved RTMPose model, including the following steps:

[0100] Step 5.1) Input the box packaging region of interest image into the improved RTMPose model, and the output result includes the box packaging surface preset feature points and key point detection confidence;

[0101] Step 5.2) Set the key point confidence threshold to 0.7, and only keep the key points with confidence higher than the threshold for subsequent calculation;

[0102] Step 5.3) Use the geometric prior of box packaging structure to verify the consistency of the detection result and eliminate abnormal points;

[0103] Step 6) Given the 2D image coordinates (u, v) of a pixel point and the depth information D(u, v) of the corresponding point, convert it to three-dimensional coordinates (X, Y, Z) in the depth camera coordinate system to obtain the 3D coordinates of the box packaging unobstructed key points, calculated as follows:

[0104]

[0105] Where f x And f y Is the focal length in the pixel coordinate system, c x And c y Is the position of the image center;

[0106] Step 7) Use the pose estimation algorithm EPnP to optimize and solve the 3D key point set to obtain the 6-DOF pose parameters of the box packaging in the camera coordinate system, including the following steps:

[0107] Step 7.1) According to the 2D-3D correspondence of the key points in step 6) and the intrinsic matrix, distortion coefficient and initialization pose parameters of the 3D camera, output the pose parameter rotation matrix R and translation matrix t;

[0108] Step 7.2) Based on the pose parameters obtained in step 7.1), the re-projection error function is iteratively optimized by using a non-linear optimization algorithm LM to further refine the rotation matrix R and translation matrix t to minimize the re-projection error, the following formula is the optimization process;

[0109]

[0110] where p i is the 2D image point, P i is the 3D point in the world coordinate system coordinate, K is the intrinsic matrix.

Claims

1. A fast pose calculation method for box packaging based on the lightweight Yolov8-RTMPose model, characterized in that: The calculation method comprises the following steps: Step 1) Use a 3D camera to acquire RGB and depth images of the box packaging, and perform preprocessing such as image denoising and data alignment, including the following steps: Step 1.1) Use a 3D camera to acquire the RGB image and depth image of the box packaging, and use a mean filter to denoise the image. Centered around pixel (i, j), select a k×k neighborhood and output the final pixel value using the formula: Where I(i,j) is the grayscale value of the original image, I`(i,j) is the pixel value after filtering, k controls the degree of smoothing, and r is the distance the window expands outward; Step 1.2) Normalize the RGB three-channel pixel values ​​of the image from the range of 0-255 to the range of 0-1, and align the RGB normalized values ​​of the pixels with the depth data; Step 2) Improve the Yolov8 model structure to make it lightweight, including the following steps: Step 2.1) Introduce Efficient RepGFPN into the original Yolov8 feature fusion module. The specific process is as follows: (a) After the input image is extracted layer by layer by the backbone network, a multi-scale candidate feature map is generated. Depending on the input size, the spatial resolution of the feature map is 1 / 32, 1 / 16, or 1 / 8 of the original image, corresponding to the detection requirements of large, medium, and small box-shaped objects, respectively. (b) Introducing the re-parameterized structure RepConv, in the training phase, multi-branch parallel convolution is used. In the inference phase, the multi-branch structure is folded into a single 3×3 convolution through parameter fusion; (c) After extraction, 1×1 convolution is used for channel mapping, uniformly adjusted to 256 channels, and sent to the detection head module of Yolov8; Step 2.2) Based on the original CIoU loss function, two balancing factors are added: the quotient of the square of the height difference and the square of the height ratio, and the quotient of the square of the width difference and the square of the width ratio, to improve it into the BetterCIoU loss function, which is calculated as follows: Where IoU is the intersection-over-union ratio between the predicted box and the real box, αv is the balance factor, and ρ 2 (b p ,b gt ) is the square of the distance between the two box centers, ρ 2 (h p ,h gt ) is the square of the height difference, ρ 2 (w p ,w gt ) is the square of the width difference, c is the center point distance ratio, c h is the height ratio, c w is the width ratio; Step 2.3) Collect multi-size and multi-view box packaging image data to create a box packaging object detection dataset. This dataset consists of the following categories: small-sized desktop-level fine detection uses the Desktop Object Dataset, which is collected by camera; medium-sized box packaging uses the Stacked Carton Dataset, a public dataset; and large-sized box objects use the Container Dataset, which integrates open source data from the Internet and contains 300 sets of container images. The dataset is annotated to record the box packaging bounding box information and category information, and finally divided into training, validation, and test sets according to a 6:2:2 ratio. Step 2.4) Use the improved Yolov8 network structure to train on the above box packaging dataset. The optimizer uses the stochastic gradient descent method and the learning rate is dynamically adjusted using the cosine annealing strategy. Step 3) Obtaining an image of the box packaging region of interest using the improved lightweight Yolov8 model, including the following steps: Step 3.1) Input the RGB image into the improved Yolov8 model to perform target recognition on the box packaging and output the detection result of the box packaging, which includes the confidence level and the coordinates of the upper left and lower right corners of the box packaging area in the RGB image; Step 3.2) Based on the confidence level in the detection results, all detection boxes are screened, a confidence threshold of 0.65 is set, and low-confidence detection boxes are filtered out; Step 3.3) Extract the region of interest based on the detection frame coordinates, and perform adaptive size cropping on the original RGB image to obtain the region of interest image of the box packaging; Step 4) Improve the RTMPose model to make it lightweight, including the following steps: Step 4.1) Design a multimodal perception convolution module and optimize the 7x7Conv module in RTMPose. The specific process is as follows: (a) The original 7×7 convolution is decomposed into three heterogeneous convolutional streams: DWConv3×3 high-resolution fine-grained stream, DilatedConv5x5 mid-range spatial correlation stream and global context prior stream, which is composed of the spatial compression excitation module SCSEBlock; (b) Covering the multi-scale spatial context of the box-wrapped keypoint region via mixed receptive fields; (c) A gated fusion mechanism is used to dynamically integrate multi-branch features and embed motion perception channel attention. The fused features are then reorganized across channels using a 1×1 convolution. Step 4.2) Replace the global attention unit in RTMPose with the NATTEN module, which uses the neighborhood attention mechanism at its core and limits the self-attention calculation to a k×k local area around each target pixel. The specific process is as follows: (a) The Overlapping Tokenizer module is used to segment the image and generate tokens. (b) Use the layer normalization structure to normalize the input image and also normalize the channel direction. The specific calculation formula is as follows: Where u and σ are the mean and standard deviation, γ and β are learnable scaling and translation parameters; (c) The neighborhood attention module adopts a local adaptive window attention mechanism, so that each token only focuses on its local neighborhood; (d) The multi-layer perceptron module consists of two linear layers and a Gaussian error linear unit (GELU) activation function, which uses nonlinear mapping to improve the ability to express the boundaries and detailed features of box packaging; Step 4.3) Create a box packaging key point dataset. First, define the preset feature points on the box packaging surface. According to different shooting angles, the dataset is divided into the following types: (1) the view captures one plane, and sets four unobstructed key points and four obstructed key points; (2) the view captures two planes, and sets six unobstructed key points and two obstructed key points; (3) the view captures three planes, and sets seven unobstructed key points and one obstructed key point. Use the Labelme annotation tool to manually annotate the pixel coordinates of each key point and record the visibility label, 0 for obstructed and 1 for visible; finally, convert the annotated data into the MMPose structured format supported by RTMPose. Step 4.4) Use the optimized RTMPose keypoint detection network to train on the constructed box packaging keypoint annotation dataset; Step 5) Obtaining the 2D key points of the box packaging through the improved RTMPose model includes the following steps: Step 5.1) Input the image of the box packaging area of ​​interest into the improved RTMPose model, and the output result includes the preset feature points and key point detection confidence of the box packaging surface; Step 5.2) Set the keypoint confidence threshold to 0.7 and only retain keypoints with confidence higher than the threshold for subsequent calculations; Step 5.3) Use the geometric prior of the box packaging structure to verify the consistency of the detection results and eliminate abnormal points; Step 6) Given the 2D image coordinates (u, v) of the pixel point and the depth information D(u, v) of the corresponding point, convert them to the 3D coordinates (X, Y, Z) in the depth camera coordinate system to obtain the 3D coordinates of the unobstructed key points of the box packaging, calculated as follows; where f x With f y is the focal length in the pixel coordinate system, c x with c y is the position of the image center; Step 7) Using the pose estimation algorithm EPnP, EPnP uses xxx to optimize and solve the 3D key point set to obtain the 6-DOF pose parameters of the box package in the camera coordinate system, including the following steps: Step 7.1) Based on the 2D-3D correspondence of the key points in step 6) and the intrinsic parameter matrix, distortion coefficients and initial pose parameters of the 3D camera, output the pose parameter rotation matrix R and translation matrix t; Step 7.2) Based on the pose parameters obtained in step 7.1), a nonlinear optimization algorithm is used to iteratively optimize the reprojection error function to further refine the rotation matrix R and the translation matrix t to minimize the reprojection error. The following formula describes the optimization process. p i is a 2D image point, P i is the coordinate of the 3D point in the world coordinate system, K is the internal parameter matrix.

Images