Method and device for detecting three-dimensional pose of cargo stack based on two-dimensional image and geometric constraint

Abstract

The application relates to a cargo stack three-dimensional position detection method and device based on a two-dimensional image and geometric constraints, and is applied to the technical fields of computer vision and logistics automation, and comprises the following steps: through an edge straight line screening strategy, in combination with robust fitting of a random sample consensus algorithm, real physical boundaries can be accurately recognized from complex visual interference such as adhesive tapes, labels and printed patterns on the surface of a cargo box, the accuracy and robustness of edge detection are significantly improved, and the high precision of final three-dimensional position calculation is ensured; secondly, only an ordinary industrial camera needs to be used, and expensive three-dimensional sensors such as laser radars or depth cameras are not needed, so that the hardware cost and maintenance complexity of the system are greatly reduced, and the cost efficiency is extremely high; thirdly, efficient image processing and geometric calculation algorithms are mainly used, the calculation complexity is low, and the demand for real-time detection of cargo stacks on an industrial automatic assembly line can be met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and logistics automation, and particularly relates to a method and device for detecting three-dimensional pose of a cargo stack based on two-dimensional images and geometric constraints. BACKGROUND

[0002] In modern automated warehouses, accurate three-dimensional pose detection of cargo stacks on conveying lines is a key link to ensure safe, efficient transfer, stacking and storage of goods. Traditional detection methods, such as manual visual inspection, are inefficient and unreliable, and cannot meet the high-speed automated logistics requirements.

[0003] To achieve automated detection, there are various solutions in the prior art. For example, photoelectric sensors are deployed on both sides of the conveying line to determine whether goods exist or are out of limits. This solution is cost-effective, but can only provide rough, discrete position information and cannot obtain accurate three-dimensional pose of the cargo stack. Another type of solution uses three-dimensional sensors such as laser radars or depth cameras to directly obtain three-dimensional point cloud data of the cargo stack by actively emitting laser or structured light, which can achieve high-precision pose measurement. However, such three-dimensional sensors have high hardware costs, complex system integration and maintenance, and their performance may decrease in certain specific materials (such as transparent or highly reflective surfaces) or complex lighting environments, limiting their large-scale industrial applications.

[0004] To strike a balance between cost and accuracy, some technical solutions attempt to use only ordinary two-dimensional image cameras for detection. Such solutions typically perform edge detection on the cargo box image, determine the intersection of the edge lines to obtain the profile vertices of the cargo box, and finally use perspective projection algorithms and re-projection error optimization to calculate the pose. However, in actual industrial environments, the cargo box surface often has packing tape, printed patterns, labels, and complex background lighting, which can generate a large number of false edges and lines in the image. The prior art does not provide a sufficient solution to effectively handle these visual disturbances, resulting in a serious lack of robustness in filtering edges and vertices, making it difficult to accurately distinguish between real cargo box physical boundaries and false internal lines, ultimately leading to failed or low-precision pose detection. Therefore, how to use only low-cost two-dimensional images to effectively overcome the above visual disturbances and achieve high-precision, high-robustness three-dimensional pose detection of cargo stacks is a technical problem that needs to be solved in the field. SUMMARY

[0005] Therefore, the present application aims to provide a method and device for detecting the three-dimensional pose of a cargo stack based on a two-dimensional image and geometric constraints, so as to solve the technical problem that, when a two-dimensional image is used to detect the pose of a cargo stack, visual interference caused by factors such as the surface tape, label, pattern and complex background lighting of the container makes it difficult to accurately and robustly identify the real boundary of the container, and finally makes it difficult to balance the cost, accuracy, robustness and real-time performance of the detection.

[0006] According to a first aspect of an embodiment of the present application, a method for detecting the three-dimensional pose of a cargo stack based on a two-dimensional image and geometric constraints is provided, and the method comprises the following steps:

[0007] A plurality of cargo stack images are acquired by cameras with different viewing angles, an arbitrary single-view cargo stack image is selected, target detection is performed on the surface of the cargo stack in the single-view cargo stack image, and a ROI candidate region is obtained;

[0008] The ROI candidate region is preprocessed to obtain a ROI target region;

[0009] Edge detection is performed on the ROI target region by using a Canny operator to obtain edge points;

[0010] Hough transform line detection is performed on the ROI target region to obtain candidate straight line segments, and an edge straight line screening strategy is executed to screen out optimal edge line segments representing the boundary of the container from the candidate straight line segments;

[0011] The edge points are assigned to the nearest optimal edge line segment according to the distance between the edge points and each optimal edge line segment to form an edge point set, and a RANSAC algorithm is used to perform line fitting on each set, outliers are removed through iterative optimization, and the boundary straight line corresponding to each edge point set is obtained;

[0012] The intersection points of the boundary straight lines are calculated, each intersection point is screened according to a preset spatial geometric constraint, and the four corner points of the container are determined according to the screening result of the intersection points;

[0013] An internal parameter matrix and distortion coefficients are obtained by calibrating the camera, and a PnP algorithm is used to obtain the 3D pose of the container body based on the internal parameter matrix and the distortion coefficients and the four corner points of the container;

[0014] The 3D pose of the container body under different viewing angles is obtained;

[0015] A unified global coordinate system is established through joint calibration of multiple cameras, the 3D poses of the container body under different viewing angles are projected into the unified global coordinate system, and the optimal three-dimensional pose of the container is obtained according to a preset constraint condition.

[0016] Preferably,

[0017] The preprocessing of the ROI candidate region to obtain an ROI target region comprises:

[0018] The ROI candidate region is subjected to grayscale processing.

[0019] The image subjected to the grayscale processing is subjected to adaptive contrast enhancement using a CLAHE algorithm.

[0020] The grayscale image subjected to the adaptive contrast enhancement is subjected to Gaussian filtering processing.

[0021] The grayscale image subjected to the Gaussian filtering processing is subjected to morphological opening and closing operations.

[0022] The grayscale image subjected to the morphological opening and closing operations is subjected to median filtering processing to obtain the ROI target region.

[0023] Preferably,

[0024] The edge detection of the ROI target region using a Canny operator to obtain edge points comprises:

[0025] The Sobel operator is used to calculate the gradients of the image in the x and y directions to obtain gradient amplitudes and gradient directions.

[0026] The non-maximum suppression of the gradient amplitudes according to the gradient directions is performed to obtain single-pixel edges.

[0027] Two threshold values are set, the edge points greater than the high threshold value in the single-pixel edges are marked as strong edge points, the edge points between the high and low threshold values are marked as weak edge points, the edge points less than the low threshold value are suppressed to 0, and the weak edge points are subjected to connection judgment, if the weak edge points are connected with the strong edge points, the weak edge points are retained, otherwise, the weak edge points are suppressed to 0.

[0028] Preferably,

[0029] The edge straight line screening strategy comprises:

[0030] The candidate straight line segments are classified and grouped into a plurality of edge regions, a comprehensive score is calculated for each candidate straight line segment according to a preset scoring rule, and the straight line segment with the highest comprehensive score in each edge region is selected as the optimal edge line segment.

[0031] Preferably,

[0032] The comprehensive score is obtained by weighted calculation based on the normalized line segment length, position validity and edge region attribution of the candidate straight line segment.

[0033] The position validity is determined according to whether the distance between the two end points of the candidate straight line segment and the image boundary is greater than a preset boundary safety distance, if both are greater than the preset boundary safety distance, the position validity is assigned a value of 1, otherwise, the position validity is assigned a value of 0.

[0034] Preferably,

[0035] The edge region is a preset width region of four edges of the image.

[0036] Preferably,

[0037] The filtering of the intersection points according to the preset spatial geometric constraint includes:

[0038] The intersection point is located in the ROI region, and the included angle between the two intersecting straight lines constituting the intersection point is within a preset right angle tolerance range, and the distance from the intersection point to the straight line is within a preset distance threshold range;

[0039] The intersection point satisfying the condition is taken as an effective corner point.

[0040] Preferably,

[0041] The obtaining of the 3D pose of the box body through the four corner points of the box by using the PnP algorithm includes:

[0042] The PnP algorithm based on the Levenberg-Marquardt optimization method is used to solve the 3D pose of the box body surface by iteratively minimizing the re-projection error between the coordinates of the three-dimensional model corner points corresponding to the three-dimensional size of the box under the camera model and projected to the image plane and the coordinates of the corner points of the box in the two-dimensional image.

[0043] Preferably,

[0044] The obtaining of the optimal three-dimensional pose of the box according to the preset constraint condition includes:

[0045] The epipolar constraint is performed on the corresponding edge points detected across views, so that the corresponding points in different views are located on conjugate epipolar lines, and an epipolar constraint error is obtained.

[0046] The epipolar constraint error and the re-projection error obtained by the PnP algorithm are fused to obtain a combined cost function, and the final pose parameter is obtained by minimizing the combined cost function, and the optimal three-dimensional pose of the box is obtained through the final pose parameter.

[0047] According to a second aspect of an embodiment of the present application, a device for detecting the three-dimensional pose of a box pile based on a two-dimensional image and geometric constraints is provided, and the device includes:

[0048] The ROI candidate region acquisition module is configured to acquire multiple pallet images through cameras with different perspectives, select an arbitrary single-perspective pallet image, perform target detection on a pallet surface in the single-perspective pallet image, and obtain an ROI candidate region.

[0049] The preprocessing module is configured to pre-process the ROI candidate region and obtain an ROI target region.

[0050] The edge detection module is configured to perform edge detection on the ROI target region by using a Canny operator to obtain edge points.

[0051] The optimal edge line segment acquisition module is configured to perform Hough transform straight line detection on the ROI target region to obtain candidate straight line segments, and perform edge straight line screening strategy to screen out optimal edge line segments representing the boundaries of the container from the candidate straight line segments.

[0052] The boundary straight line acquisition module is configured to assign the edge points to the nearest optimal edge line segment according to the distance between the edge points and each optimal edge line segment to form an edge point set, and perform straight line fitting on each set by using a RANSAC algorithm, remove outliers through iterative optimization, and obtain a boundary straight line corresponding to each edge point set.

[0053] The corner point acquisition module is configured to calculate the intersection points of the boundary straight lines, screen the intersection points according to a preset spatial geometric constraint, and determine four corner points of the container through the screening result of the intersection points.

[0054] The 3D pose acquisition module is configured to calibrate the camera to obtain an intrinsic matrix and distortion coefficients, and obtain a 3D pose of the container by using a PnP algorithm through the four corner points of the container based on the intrinsic matrix and the distortion coefficients.

[0055] The multi-perspective pose acquisition module is configured to obtain the 3D pose of the container under different perspectives.

[0056] The multi-perspective fusion module is configured to establish a unified global coordinate system through joint calibration of multiple cameras, project the 3D pose of the container under different perspectives into the unified global coordinate system, and obtain an optimal three-dimensional pose of the container according to a preset constraint condition.

[0057] The technical scheme provided by the embodiments of the present application can include the following beneficial effects:

[0058] The application can accurately identify the real physical boundary from the complex visual interference such as adhesive tape, label and printed pattern on the surface of the box through the designed "partition competition" edge straight line screening strategy combined with the robust fitting of the random sampling consensus algorithm, significantly improves the accuracy and robustness of edge detection, and ensures the high precision of the final three-dimensional pose solution. Secondly, the application only needs to use an ordinary industrial camera, without expensive three-dimensional sensors such as laser radar or depth camera, which greatly reduces the hardware cost and maintenance complexity of the system, and has very high cost effectiveness. Thirdly, the application mainly uses efficient image processing and geometric calculation algorithm, which has low computational complexity and can meet the real-time detection requirements of the goods pile on the industrial automatic assembly line. Finally, through the multi-level image preprocessing process, the problem of uneven light in the actual scene is effectively solved, and the optional multi-view integrated framework can handle the occlusion problem under single view, further enhancing the adaptability and detection reliability of the system under complex working conditions.

[0059] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0060] The accompanying drawings, which are incorporated into and form part of the specification, illustrate an embodiment consistent with the application and, together with the specification, serve to explain the principles of the application.

[0061] Figure 1 is a flowchart of a three-dimensional pose detection method for goods pile based on two-dimensional images and geometric constraints according to an exemplary embodiment;

[0062] Figure 2 is a detailed flowchart of a box edge detection method according to another exemplary embodiment;

[0063] Figure 3 is an image preprocessing flowchart according to another exemplary embodiment;

[0064] Figure 4 is an edge straight line screening and scoring flowchart according to another exemplary embodiment;

[0065] Figure 5 is a multi-view integration method flowchart according to another exemplary embodiment;

[0066] Figure 6 is a schematic diagram of the system hardware implementation environment according to another exemplary embodiment;

[0067] In the drawings: 1-conveyor belt, 2-goods pile, 3-gantry, 4-camera. DETAILED DESCRIPTION

[0068] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description below refers to the accompanying drawings, which show, by way of example, specific embodiments with which this application can be practiced. The following detailed description is not intended to limit the application. Rather, the following detailed description includes specific details for the purpose of providing a thorough understanding of the inventive concepts. However, it will be apparent to those skilled in the art that the application can be practiced without these specific details. In some instances, well-known structures and functions have not been described in detail in order to avoid obscuring the application. The following examples are illustrative only and are not intended to limit the scope of the application in any way.

[0069] Embodiment One

[0070] Figure 1 is a flowchart of a method for detecting the 3D pose of a pallet based on a two-dimensional image and geometric constraints according to an exemplary embodiment, as shown in Figure 1 , the method comprises:

[0071] S1, obtaining multiple pallet images through cameras with different viewing angles, selecting an arbitrary single-view pallet image, performing target detection on the pallet surface in the single-view pallet image to obtain a ROI candidate region;

[0072] S2, pre-processing the ROI candidate region to obtain a ROI target region;

[0073] S3, performing edge detection on the ROI target region using a Canny operator to obtain edge points;

[0074] S4, performing straight line detection on the ROI target region using a Hough transform to obtain candidate straight line segments; performing edge straight line screening strategy to screen out optimal edge line segments representing the boundaries of the container from the candidate straight line segments;

[0075] S5, assigning the edge points to the nearest optimal edge line segment according to their distance from each optimal edge line segment to form an edge point set, and performing straight line fitting on each set using a RANSAC algorithm, and removing outliers through iterative optimization to obtain the boundary straight line corresponding to each edge point set;

[0076] S6, calculating the intersection points of each boundary straight line, and screening each intersection point according to a predetermined spatial geometric constraint, and determining the four corner points of the container through the screening results of the intersection points;

[0077] S7, calibrating the camera to obtain an intrinsic matrix and distortion coefficients; based on the intrinsic matrix and distortion coefficients, using a PnP algorithm to obtain the 3D pose of the container through the four corner points of the container;

[0078] S8, obtaining the 3D pose of the container under different viewing angles;

[0079] S9, by establishing a unified global coordinate system through multi-camera joint calibration, projecting the 3D pose of the container under different perspectives into the unified global coordinate system, and obtaining the optimal 3D pose of the container according to a preset constraint condition;

[0080] It can be understood that the embodiment provides a basic scheme for detecting the 3D pose of a container based on a two-dimensional image collected by a single camera. It should be noted that the scheme is intended to fully embody the complete technical process of image collection, preprocessing, core edge detection and screening, and final pose solving provided by the present application.

[0081] Please refer to Figure 6 which shows a system hardware implementation environment to which the embodiments of the present application can be applied. In a typical automated warehouse scenario, the system can include a conveyor belt 1 for transporting a pallet 2. A gantry 3 can be erected above the conveyor belt 1, and one or more industrial cameras 4 are fixedly installed on the gantry 3. The lens of the camera 4 is perpendicular or at a certain angle to the downward direction to ensure that when the pallet 2 passes through, a clear and complete image of the top or side of the pallet can be captured. In the present embodiment, it is assumed that one camera 4 is used, which is installed at a position and angle that can capture the main visible surface of the target container.

[0082] When a pallet carrying a pallet 2 enters the field of view of the camera 4 and triggers a shooting signal, the system starts the detection process.

[0083] First, the system performs image acquisition. The camera 4 captures a two-dimensional color image containing the pallet 2, for example, an image with a resolution of 1920x1080 pixels.

[0084] As an optional implementation, in some application scenarios, for example, when the image contains multiple containers or the background is complex, region of interest target detection can be performed first. This step can use a lightweight deep learning target detection model (such as the YOLO series) or a traditional image segmentation method to quickly locate the approximate bounding box of each container to be detected in the entire image. This step helps to focus the subsequent processing process on the effective area containing a single container, thereby reducing the interference of irrelevant background and improving processing efficiency. For the description of the subsequent steps, it is assumed that an image region containing only a single target container has been obtained.

[0085] Correspondingly, the process enters the core container edge detection. The goal of this step is to accurately extract the geometric information representing the physical boundary of the container from the container image that may be full of visual interference. Please refer to Figure 2 This step is further divided into a series of sub-steps.

[0086] The first step is image preprocessing S31: due to the variable lighting conditions in the industrial field, and the presence of noise such as stains and reflections on the surface of the box, it is essential to preprocess the original image. Please refer to the following description in combination with Figure 3 In an embodiment of the present application, the preprocessing S31 process includes the following consecutive operations: first, perform grayscale processing (step S311) to convert the three-channel color image captured by the camera into a single-channel grayscale image. This is intended to reduce the data dimension and reduce the computational complexity of subsequent image processing, while retaining the brightness information required for edge detection, which specifically includes:

[0087] The color image contains color information of three channels of RGB. In the box corner detection task, the focus is on the structural features of the image rather than the color information, and it is necessary to convert the RGB image to a grayscale image. The conversion from RGB to grayscale uses the weighted average method:

[0088]

[0089] where R, G, and B represent the red, green, and blue channel values of the pixel point, respectively, and the weight coefficient is determined by the sensitivity of the human eye to different colors. Grayscale conversion converts three-channel images to a single channel, reducing the data volume while retaining the main structural information of the image.

[0090] Subsequently, to enhance the image contrast (step S312), an adaptive histogram equalization algorithm with limited contrast can be used to process the grayscale image. It can be understood that, unlike global histogram equalization, this algorithm can effectively enhance the local contrast of the image under different lighting conditions by dividing the image into several sub-regions and performing histogram equalization on each sub-region independently, so that the edges of the box can be clearly highlighted even under uneven lighting conditions. To avoid excessive amplification of noise, the algorithm can also set an upper limit for the contrast gain of each sub-region, which specifically includes:

[0091] To enhance the local contrast of the image, the CLAHE algorithm is used to process the grayscale image. The histogram equalization of each local region in the image can be expressed as:

[0092]

[0093] where, is the number of pixels at the gray level , is the total number of pixels in the region, is the number of gray levels, and are the target gray range.

[0094] Next, Gaussian filtering is performed (step S313), for example, a 5x5 size Gaussian kernel is used to perform convolution operation on the enhanced image. As a linear smoothing filter, Gaussian filtering can effectively suppress Gaussian noise in the image that meets the normal distribution, thereby making the image smoother and laying the foundation for subsequent edge detection, specifically including:

[0095] In order to suppress the noise in the image, the gray-scale image is subjected to Gaussian filtering processing. The Gaussian filter is a linear smoothing filter, and its weight coefficient satisfies the two-dimensional Gaussian distribution:

[0096]

[0097] Where (x, y) represents the position of the pixel point relative to the kernel center, is the standard deviation of the Gaussian distribution. According to the Gaussian distribution, the filter gives a higher weight to the center pixel and gradually reduces the weight of the surrounding pixels, forming a balanced smoothing effect. The discrete Gaussian kernel is used to perform convolution operation on the image:

[0098]

[0099] Wherein, is the pixel value of the filtered image at position is the pixel value of the original image at the corresponding position, is the Gaussian kernel coefficient. During the processing of the image, the convolution operation can preserve important structural features while suppressing high-frequency noise.

[0100] In addition, in order to eliminate the possible small bright spots or dark spots (such as noise caused by dust or small reflections) in the image and fill the possible small cracks in the edge, the embodiment can also sequentially perform morphological opening operation and closing operation (step S314). Specifically, the opening operation can disconnect small connections and eliminate isolated bright spots, while the closing operation can fill small holes and connect adjacent regions. For example, a 3x3 structure element can be used to perform the operation, specifically including:

[0101] In order to further optimize the image quality, morphological opening operation and closing operation are used, and the structure element is , the original image is , and the calculation formula of the opening operation and the closing operation is as follows:

[0102]

[0103] Wherein, and represent erosion and dilation operations respectively. The opening operation can remove small bright details while maintaining the overall shape and size of larger objects; the closing operation can fill small holes and connect adjacent objects. ​

[0104] As the last step of preprocessing, the image can be further processed by median filtering (step S315), for example using a 3x3 size median filter. Median filtering is a nonlinear filtering technique that can effectively remove salt and pepper noise by sorting the pixel values in the neighborhood of a pixel and replacing the center pixel value with the median value, and can better preserve the edge definition compared to Gaussian filtering. Specifically, it includes:

[0105] Finally, median filtering is used to further suppress noise while preserving edge features. For a median filter with a window size of :

[0106]

[0107] As a nonlinear technique, median filtering replaces each pixel with the median value of its neighborhood, which can effectively remove salt and pepper noise and better preserve edge integrity compared to linear filters.

[0108] After the above series of preprocessing S31, a gray-scale image with suppressed noise and enhanced edges can be obtained, laying a solid foundation for subsequent edge detection S32.

[0109] In step S32, the pre-processed image is first edge detected using the Canny operator. As a multi-stage edge detection algorithm, the Canny operator can detect relatively accurate and continuous edge points by calculating image gradients, performing non-maximum suppression to refine edges, and using double thresholds and hysteresis tracking to connect edges, thereby forming a binary edge map. Specifically, it includes:

[0110] First, the Sobel operator is used to calculate the gradients of the image in the x and y directions, respectively. The mathematical expression of the Sobel operator is:

[0111]

[0112] wherein, is the input image, denotes convolution operation;

[0113] Then, the gradient amplitude and direction are calculated:

[0114]

[0115] Non-maximum suppression:

[0116] In order to obtain a single-pixel edge , non-maximum suppression is performed on the gradient amplitude . First, the gradient direction The gradient is quantized into four directions: 0°, 45°, 90°, and 135°. Then, interpolation is performed on each pixel along the gradient direction. If the gradient magnitude at a point is not a local maximum, it is suppressed to 0. Non-maximum suppression is expressed as:

[0117]

[0118] Dual threshold detection and edge connectivity:

[0119] The dual-threshold algorithm filters and connects edges by setting two thresholds, high and low. and ( ), will be greater than Edge points are marked as strong edge points; those between and Edge points between them are marked as weak edge points; those smaller than Edge points are suppressed to 0. Weak edge points are judged for connection. If a weak edge point is connected to a strong edge point, it is retained; otherwise, it is suppressed to 0.

[0120] Subsequently, to extract structured line information from discrete edge points, this embodiment employs a probabilistic Hough transform to detect all candidate line segments on the Canny edge map. Compared to the standard Hough transform, the probabilistic Hough transform does not accumulate all points but only analyzes a random subset of the edge point set and estimates the probability that these points constitute a line, thus achieving higher computational efficiency. The output of this transform is a set of line segments, each defined by its start and end coordinates. Parameters such as the minimum line segment length (e.g., 50 pixels) and the maximum allowed break gap can be set to filter out excessively short or discontinuous line segments. Specifically, this includes:

[0121] To improve computational efficiency, the probabilistic Hough transform (PHT) is used to map the straight line from the image space to the parameter space. In polar coordinates, any straight line can be represented as:

[0122]

[0123] in, This represents the perpendicular distance from the line to the origin of the coordinate system. Indicates perpendicular line and The included angle of the axis, To represent any point on a straight line, the main parameters of PHT include: Resolution (pixels) Resolution (radians), minimum voting threshold Minimum line segment length and the maximum line segment gap The line segment detection function for PHT is:

[0124]

[0125] wherein, is the vote accumulation value in the parameter space, is the detected line segment length, is the gap distance between line segments, due to the influence of background and box surface texture, PHT realizes robust line detection and identification of a large number of candidate line segments through multiple parameter constraints.

[0126] In order to accurately identify the straight lines representing the four physical boundaries of the container from the candidate straight line segments, the application proposes a set of core edge filtering and classification strategies, i.e. step S33, please refer to Figure 4 The detailed process of the strategy is as follows:

[0127] Step S331, line classification and partitioning, this step aims to structurally organize all candidate line segments, first, classify them according to their angles, for example, classify the lines into two categories of horizontal lines and vertical lines through an angle threshold of 45°, then, according to the position of the line segment in the image, it is classified into a pre-defined edge area, in this embodiment, the image can be divided into four edge areas of top, bottom, left and right along the horizontal and vertical directions. For example, the area of the top 15% height of the image is defined as the "top edge area", the area of the bottom 15% height of the image is defined as the "bottom edge area", the area of the left 15% width of the image is defined as the "left edge area", and the area of the right 15% width of the image is defined as the "right edge area", in this way, a candidate line segment classified as a "horizontal line" will be classified into the "top edge area" if most of it is located in the upper part of the image, which specifically includes:

[0128] First, determine the pixel range of the edge area through the edge threshold calculation formula:

[0129]

[0130] wherein The default setting is 0.15, i.e. 15% of the minimum side length of the image as the edge area width, for each detected line, the system classifies it into the corresponding edge area according to its geometric characteristics and position information, the horizontal line segment is determined to belong to the top edge area or the bottom edge area through its longitudinal coordinate range, the vertical line segment is determined to belong to the left edge area or the right edge area through its transverse coordinate range, the judgment criterion is whether the line segment endpoint coordinates satisfy the corresponding edge threshold condition.

[0131] Step S332, calculate the comprehensive score. In this step, a comprehensive score will be calculated for each candidate line segment to quantify its possibility as a real container boundary. The score can be calculated by weighting multiple dimensional features, which specifically includes:

[0132] The comprehensive score of each straight line is calculated to select the optimal edge, and the straight line score formula is as shown below:

[0133]

[0134] wherein, denotes the normalized line segment length, and the calculation method is wherein is the line segment length, and are the image height and width, respectively; denotes the position validity score, which is 1.0 when both ends of the line segment are far away from the image boundary, and 0.0 otherwise; denotes the edge region score, which is 1.0 when the line segment is located in the edge region, and 0.0 otherwise; , and are the length weight, position weight and convenience weight, respectively, and are set to , , ;

[0135] The position validity judgment adopts the boundary distance criterion, and requires that the line segment endpoint coordinates satisfy:

[0136]

[0137] wherein margin is the boundary safety distance, and is set to 5 pixels;

[0138] Finally, the system selects the straight line with the highest comprehensive score from the straight line set of the upper, lower, left and right edge regions as the optimal edge line segment of the box, effectively excluding the interference lines inside the image (such as the horizontal packing adhesive tape), and ensuring the accuracy and stability of the detection result.

[0139] Step S333, selecting the optimal straight line in each region. After completing the comprehensive score calculation of all candidate straight line segments, the system executes a "partition competition" selection mechanism. Specifically, among all the candidate straight lines in the "upper edge region", the straight line with the highest comprehensive score is selected as the optimal edge line segment of the region. Similarly, the straight line with the highest score is also selected in the "lower edge region", "left edge region" and "right edge region", respectively. In this way, finally, the four optimal edge line segments that best represent the upper, lower, left and right four physical boundaries of the box can be selected from hundreds of candidate straight lines.

[0140] Return to Figure 2After obtaining four optimal edge segments, the process proceeds to step S34 for more accurate line fitting. Although the optimal edge segments are very close to the true boundary, they are still approximations based on the Hough transform. To obtain a more accurate line equation, this embodiment uses all edge points previously detected by the Canny operator for robust fitting. First, all edge points are assigned to the nearest line segment based on their perpendicular distance to the four optimal edge segments, thus forming four sets of edge points. Theoretically, each set of points corresponds to a boundary of the cargo box, but it may still contain a small number of noise points (i.e., outliers) that do not belong to that boundary. Specifically, these include:

[0141] Given an edge image and image size Calculate each edge point The distance to the optimal edge line is used for filtering, and a distance threshold is applied:

[0142]

[0143] in, Represents the parameters of a line. This represents the distance from a point to a line. When the distance is below a threshold, each edge point is assigned to the nearest line. The edge points are successfully classified into four different box boundaries (top, bottom, left, and right edges), effectively separating the real box edges from the noise generated by the label or background texture.

[0144] To eliminate the influence of outliers, this embodiment employs a random sampling consensus algorithm to fit a straight line to each edge point set. The random sampling consensus algorithm is an iterative process: in each iteration, two points are randomly selected from the point set to fit a temporary straight line; then, the distances from all other points in the point set to this temporary straight line are calculated, and points with distances less than a preset threshold are defined as "inner group points"; this process is repeated multiple times, and finally, the straight line corresponding to the model with the most inner group points is selected as the final fitting result for that point set. By executing this algorithm on each of the four edge point sets, four straight line equations that accurately describe the cargo box boundary are obtained, specifically including:

[0145] Based on the edge point clustering results, the RANSAC (Random Sample Consensus) algorithm is used to fit a straight line to each set. Outliers are removed through iterative optimization to obtain accurate boundary lines and edge point sets. A straight line can be represented as:

[0146]

[0147] Where k is the slope of the line and b is the intercept of the line. The point coordinates in the point set. The RANSAC algorithm finds the optimal fitting result by iterative optimization, first estimates the model for two randomly selected points, calculates the straight line parameters, then counts the number of inliers and evaluates the model, and finally filters the inliers by judging whether the perpendicular distance of the point to the straight line is less than the set threshold. The algorithm selects the optimal model through multiple iterations and retains the inlier set.

[0148] Subsequently, in step S35, corner point detection and sorting are performed. By simultaneously solving the equations of the four boundary straight lines obtained in the previous step, the intersection points between them can be calculated, thereby obtaining a set of candidate corner points. In theory, there are six intersection points of the four straight lines, but due to the perspective effect, this embodiment is concerned with the four corner points that constitute the visible top surface of the container. To filter out the correct corner points from the candidate corner points, this embodiment imposes spatial geometric constraints. A key constraint is the angle constraint: since the container is a cuboid, its adjacent boundary lines are perpendicular to each other in three-dimensional space, and after projection onto the two-dimensional image, their included angle should also be close to 90 degrees. Therefore, those corner points formed by the intersection of two straight lines with an included angle within a predetermined right angle tolerance range (e.g., 80° to 100°) can be filtered out. In addition, the region of interest information obtained in the above can be combined to eliminate corner points located outside the region. Through these filters, the four corner point coordinates of the container on the two-dimensional image are finally determined, and they can be sorted according to their positional relationship (e.g., clockwise or counterclockwise). Specifically, it includes:

[0149] To obtain the accurate corner point coordinates of the upper surface of the container, based on the straight line fitting result, an angle point detection method based on straight line intersection calculation and clustering is proposed, and the corner points are screened and sorted through spatial geometric constraints. For two straight lines and , the intersection point coordinates can be obtained by solving the linear equation system:

[0150]

[0151] The intersection point coordinates are obtained:

[0152]

[0153] An effective corner point is defined as a point that satisfies all the following conditions :

[0154] (1) The point is located within the ROI region: ;

[0155] (2) The angle between the two intersecting lines and forming the corner point is close to a right angle: ;

[0156] (3) the distance from the corner point to the line is within a threshold range: ;

[0157] where the angle tolerance is The empirical value is set to 10° to accommodate slight deviations from perfect perpendicularity in real images, and the distance threshold is and Used to filter out false intersection points while retaining valid corner points, the specific values are determined through experimental calibration for the target box type.

[0158] At this point, the box edge detection step S30 is completed, and then the pose solving is performed, which aims to solve the relative pose between the camera and the object according to the known two-dimensional image points and the corresponding three-dimensional space points, which is called the PnP problem in the field of computer vision.

[0159] In this embodiment, the inputs of the pose solving include:

[0160] The camera intrinsic matrix K and distortion coefficients d calibrated in advance;

[0161] The known three-dimensional model of the box, specifically, the three-dimensional coordinates of the four vertices of the box in its own model coordinate system, for example, a coordinate system with a certain corner point of the box as the origin can be defined, and the four vertex coordinates are (0, 0, 0), (L, 0, 0), (L, W, 0), and (0, W, 0), where L and W are the length and width of the box;

[0162] This embodiment uses an iterative PnP algorithm based on the Levenberg-Marquardt optimization method to solve the pose, the goal of this algorithm is to find a rotation matrix R and a translation vector T (which together constitute the transformation from the model coordinate system to the camera coordinate system), so that the sum of the distances (i.e. the re-projection error) between the two-dimensional coordinates of the three-dimensional model corners after the (R, T) transformation and the actual observed two-dimensional corner points coordinates, and the two-dimensional coordinates of the corners projected onto the image plane by the camera intrinsic K, is minimized. The Levenberg-Marquardt algorithm is a highly efficient nonlinear optimization algorithm that combines the advantages of gradient descent and Gauss-Newton methods, and can quickly and stably converge to the optimal solution. The output of the algorithm is the rotation matrix R and the translation vector T of the box relative to the camera coordinate system, and based on the rotation matrix R and the translation vector T of the camera coordinate system, the three-dimensional coordinates of the four corners are solved, that is, the 3D pose of the box.

[0163] Through the complete flow of the embodiment, even in the case of complex visual interference such as reflective tape and printed text on the surface of the box, the system can accurately calculate the three-dimensional pose thereof in the camera coordinate system, experiments show that the position error can be controlled within 5mm, and the attitude error can be less than 1°, fully demonstrating the robustness and high precision of the method.

[0164] It needs to be emphasized that before performing the PnP algorithm, the camera 4 needs to be calibrated, and the calibration process can be completed by using a standard checkerboard calibration board. Specifically, by shooting images of the calibration board at different positions and attitudes, and using mature calibration algorithms such as Zhang Zhengyou calibration method, the internal parameter matrix K and the distortion coefficient d of the camera 4 can be accurately calculated. It can be understood that the internal parameter matrix K describes the projection relationship of the three-dimensional point in the camera coordinate system to the two-dimensional image plane pixel coordinates, and the distortion coefficient d is used to correct the radial and tangential distortion of the image caused by the physical characteristics of the lens. These parameters are the basis for subsequent accurate three-dimensional pose solving, and are stored in the system for calling after calibration. At the same time, the three-dimensional physical dimensions (for example, length 400mm, width 300mm, height 200mm) of the box to be detected are input into the system as prior knowledge; the PnP specific algorithm includes:

[0165] The specific solving process is as follows:

[0166] First stage: camera calibration

[0167] Input: n checkerboard images (n=9)

[0168] Output: internal parameter matrix K, distortion coefficient d

[0169] Second stage: box corner PnP

[0170] Input: 4 corner image coordinates, box size LxW, camera parameters K, d

[0171] Output: camera extrinsic parameters R, t

[0172] Third stage: three-dimensional coordinate calculation

[0173] Input: corner world coordinates, extrinsic parameters R, t

[0174] Output: corner three-dimensional coordinates (camera coordinate system)

[0175] First stage:

[0176] Known:

[0177] The box picture taken by the camera A;

[0178] The two-dimensional image coordinates of the four corner points on the surface of the box (rectangle)

[0179] Actual dimensions of the box surface: length L, width W

[0180] Objective: 3D world coordinates of the four corner points

[0181] Step 1: Solve K and d using chessboard calibration

[0182] Chessboard calibration data collection

[0183] Use a standard chessboard (e.g., 9x6 grid) to take n images (n≥3) from multiple angles, and detect m corner points in each image.

[0184] Chessboard corner world coordinate setting:

[0185] (i,j is the grid index)

[0186] is the size of each individual grid in the chessboard

[0187] Homography matrix estimation (solve )

[0188]

[0189] For each chessboard image k, establish a planar homography transformation:

[0190]

[0191] Solve each : Establish 2m linear equations for m corner points to solve 9 parameters.

[0192] Intrinsic matrix K solution

[0193] Use the orthogonality constraint of the rotation matrix:

[0194] (orthogonal constraint)

[0195] (modulus constraint)

[0196] Let , each image provides 2 constraint equations, and n images provide 2n equations.

[0197] Solve b by SVD decomposition, then calculate the intrinsic parameters:

[0198]

[0199]

[0200]

[0201]

[0202]

[0203] Intrinsic matrix:

[0204] Initial extrinsic estimation

[0205] For each image From Compute initial extrinsics:

[0206]

[0207]

[0208]

[0209]

[0210]

[0211] Where

[0212] Distortion coefficients d solving

[0213] Distortion model (horizontal + vertical):

[0214]

[0215]

[0216] Construct total reprojection error:

[0217]

[0218] Optimize all parameters by Levenberg-Marquardt algorithm:

[0219] Intrinsic parameters:

[0220] Distortion coefficients:

[0221] Extrinsic parameters:

[0222] .

[0223] Second stage: Camera extrinsic R, t solving

[0224] Establish the world coordinate system of the box

[0225] Set the world coordinates of the four corner points of the bin surface:

[0226]

[0227] PnP problem setup

[0228] Given:

[0229] Calibration obtained camera parameters: K, d

[0230] World coordinates of the four corner points:

[0231] Corresponding distorted image coordinates:

[0232] Solve: camera extrinsic parameters

[0233] Where the projection equation is:

[0234]

[0235] PnP algorithm solution

[0236] Control point selection

[0237]

[0238] Barycentric coordinate calculation

[0239] Denote as a linear combination of other control points:

[0240]

[0241] Where can be solved by linear equations.

[0242] Linear constraint construction

[0243] Establish a constraint equation for each observation point:

[0244]

[0245] Construct the linear equation set Mx = 0, where x contains 12 control point coordinate components.

[0246] Distance constraint solution

[0247] Utilize the invariance of distance between control points

[0248] (World coordinate system = camera coordinate system)

[0249] Solve the linear equations for the scale parameters to find the control point coordinates in the camera coordinate system.

[0250] Extrinsic parameter computation

[0251] Compute R and t by the absolute orientation method:

[0252] Centroid computation:

[0253]

[0254]

[0255] Covariance matrix:

[0256]

[0257] SVD decomposition , rotation matrix:

[0258]

[0259] Translation vector:

[0260]

[0261] Third stage: 3D corner point computation

[0262] Camera coordinate system coordinates

[0263] Coordinates of the four corners in the camera coordinate system:

[0264]

[0265]

[0266]

[0267] World coordinate system coordinates

[0268] Coordinates relative to a world coordinate system, by inverse transformation:

[0269]

[0270] Re-projection verification

[0271] Re-project the computed 3D coordinates to the image:

[0272] Camera coordinate projection:

[0273]

[0274]

[0275]

[0276] Distortion correction:

[0277]

[0278]

[0279]

[0280]

[0281]

[0282] Re-projection error (only computed):

[0283]

[0284] Multi-view integration:

[0285] See Figure 5 for an overview of the multi-view integration method, which first requires joint calibration of camera A and camera B (step S73) during the system deployment phase, which includes:

[0286] To extend the single-view method to the multi-camera scenario, a multi-view integration framework is proposed, which uses the single-view processing pipeline to independently process each view based on the N cameras observing the goods pile, and integrates cross-view information through a unified optimization framework.

[0287] First, a global coordinate system is established through a unified camera calibration process:

[0288]

[0289] where represents a point in the world coordinate system; represents the camera coordinates of view and and are the rotation and translation parameters of each camera, respectively, and a rigid body transformation is applied to establish a common reference coordinate system for all cameras to integrate information from multiple views.

[0290] When the goods pile 2 enters the overlapping field of view, both cameras simultaneously or sequentially capture images in step S71;

[0291] The system performs single-view independent processing in step S72;

[0292] Based on this, the system performs cross-view fusion and optimization in step S74. This step utilizes the geometric consistency between multiple views to solve for or optimize the pose. Specifically, it can be done in the following way:

[0293] Using the preliminary pose calculated by camera B, we obtain the estimated coordinates of the four three-dimensional corner points of the cargo box in the global coordinate system.

[0294] For a corner point that can be observed by both camera A and camera B, find its two-dimensional coordinates in the image of camera A and its corresponding two-dimensional coordinates in the image of camera B.

[0295] Construct a combinatorial optimization objective function that aims to find the globally optimal cargo container pose by simultaneously minimizing multiple error terms, specifically including:

[0296] Perform cross-view detection fusion, and for each detected cargo surface, project the detection results from multiple views onto a unified coordinate system:

[0297]

[0298] in It's a camera. The projection matrix, containing the view 2D point coordinates detected The Direct Linear Transformation (DLT) algorithm is used to efficiently solve for uniform coordinates. Treating this as a least-squares problem, the algorithm employs triangulation to find the optimal 3D point that minimizes reprojection errors across all views.

[0299] Improve PnP-based pose estimation through multi-view constraints. For corresponding edge points detected across views. and Perform polar constraints:

[0300]

[0301] in It is a view and The fundamental matrix between them ensures that corresponding points in different views lie on the conjugate epipolar line, providing a geometric verification mechanism for feature matching. The final pose parameters are obtained by minimizing the combined cost function.

[0302]

[0303] in This represents the reprojection error of the PnP estimate. This indicates the polar constraint error. is a weight factor, the cost function balances the single-view pose accuracy and multi-view geometric consistency by combining the re-projection error with the epipolar constraint.

[0304] Finally, a more accurate and robust final bin pose (in the global world coordinate system) is output in step S75 after joint optimization under multi-view geometric constraints.

[0305] It can be understood that the scheme provided in the embodiment effectively solves the problem that single-view information is incomplete due to occlusion and cannot be detected, and the accuracy of the final output pose is generally better than that of any single-view independent solution by fusing multi-view information and using more observation data and geometric constraints.

[0306] Embodiment Two

[0307] The embodiment provides a bin stack three-dimensional pose detection device based on two-dimensional images and geometric constraints, and the device comprises:

[0308] An ROI candidate region acquisition module is configured to acquire a plurality of bin stack images through cameras with different viewing angles, select an arbitrary single-view bin stack image, perform target detection on a bin stack surface in the single-view bin stack image, and obtain an ROI candidate region.

[0309] A preprocessing module is configured to pre-process the ROI candidate region to obtain an ROI target region.

[0310] An edge detection module is configured to perform edge detection on the ROI target region by using a Canny operator to obtain edge points.

[0311] An optimal edge line segment acquisition module is configured to perform Hough transform straight line detection on the ROI target region to obtain candidate straight line segments, and perform edge straight line screening strategy to screen out optimal edge line segments representing bin boundaries from the candidate straight line segments.

[0312] A boundary straight line acquisition module is configured to assign the edge points to the nearest optimal edge line segment according to the distance between the edge points and each optimal edge line segment to form an edge point set, and perform straight line fitting on each set by using a RANSAC algorithm, and remove outliers through iterative optimization to obtain a boundary straight line corresponding to each edge point set.

[0313] An angle point acquisition module is configured to calculate the intersection points of the boundary straight lines, screen the intersection points according to a preset spatial geometric constraint, and determine four corner points of the bin through the intersection point screening result.

[0314] A 3D pose acquisition module is configured to calibrate the camera to obtain an intrinsic matrix and distortion coefficients, and obtain a bin body 3D pose through the four corner points of the bin by using a PnP algorithm based on the intrinsic matrix and the distortion coefficients.

[0315] Multi-view pose acquisition module: used for acquiring the 3D pose of the box under different views;

[0316] Multi-view fusion module: used for establishing a unified global coordinate system through multi-camera joint calibration, projecting the 3D pose of the box under different views into the unified global coordinate system, and acquiring the optimal three-dimensional pose of the box according to a preset constraint condition.

[0317] It can be understood that the same or similar parts in the above embodiments can be mutually referred to, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0318] It should be noted that, in the description of the present application, the terms "first", "second", etc. are only used for the purpose of description, and cannot be understood as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise specified, the meaning of "a plurality of" is at least two.

[0319] Any process or method descriptions in flow charts or otherwise described herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for implementing specific logic functions or other processes, and the various embodiments of the present application include additional implementations in which the functions described with reference to the figures are implemented in hardware, software, firmware, or combinations thereof. Such implementations can be performed within one or more appropriate processing devices, such as a general purpose computer, a special purpose computer, or other programmable processing device to produce the functions described in this disclosure. It will be understood that the embodiments of the present application can be implemented in a variety of ways, and that the application encompasses computer-readable media having stored thereon code means for implementing the processes described in the figures.

[0320] It should be understood that the parts of the present application can be realized by hardware, software, firmware or their combinations. In the above embodiments, a plurality of steps or methods can be realized by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if realized by hardware, and as in another embodiment, it can be realized by any one or their combinations of the following technologies known in the art: discrete logic circuit with logic gate circuit for implementing logic function on data signal, special integrated circuit with suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA), etc.

[0321] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment method can be completed by a program instructing the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0322] In addition, each function unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware, or in the form of a software function module. When the integrated module is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0323] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0324] In the description of the present specification, the description referring to the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0325] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and are not to be construed as limiting the present application, and those of ordinary skill in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for detecting the three-dimensional pose of a pallet based on two-dimensional images and geometric constraints, characterized in that, The method comprises: obtaining multiple pallet images through different view cameras, selecting an arbitrary single-view pallet image, performing target detection on the pallet surface in the single-view pallet image to obtain an ROI candidate region; preprocessing the ROI candidate region to obtain an ROI target region; performing edge detection on the ROI target region using a Canny operator to obtain edge points; performing straight line detection on the ROI target region using a Hough transform to obtain candidate straight line segments; performing an edge straight line screening strategy to screen optimal edge line segments representing the boundaries of the container from the candidate straight line segments; assigning the edge points to the nearest optimal edge line segment according to the distance between the edge points and each optimal edge line segment to form an edge point set, and performing straight line fitting on each set using a RANSAC algorithm, and removing outliers through iterative optimization to obtain the boundary straight line corresponding to each edge point set; calculating the intersection points of the boundary straight lines, screening the intersection points according to a preset spatial geometric constraint, and determining the four corner points of the container through the screening results of the intersection points; calibrating the camera to obtain an intrinsic matrix and distortion coefficients; based on the intrinsic matrix and the distortion coefficients, using a PnP algorithm to obtain the 3D pose of the container through the four corner points of the container; obtaining the 3D pose of the container under different views; establishing a unified global coordinate system through multi-camera joint calibration, projecting the 3D pose of the container under different views into the unified global coordinate system, and obtaining the optimal three-dimensional pose of the container according to a preset constraint condition.

2. The method of claim 1, wherein the preprocessing of the ROI candidate region to obtain an ROI target region comprises: gray-scale processing the ROI candidate region; performing adaptive contrast enhancement on the gray-scale processed image using a CLAHE algorithm; performing Gaussian filter processing on the adaptive contrast enhanced gray-scale image; performing morphological opening and closing operations on the Gaussian filtered gray-scale image; performing median filter processing on the gray-scale image after the morphological opening and closing operations to obtain the ROI target region.

3. The method of claim 2, wherein the edge detection on the ROI target region using a Canny operator to obtain edge points comprises: using a Sobel operator to calculate the gradient of the image in the x and y directions respectively to obtain gradient amplitude and gradient direction; performing non-maximum suppression on the gradient amplitude according to the gradient direction to obtain single-pixel edges; setting high and low threshold values, marking edge points greater than the high threshold value in the single-pixel edges as strong edge points, marking edge points between the high and low threshold values as weak edge points, and suppressing edge points less than the low threshold value to 0, and performing connection judgment on the weak edge points, if the weak edge points are connected with the strong edge points, then they are retained, otherwise they are suppressed to 0.

4. The method of claim 3, wherein the edge straight line screening strategy comprises: ​ ​ ​ The candidate straight line segments are classified and attributed to preset edge regions, a comprehensive score is calculated for each candidate straight line segment according to a preset scoring rule, and a straight line segment with the highest comprehensive score in each edge region is selected as the optimal edge line segment.

5. The method of claim 4, wherein, The comprehensive score is obtained by weighted calculation based on the normalized line segment length, position validity and edge region attribution of the candidate straight line segment; The position validity is determined according to whether the distances between the two end points of the candidate straight line segment and the image boundary are greater than a preset boundary safety distance, and if both distances are greater than the preset boundary safety distance, the position validity is assigned a value of 1, otherwise, the position validity is assigned a value of 0.

6. The method of claim 5, wherein, The edge region is a preset width region of the four edges of the image.

7. The method of claim 6, wherein, The filtering of the intersection points according to the preset spatial geometric constraints comprises: The intersection point is located in the ROI region, and the included angle between the two intersecting straight lines constituting the intersection point is within a preset right angle tolerance range, and the distance from the intersection point to the straight line is within a preset distance threshold range; The intersection point satisfying the conditions is taken as an effective corner point.

8. The method of claim 7, wherein, The 3D pose of the box body is obtained by the PnP algorithm through the four corner points of the box comprises: The PnP algorithm based on the Levenberg-Marquardt optimization method is used to solve the 3D pose of the box surface by iteratively minimizing the re-projection error between the coordinates of the three-dimensional model corner points corresponding to the three-dimensional size of the box under the camera model projected onto the image plane and the coordinates of the box corner points on the two-dimensional image.

9. The method of claim 8, wherein, The optimal three-dimensional pose of the box is obtained according to the preset constraint condition comprises: Performing epipolar constraint on the corresponding edge points detected across views so that the corresponding points in different views are located on conjugate epipolar lines, and obtaining epipolar constraint error; The epipolar constraint error and the re-projection error obtained by the PnP algorithm are fused to obtain a combined cost function, and the final pose parameters are obtained by minimizing the combined cost function, and the optimal three-dimensional pose of the box is obtained through the final pose parameters.

10. A device for detecting the three-dimensional pose of a palletized load based on two-dimensional images and geometric constraints, characterized in that, The device comprises: An ROI candidate region acquisition module: configured to obtain a plurality of pallet images through cameras with different viewing angles, select an arbitrary single-view pallet image, and perform target detection on the surface of the single-view pallet image to obtain an ROI candidate region; A preprocessing module: configured to pre-process the ROI candidate region to obtain an ROI target region; An edge detection module: configured to perform edge detection on the ROI target region using a Canny operator to obtain edge points; An optimal edge line segment acquisition module: configured to perform Hough transform straight line detection on the ROI target region to obtain candidate straight line segments, and perform edge straight line screening strategy to screen out optimal edge line segments representing the boundary of the box from the candidate straight line segments. The boundary straight line acquisition module is configured to assign the edge points to the nearest optimal edge line segment according to the distance of the edge points to each optimal edge line segment to form an edge point set, and perform straight line fitting on each set by using a RANSAC algorithm, and remove outliers by iterative optimization to acquire a boundary straight line corresponding to each edge point set; The corner point acquisition module is configured to calculate intersection points of the boundary straight lines, screen the intersection points according to a preset spatial geometric constraint, and determine four corner points of the container box according to the screening result of the intersection points; The 3D pose acquisition module is configured to calibrate the camera to acquire an intrinsic matrix and a distortion coefficient, acquire a 3D pose of the container box by using a PnP algorithm based on the intrinsic matrix and the distortion coefficient, and acquire a 3D pose of the container box under different perspectives; The multi-perspective pose acquisition module is configured to acquire the 3D pose of the container box under different perspectives; The multi-perspective fusion module is configured to establish a unified global coordinate system by joint calibration of the multiple cameras, project the 3D pose of the container box under different perspectives into the unified global coordinate system, and acquire an optimal 3D pose of the container box according to a preset constraint condition.

Images