Cargo instance segmentation and nine-dimensional pose estimation method and system

Abstract

This invention provides a method and system for cargo instance segmentation and nine-dimensional attitude estimation. The method involves spatially calibrating a camera and LiDAR simultaneously to obtain accurate extrinsic and intrinsic parameters, and then synchronously acquiring RGB images of the target scene and LiDAR point cloud data using the calibrated sensors. Next, a cross-modal attention mechanism is used to achieve deep fusion and output the instance segmentation result. Subsequently, color information is mapped to the point cloud based on the calibration parameters to generate a colored point cloud. Finally, combining color and geometric features, nine-dimensional attitude parameters (position, orientation, and size) are output. Through multimodal deep fusion and colored point cloud collaborative processing, this method solves the technical problems of limited two-dimensional information, sparse and noisy point clouds, difficulty in instance separation, insufficient multimodal fusion, and incomplete attitude information, thereby improving the accuracy of automated logistics operations.

Description

Technical Field

[0001] This invention relates to the field of industrial-grade logistics automation technology, and in particular to a method and system for cargo instance segmentation and nine-dimensional attitude estimation. Background Technology

[0002] Logistics automation has become a core direction for improving warehousing efficiency and reducing labor costs, and is widely used in various scenarios such as manufacturing, warehousing sorting, and loading and unloading. Accurate cargo identification, instance segmentation, and spatial pose estimation, as the core perception links of logistics automation systems, directly determine the accuracy and reliability of automated operations such as robotic arm grasping, automatic sorting, and intelligent stacking, and are key technological supports for achieving unmanned warehousing operations. In practical applications, logistics scenarios face complex situations such as cargo stacking, mutual occlusion, and changes in lighting, placing extremely high demands on the robustness, accuracy, and completeness of perception technologies.

[0003] Currently, cargo perception technologies in the logistics field mainly rely on single-modal data processing or simple multimodal result fusion. Among them, traditional methods are mostly based on two-dimensional image analysis to achieve cargo recognition, completing target detection and segmentation by extracting appearance features such as texture and color from the image; another type of technology uses radar to collect three-dimensional point cloud data, relying on the geometric information of the point cloud for target localization and contour extraction.

[0004] However, existing technologies suffer from several insurmountable shortcomings in complex logistics scenarios: First, two-dimensional image data can only provide planar appearance information of goods, failing to acquire precise three-dimensional spatial dimensions and orientation, thus making it difficult to meet the spatial positioning accuracy requirements of automated operations. Second, radar-acquired point cloud data is inherently sparse and susceptible to environmental noise interference; directly using it for segmentation leads to insufficient accuracy and an inability to accurately delineate the outline of goods. Third, in real-world scenarios where goods are stacked and mutually occluded, single-modal data struggles to effectively distinguish different goods instances, resulting in difficulties in instance separation and impacting the accuracy of subsequent operations. Fourth, 6D pose estimation cannot fully describe the spatial state of goods, making it difficult to support high-precision automated operations. Fifth, existing multimodal fusion methods fail to fully leverage the respective advantages of rich image texture details and precise point cloud geometric information, resulting in poor fusion effects and an inability to synergistically improve perception performance. These problems directly lead to existing technologies failing to meet the accuracy, reliability, and completeness requirements of automated logistics operations, becoming a key bottleneck restricting the development of smart logistics to a higher level. Summary of the Invention

[0005] This invention provides a method and system for cargo instance segmentation and nine-dimensional attitude estimation to solve the problem of low accuracy in automated logistics operations in the prior art.

[0006] On one hand, the present invention provides a method for cargo instance segmentation and nine-dimensional pose estimation, including:

[0007] Spatial synchronous calibration of the camera and lidar used for data acquisition is performed to obtain calibration parameters including sensor extrinsic and intrinsic parameters;

[0008] The calibrated camera and LiDAR simultaneously acquire RGB images of the target scene and LiDAR point cloud data.

[0009] The RGB image and the LiDAR point cloud are segmented into instances respectively. Then, the semantic and geometric features of the RGB image and the LiDAR point cloud data are fused through a cross-modal feature fusion mechanism to output the cargo instance-level segmentation result.

[0010] Based on the calibration parameters, the color information of the RGB image is mapped onto the LiDAR point cloud data corresponding to the cargo instance-level segmentation result to generate a color point cloud for each cargo instance;

[0011] The colored point cloud is subjected to geometric processing that fuses color information, and the nine-dimensional pose parameters of the cargo instance-level segmentation result are output. The nine-dimensional pose parameters include three-dimensional position coordinates, cargo three-dimensional orientation, and cargo three-dimensional dimensions.

[0012] Optionally, the step of performing instance segmentation on the RGB image and the LiDAR point cloud respectively, and fusing the semantic and geometric features of the RGB image and the LiDAR point cloud data through a cross-modal feature fusion mechanism to output cargo instance-level segmentation results includes:

[0013] The RGB image is processed using an improved Mask R-CNN network to output image instance segmentation results. The improved Mask R-CNN network introduces a coordinate attention mechanism into the feature pyramid network.

[0014] The PointGroup network is used to perform instance clustering on the LiDAR point cloud data, and the point cloud instance segmentation results are output.

[0015] The image instance segmentation result and the point cloud instance segmentation result are fused through a cross-modal attention fusion mechanism to obtain fused features;

[0016] Based on the fusion features, after nonmaximum suppression and cross-modal verification, the final cargo instance-level segmentation result is output.

[0017] Optionally, the step of fusing the image instance segmentation result and the point cloud instance segmentation result through a cross-modal attention fusion mechanism to obtain fused features includes:

[0018] Image features are extracted from the image instance segmentation results, and point cloud features are extracted from the point cloud instance segmentation results;

[0019] The image features and the point cloud features are concatenated and input into a sub-network consisting of global pooling and fully connected layers for feature analysis to generate dynamic weights.

[0020] The dynamic weights are normalized using Softmax to output image feature weights, point cloud feature weights, and interaction feature weights.

[0021] A cross-attention mechanism is used to generate interactive features by using image features as queries and point cloud features as keys and values.

[0022] Substitute the image features, point cloud features, image feature weights, point cloud feature weights, interactive features, and interactive feature weights into the fusion formula to generate fused features.

[0023] The fusion formula is as follows:

[0024] F_inter = ;

[0025] ;

[0026] in, As a feature of fusion, Image feature weights, For point cloud feature weights, For interactive feature weights, For image features, Point cloud features F_inter For interactive features.

[0027] Optionally, the step of mapping the color information of the RGB image onto the LiDAR point cloud data corresponding to the cargo instance-level segmentation result according to the calibration parameters, to generate a color point cloud for each cargo instance, includes:

[0028] Based on the calibration parameters, the three-dimensional points in the LiDAR point cloud data corresponding to the cargo instance-level segmentation result are transformed from the LiDAR coordinate system to the camera coordinate system, and then projected onto the pixel plane of the RGB image to obtain the pixel coordinates corresponding to the three-dimensional points.

[0029] For each pixel coordinate, a color value is calculated from the RGB image using bilinear interpolation;

[0030] The color values ​​are assigned to the corresponding 3D points to generate a color point cloud for each cargo instance.

[0031] Optionally, the colored point cloud is subjected to geometric processing that fuses color information to output the three-dimensional position coordinates of the cargo instance-level segmentation result, including:

[0032] Calculate the average geometric coordinates of all three-dimensional points in the color point cloud to obtain the geometric centroid;

[0033] The color difference value is obtained by calculating the Euclidean distance between the color value of each three-dimensional point in the color point cloud and the average color at the geometric centroid.

[0034] Based on the color difference value, a weight is assigned to each 3D point using a Gaussian function; the smaller the color difference value, the higher the weight of the 3D point.

[0035] The geometric coordinates of each three-dimensional point are weighted and averaged according to the weights to calculate the three-dimensional position coordinates of the cargo instance.

[0036] Optionally, the colored point cloud is subjected to geometric processing that fuses color information to output the cargo's 3D orientation as a cargo instance-level segmentation result, including:

[0037] Principal component analysis is performed on the geometric coordinates of the colored point cloud to obtain three orthogonal principal direction vectors, and an initial orientation estimate is determined based on the principal direction vectors; the principal direction vectors are V1, V2 and V3, where V1 is greater than V2 and V3.

[0038] The colored point cloud is projected onto a two-dimensional plane composed of V1 and V2, and the color information of the three-dimensional points is preserved to generate a textured two-dimensional projection image.

[0039] Extract the orientation sensitivity features of the two-dimensional projection image;

[0040] Based on the orientation sensitivity feature, the first similarity between the features of the two-dimensional projection image in the 0-degree rotation state and the features of the predefined positive template, and the second similarity between the features of the image in the 180-degree rotation state and the features of the predefined positive template are calculated respectively.

[0041] Compare the first similarity with the second similarity, and select the rotation state direction corresponding to the higher value of the first similarity and the second similarity as the direction to eliminate ambiguity;

[0042] Based on the disambiguation direction and the initial orientation estimate, the three-dimensional orientation of the cargo is output.

[0043] Optionally, after projecting the colored point cloud onto a two-dimensional plane composed of V1 and V2, and retaining the color information of the three-dimensional points to generate a textured two-dimensional projection image, the method further includes:

[0044] When the texture direction feature discrimination of the two-dimensional projection image is lower than a preset threshold, the two-dimensional projection image is input into a pre-trained lightweight convolutional neural network.

[0045] A continuous angular offset is calculated by regression using the lightweight convolutional neural network.

[0046] The initial orientation estimate obtained based on principal component analysis is fine-tuned by the angular offset to obtain the three-dimensional orientation of the cargo.

[0047] Optionally, the colored point cloud is subjected to geometric processing that fuses color information to output the three-dimensional dimensions of the cargo in the cargo instance-level segmentation result, including:

[0048] Based on the object coordinate system determined by the three-dimensional orientation of the cargo, the axis-aligned bounding box of the colored point cloud is calculated to obtain an initial size estimate;

[0049] The initial size estimate is matched with a pre-stored library of standard logistics box size templates;

[0050] If a match is successful, the matched template size will be used as the three-dimensional size of the goods.

[0051] Optionally, after matching the initial size estimate with a pre-stored library of standard logistics box size templates, the method further includes:

[0052] If the matching fails, the main color of the colored point cloud is calculated, and the color distribution of the three-dimensional points in the boundary region of the colored point cloud along each coordinate axis is analyzed.

[0053] When the difference in color distribution exceeds the color difference threshold, it is determined to be an occluded boundary. The initial size estimate is optimized by virtually expanding the bounding box boundary until the predicted color changes abruptly, and the three-dimensional size of the cargo is output.

[0054] On the other hand, the present invention also provides a cargo instance segmentation and nine-dimensional pose estimation system, comprising:

[0055] The calibration module is used to perform spatial synchronous calibration of the camera and lidar used for data acquisition, and to obtain calibration parameters including sensor extrinsic and intrinsic parameters.

[0056] The data acquisition module is used to simultaneously acquire RGB images of the target scene and LiDAR point cloud data through the calibrated camera and LiDAR.

[0057] The instance segmentation module is used to perform instance segmentation on the RGB image and the LiDAR point cloud respectively, and to fuse the semantic and geometric features of the RGB image and the LiDAR point cloud data through a cross-modal feature fusion mechanism to output the cargo instance-level segmentation result;

[0058] The color point cloud generation module is used to map the color information of the RGB image onto the lidar point cloud data corresponding to the cargo instance-level segmentation result according to the calibration parameters, so as to generate a color point cloud for each cargo instance.

[0059] The pose estimation module is used to perform geometric processing on the colored point cloud by fusing color information, and output nine-dimensional pose parameters of the cargo instance-level segmentation result. The nine-dimensional pose parameters include three-dimensional position coordinates, cargo three-dimensional orientation, and cargo three-dimensional dimensions.

[0060] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the cargo instance segmentation and nine-dimensional pose estimation method as described above.

[0061] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cargo instance segmentation and nine-dimensional pose estimation method as described above.

[0062] On the other hand, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the cargo instance segmentation and nine-dimensional pose estimation method as described above.

[0063] This invention provides a method and system for cargo instance segmentation and nine-dimensional attitude estimation. The method involves spatially calibrating a camera and LiDAR simultaneously to obtain accurate extrinsic and intrinsic parameters, and then synchronously acquiring RGB images of the target scene and LiDAR point cloud data using the calibrated sensors. Next, a cross-modal attention mechanism is used to achieve deep fusion and output the instance segmentation result. Subsequently, color information is mapped to the point cloud based on the calibration parameters to generate a colored point cloud. Finally, combining color and geometric features, nine-dimensional attitude parameters (position, orientation, and size) are output. Through multimodal deep fusion and colored point cloud collaborative processing, this method solves the technical problems of limited two-dimensional information, sparse and noisy point clouds, difficulty in instance separation, insufficient multimodal fusion, and incomplete attitude information, thereby improving the accuracy of automated logistics operations. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0065] Figure 1This is a flowchart illustrating the cargo instance segmentation and nine-dimensional pose estimation method provided in an embodiment of the present invention;

[0066] Figure 2 This is a schematic diagram of the cargo instance segmentation and nine-dimensional attitude estimation system provided in an embodiment of the present invention;

[0067] Figure 3 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0069] Figure 1 This is a flowchart illustrating the cargo instance segmentation and nine-dimensional pose estimation method provided in an embodiment of the present invention.

[0070] like Figure 1 As shown, the cargo instance segmentation and nine-dimensional pose estimation method provided in this embodiment of the invention includes the following steps:

[0071] 101. Perform spatial synchronization calibration on the camera and lidar used for data acquisition to obtain calibration parameters, including the sensor's extrinsic and intrinsic parameters.

[0072] The intrinsic parameters mainly include the camera's inherent parameters, such as focal length, principal point coordinates, and distortion coefficients. These intrinsic parameters determine the mapping relationship between the camera's projection of three-dimensional spatial points onto the two-dimensional image plane. The extrinsic parameters characterize the relative spatial attitude and positional relationship between the camera and the LiDAR, specifically including rotation matrices and translation vectors, used to achieve a unified transformation between the two sensor coordinate systems. During calibration, a standard calibration board is deployed, and the corresponding feature information of the calibration board in the camera image and LiDAR point cloud under different attitudes is collected. Multi-view geometric algorithms are used to solve for the optimal intrinsic and extrinsic parameters. Simultaneously, a hardware triggering mechanism ensures the spatiotemporal synchronization of data acquisition, keeping the sensor trigger delay within 1ms and the extrinsic parameter calibration error strictly controlled below 0.5 pixels, ensuring the accuracy of subsequent RGB image color information mapping to the point cloud and cross-modal feature fusion.

[0073] 102. Simultaneously acquire RGB images of the target scene and LiDAR point cloud data using the calibrated camera and LiDAR.

[0074] Specifically, the camera captures 1080P high-definition RGB images, fully recording the appearance texture, color distribution, and scene semantic information of the goods; the LiDAR simultaneously acquires 16-line 3D point cloud data, obtaining the spatial geometry, positional relationships, and depth information of the goods. To ensure the temporal consistency of the two types of data, a hardware triggering mechanism is used to achieve synchronous acquisition, strictly controlling the spatiotemporal synchronization error of data acquisition to within 1ms, avoiding feature misalignment caused by time differences; at the same time, combined with the dynamic needs of the logistics scenario, a collection frequency of 15-30Hz is set, which can match the movement speed of goods on the assembly line while ensuring the continuity and integrity of data acquisition. Addressing the issue of fluctuating light intensity in the warehouse, the camera has an automatic exposure adjustment function, which can adapt to different lighting environments in real time, ensuring the clarity and color accuracy of the RGB images, providing a high-quality, highly consistent basic data source for subsequent cross-modal feature fusion, instance segmentation, and pose estimation.

[0075] 103. Perform instance segmentation on the RGB image and the LiDAR point cloud respectively, and fuse the semantic and geometric features of the RGB image and the LiDAR point cloud data through a cross-modal feature fusion mechanism to output the cargo instance-level segmentation result.

[0076] Different segmentation algorithms can be used for instance segmentation of RGB images and LiDAR point clouds. For RGB images, a deep learning-based instance segmentation network can be used.

[0077] Specifically, instance segmentation is performed on RGB images and LiDAR point clouds respectively. Then, through a cross-modal feature fusion mechanism, the semantic and geometric features of the RGB images and LiDAR point cloud data are fused to output cargo instance-level segmentation results, including:

[0078] An improved Mask R-CNN network is used to process RGB images and output image instance segmentation results. The improved Mask R-CNN network introduces a coordinate attention mechanism into the feature pyramid network.

[0079] Specifically, an improved Mask R-CNN network is used to perform instance segmentation on synchronously acquired RGB images. A coordinate attention mechanism introduced after the feature pyramid network accurately captures the spatial location information and boundary features of cargo targets in the image, effectively improving the ability to distinguish cargo boundaries in stacked and occluded scenarios. Simultaneously, the backbone of the Mask R-CNN network is replaced with the more computationally efficient RegNetX-GF or VoVNetV2. Combined with simulated stacking and illumination robustness training designed for logistics scenarios, as well as an optimized loss function with added edge focusing loss, the clarity of the segmentation mask boundaries and the accuracy of classification are further guaranteed. Finally, the Mask R-CNN network outputs the pixel-level mask and class confidence score for each cargo target, forming preliminary image instance segmentation results.

[0080] PointGroup network is used to perform instance clustering on LiDAR point cloud data and output point cloud instance segmentation results.

[0081] Specifically, the PointGroup network is used to perform instance clustering processing on the raw point cloud data acquired by LiDAR. The PointGroup network has a powerful ability to mine point cloud geometric features and can achieve semantic grouping and geometric segmentation of point clouds through efficient instance clustering algorithms, addressing the characteristics of sparse and noisy point cloud data. In the processing, the PointGroup network first performs voxelization on the raw point cloud, and then completes instance clustering based on the spatial location and distance relationships of the point clouds, accurately distinguishing the point cloud sets corresponding to different cargo targets. Finally, it outputs the point cloud instance segmentation results containing cargo geometric contour information and the corresponding confidence parameters.

[0082] By using a cross-modal attention fusion mechanism, the image instance segmentation results and the point cloud instance segmentation results are fused to obtain fused features.

[0083] Specifically, a cross-modal attention fusion mechanism is used to fuse image instance segmentation results with point cloud instance segmentation results to obtain fused features, including:

[0084] Image features are extracted from image instance segmentation results, and point cloud features are extracted from point cloud instance segmentation results;

[0085] Image features and point cloud features are concatenated and input into a sub-network consisting of global pooling and fully connected layers for feature analysis to generate dynamic weights.

[0086] The dynamic weights are normalized using Softmax, and the outputs are image feature weights, point cloud feature weights, and interaction feature weights.

[0087] A cross-attention mechanism is used to generate interactive features by using image features as queries and point cloud features as keys and values.

[0088] Substitute the image features, point cloud features, image feature weights, point cloud feature weights, interactive features, and interactive feature weights into the fusion formula to generate fused features.

[0089] The fusion formula is:

[0090] F_inter = ;

[0091] ;

[0092] in, As a feature of fusion, Image feature weights, For point cloud feature weights, For interactive feature weights, For image features, Point cloud features F_inter For interactive features.

[0093] Specifically, image features rich in semantic texture information are extracted from the image instance segmentation results output by the improved Mask R-CNN network, and point cloud features containing spatial geometric structures are extracted from the point cloud instance segmentation results output by the PointGroup network.

[0094] Subsequently, the image features and point cloud features are concatenated dimensionally to form joint features, which are then input into a subnetwork consisting of a global pooling layer and a fully connected layer. The subnetwork is used to perform in-depth analysis on the correlation and complementarity of the two types of modal features to generate initial dynamic weights.

[0095] Next, Softmax normalization is performed on the initial dynamic weights to obtain normalized image feature weights, point cloud feature weights, and interaction feature weights. Then, a cross-attention mechanism is employed, using image features as the query vector and point cloud features as the key and value vectors, respectively. By calculating attention weights and performing a weighted summation of the point cloud features, interaction features that fuse the core association information of the two modalities are generated, satisfying the following conditions: F_inter = ( (For cross-attention interaction operations); finally, the image features, point cloud features, image feature weights corresponding to the image features, point cloud feature weights corresponding to the point cloud features, and interaction features and interaction feature weights are substituted into the fusion formula, and the fused features are obtained through element-wise multiplication (⊙) and accumulation operations. .

[0096] Based on the fusion features, after nonmaximum suppression and cross-modal verification, the final cargo instance-level segmentation result is output.

[0097] The process involves eliminating redundant segmentation candidates using non-maximum suppression, retaining only the most confident and accurate segmentation proposals. Consistency checks are then performed on the segmentation results for both image and point cloud modes to eliminate abnormal results caused by single-modal noise or false detections, ensuring the reliability of the segmentation results. After this dual screening and verification, a unique instance ID is assigned to each valid cargo target. The final output is a cargo instance-level segmentation result that accurately distinguishes individual cargo instances, effectively addressing cargo stacking and occlusion issues in logistics scenarios.

[0098] 104. Based on the calibration parameters, map the color information of the RGB image onto the LiDAR point cloud data corresponding to the cargo instance-level segmentation results to generate a color point cloud for each cargo instance.

[0099] Among them, generating a colored point cloud for each cargo instance can provide a data source for spacing color and geometric information. The color information is used to improve pose estimation, which is suitable for cargo with obvious texture.

[0100] Specifically, based on the calibration parameters, the color information of the RGB image is mapped onto the LiDAR point cloud data corresponding to the cargo instance-level segmentation results, generating a color point cloud for each cargo instance, including:

[0101] Based on the calibration parameters, the 3D points in the LiDAR point cloud data corresponding to the cargo instance-level segmentation results are transformed from the LiDAR coordinate system to the camera coordinate system, and then projected onto the pixel plane of the RGB image to obtain the pixel coordinates corresponding to the 3D points.

[0102] Here, it is assumed that the sensor calibration parameters are obtained as extrinsic parameters: rotation matrix R = [[r11,r12,r13],[r21,r22,r23],[r31,r32,r33]], translation vector T = [tx,ty,tz], camera intrinsic parameters are focal length fx = 1200px, fy = 1200px, principal point coordinates cx = 960px, cy = 540px. For a specific 3D point P_lidar = (1.2m, 0.8m, 3.5m) in the lidar point cloud data corresponding to cargo instance A.

[0103] Based on the extrinsic parameters, a coordinate transformation is performed: P_cam = R × P_lidar + T, to calculate the point P_cam = (0.95m, 0.72m, 3.2m) in the camera coordinate system; then, it is projected onto the pixel plane through the camera intrinsic parameters. Using the projection formulas u = fx × (P_cam.x / P_cam.z) + cx and v = fy × (P_cam.y / P_cam.z) + cy, the values ​​are substituted to calculate u = 1200 × (0.95 / 3.2) + 960 ≈ 1246.875px and v = 1200 × (0.72 / 3.2) + 540 ≈ 792px. Finally, the pixel coordinates (u, v) corresponding to this 3D point are obtained as (1246.88px, 792.0px).

[0104] For each pixel coordinate, the color value is calculated from the RGB image using bilinear interpolation;

[0105] After obtaining the pixel coordinates (1246.88px, 792.0px), the coordinates of the four adjacent integer pixels surrounding these pixel coordinates are determined as follows: top left (1246px, 792px), top right (1247px, 792px), bottom left (1246px, 793px), and bottom right (1247px, 793px). Assume that the color values ​​of these four pixels read from the RGB image are: (1246, 792) = (255, 250, 245), (1247, 792) = (254, 249, 244), (1246, 793) = (253, 248, 243), and (1247, 793) = (252, 247, 242). Calculate the horizontal interpolation weight dx=0.88 and the vertical interpolation weight dy=0.0. First, perform horizontal interpolation: the interpolation values ​​for the top left and top right corners are R1=255×(1-0.88)+254×0.88≈254; G1=250×0.12+249×0.88≈249, B1=245×0.12+244×0.88≈244; the interpolation values ​​for the bottom left and bottom right corners are R2=253×0.12+252×0.88≈252, G2=248×0.12+247×0.88≈247, B2=243×0.12+242×0.88≈242. Then, perform vertical interpolation to finally obtain the color value of the pixel coordinates as (R,G,B)=(254,249,244).

[0106] Assign color values ​​to the corresponding 3D points to generate a colored point cloud for each cargo instance.

[0107] For example, the LiDAR point cloud corresponding to cargo instance A contains 1200 three-dimensional points. The pixel coordinates of each three-dimensional point are obtained through the above method, and the RGB color value is obtained through bilinear interpolation. For example, the 3D point P1 = (1.2m, 0.8m, 3.5m) corresponds to the color (254, 249, 244), the 3D point P2 = (1.25m, 0.82m, 3.55m) corresponds to the color (253, 248, 243), the 3D point P3 = (1.18m, 0.78m, 3.48m) corresponds to the color (255, 250, 245), and so on. All 1200 3D points are bound to their corresponding RGB color values ​​one by one, forming a point set in which each point contains six-dimensional information (x, y, z, r, g, b), ultimately generating the color point cloud of cargo instance A. Similarly, the above operation is repeated on the point cloud data of other cargo instances to generate their own independent color point clouds, realizing the distinction between the color point clouds of different cargo instances.

[0108] 105. Perform geometric processing on the colored point cloud to fuse color information, and output the nine-dimensional pose parameters of the cargo instance-level segmentation result. The nine-dimensional pose parameters include three-dimensional position coordinates, cargo three-dimensional orientation, and cargo three-dimensional dimensions.

[0109] The nine-dimensional pose parameters of the output cargo instance-level segmentation result can be divided into the three-dimensional position coordinates of the output cargo instance, the three-dimensional orientation of the cargo, and the three-dimensional dimensions of the cargo.

[0110] The process involves geometric processing of the colored point cloud, fusing color information, to output the 3D position coordinates of the cargo instance-level segmentation result, including:

[0111] Calculate the average geometric coordinates of all three-dimensional points in the color point cloud to obtain the geometric centroid.

[0112] The color difference value is obtained by calculating the Euclidean distance between the color value of each 3D point in the color point cloud and the average color at the geometric centroid.

[0113] Based on the color difference value, a weight is assigned to each 3D point using a Gaussian function; the smaller the color difference value, the higher the weight of the 3D point.

[0114] The geometric coordinates of each 3D point are weighted and averaged to calculate the 3D position coordinates of the cargo instance.

[0115] Specifically, the geometric processing of color information is performed on the color point cloud and RGB color information of a single cargo instance. First, all three-dimensional points in the color point cloud are traversed, and the average values ​​of the x-coordinate, y-coordinate, and z-coordinate of all three-dimensional points are calculated. These three values ​​constitute the geometric centroid of the color point cloud, C_geo = (x_avg, y_avg, z_avg). For example, if a cargo color point cloud contains 500 three-dimensional points, and the sum of its x-coordinates is 500m, the sum of its y-coordinates is 300m, and the sum of its z-coordinates is 1000m, then the geometric centroid C_geo = (1.0m, 0.6m, 2.0m). Next, the RGB color values ​​of all three-dimensional points are converted to the CIELAB color space, and the average color at the geometric centroid C_geo is calculated, which is the component mean of the LAB values ​​of all points (L_avg, a_avg, b_avg). Then, the Euclidean distance formula is used:

[0116] ;

[0117] Calculate the color difference value d_i between the LAB value of each 3D point and the average LAB value; then, use a Gaussian function to assign weights to each point. If the color difference value d_i = 1 (close to the main color), the weight w_i ≈ exp(-1 / (50)) ≈ 0.98; if d_i = 5 (larger color difference), the weight w_i ≈ exp(-25 / 50) = 0.606, realizing the allocation logic that the smaller the color difference value, the higher the weight; finally, calculate the weighted average of the x, y, and z coordinates of each 3D point to obtain the 3D position coordinates of the cargo instance.

[0118] In addition, geometric processing is performed on the colored point cloud to fuse color information, and the cargo instance-level segmentation result is output, including the cargo's 3D orientation, which includes:

[0119] Principal component analysis is performed on the geometric coordinates of the colored point cloud to obtain three orthogonal principal direction vectors, and the initial orientation estimate is determined based on the principal direction vectors.

[0120] Among them, the main direction vectors are V1, V2 and V3, where V1 is greater than V2 and V3 is greater than V3.

[0121] The colored point cloud is projected onto a two-dimensional plane composed of V1 and V2, and the color information of the three-dimensional points is preserved to generate a textured two-dimensional projection image.

[0122] Extract orientation sensitivity features from two-dimensional projected images.

[0123] Based on orientation-sensitive features, the first similarity between the features of the two-dimensional projected image in the 0-degree rotation state and the features of the predefined positive template is calculated, as well as the second similarity between the features of the image in the 180-degree rotation state and the features of the predefined positive template.

[0124] Compare the first similarity and the second similarity, and select the rotation direction corresponding to the higher value of the first similarity and the second similarity as the direction to eliminate ambiguity.

[0125] Based on the unambiguous direction and the initial orientation estimate, the three-dimensional orientation of the cargo is output.

[0126] Specifically, the geometric coordinates (x, y, z) of all 3D points in the color point cloud are extracted. Color information is removed, and only spatial location data is retained. Principal component analysis is used to calculate the covariance matrix of the geometric coordinates of the 3D points. The eigenvalues ​​and corresponding eigenvectors of the covariance matrix are solved to obtain three orthogonal principal direction vectors V1, V2, and V3, where V1 is greater than V2, which is greater than V3. V1 is the main extension direction of the cargo, V2 is the secondary extension direction, and V3 is the normal. An initial coordinate system is then constructed using the three principal direction vectors. Based on the transformation relationship between the initial coordinate system and the global coordinate system of the logistics, the initial orientation estimate R_pca of the cargo is determined. For example, V1=(0.98,0.15,0.02), V2=(-0.15,0.98,0.03), and V3=(0.01,0.02,0.998), corresponding to initial orientations roll=0.5°, pitch=1.2°, and yaw=30.8°.

[0127] Construct a two-dimensional plane with V1 as the x-axis and V2 as the y-axis. Through projection transformation, map each three-dimensional point (x, y, z) in the color point cloud to the two-dimensional plane to obtain two-dimensional coordinates (u, v). At the same time, retain the RGB color information corresponding to each point to generate a textured two-dimensional projection map. For example, the two-dimensional coordinates of a three-dimensional point (1.2, 0.8, 2.0) after projection are (256, 240) and the color value is (254, 249, 244), which corresponds to the color of the pixel in the image.

[0128] Next, orientation-sensitive feature extraction is used to extract features from the two-dimensional projected image. Specifically, the image patch size is 64x64 pixels, the cell unit size is 32x32 pixels, the cell units within the patch are arranged in a 2x2 pattern, the patch sliding step size is 32 pixels, and the gradient direction is quantized into 9 intervals. For a 512x512 image, the final oriented gradient histogram feature vector is obtained. Simultaneously, the gray-level co-occurrence matrix of the image is calculated, taking four statistics—contrast, correlation, energy, and homogeneity—calculated at 0°, 45°, 90°, and 135°, and then averaging them to obtain a 4-dimensional gray-level co-occurrence matrix feature vector.

[0129] Finally, the eigenvectors of the directional gradient histogram and the eigenvectors of the gray-level co-occurrence matrix are concatenated to form the final directional sensitivity eigenvectors.

[0130] Then, the predefined positive template features are retrieved, and the cosine similarity between the feature vector of the two-dimensional projected image in the 0-degree rotation state and the positive template features in the 180-degree rotation state is calculated respectively: if the first similarity is 0.93 when rotating at 0 degrees and the second similarity is 0.67 when rotating at 180 degrees, then the direction corresponding to the 0-degree rotation is selected as the direction to eliminate ambiguity and determine the positive orientation of the goods.

[0131] Finally, the deambiguity direction and the initial orientation estimate are fused and calibrated. Based on the main direction vector of the initial orientation, the orientation consistency of V1 and V2 is adjusted according to the deambiguity direction to avoid 180-degree reversal. Finally, the three-dimensional orientation parameters of the goods in the global logistics coordinate system are output, achieving unambiguous and high-precision orientation estimation.

[0132] In addition, after projecting the colored point cloud onto a two-dimensional plane composed of V1 and V2, and preserving the color information of the three-dimensional points to generate a textured two-dimensional projection image, the process also includes:

[0133] When the texture orientation feature discrimination of the 2D projection image is lower than a preset threshold, the 2D projection image is input into a pre-trained lightweight convolutional neural network.

[0134] A continuous angular offset is calculated using a lightweight convolutional neural network regression.

[0135] The initial orientation estimate obtained from principal component analysis is fine-tuned by adjusting the angle offset to obtain the three-dimensional orientation of the cargo.

[0136] Specifically, when the texture orientation feature discriminativeness of a 2D projected image is lower than a preset threshold, the 2D projected image is input into a pre-trained lightweight convolutional neural network. The preset threshold is determined statistically from training samples and represents the discriminativeness of texture orientation information. When the feature discriminativeness is lower than the threshold, it indicates that the texture orientation is not significant, and orientation ambiguity cannot be effectively eliminated through template matching. The lightweight network uses MobileNetV2 as its backbone, reducing computational complexity by simplifying the number of channels and introducing depthwise separable convolutions. After training on a large number of logistics scenario samples of goods without strong textures, it possesses accurate angular offset regression capabilities. The lightweight convolutional neural network takes the 2D projected image as input and, through feature extraction, pooling, and fully connected layer operations, regresses and outputs a continuous angular offset to correct the orientation deviation caused by pure geometric principal component analysis.

[0137] Finally, the angular offset is superimposed on the initial orientation estimate obtained based on principal component analysis, and the three-dimensional rotation angle of the initial orientation is fine-tuned to finally output an accurate three-dimensional orientation of the cargo, ensuring that highly reliable orientation estimation can still be achieved even in scenarios with insufficient texture information.

[0138] In some embodiments, geometric processing that fuses color information is performed on the color point cloud to output the three-dimensional dimensions of the cargo as a cargo instance-level segmentation result, including:

[0139] Based on the object coordinate system determined by the three-dimensional orientation of the cargo, the axis-aligned bounding box of the colored point cloud is calculated to obtain an initial size estimate.

[0140] The initial size estimate is matched with a pre-stored library of standard logistics box size templates.

[0141] If a match is successful, the matched template size will be used as the three-dimensional size of the goods.

[0142] Specifically, an object coordinate system is constructed based on the determined 3D orientation of the cargo, with the principal direction vectors V1, V2, and V3 obtained from principal component analysis serving as the x, y, and z axes of the object coordinate system, respectively. V1 corresponds to the cargo's length direction, V2 to its width direction, and V3 to its height direction, ensuring complete alignment between the object coordinate system and the cargo's actual spatial orientation.

[0143] Subsequently, the coordinates of all 3D points in the colored point cloud of the cargo instance in the object coordinate system are traversed. The maximum and minimum coordinate values ​​x_max and x_min in the x-axis direction, y_max and y_min in the y-axis direction, and z_max and z_min in the z-axis direction are calculated respectively. The initial size estimate is obtained by calculating the difference. In the initial size estimate, the length l_init = x_max - x_min, the width w_init = y_max - y_min, and the height h_init = z_max - z_min. For example, if the x-axis range of a cargo point cloud in the object coordinate system is [0.1m, 0.7m], the y-axis range is [0.2m, 0.5m], and the z-axis range is [0.3m, 0.6m], then the initial size estimate is l_init = 0.6m, w_init = 0.3m, and h_init = 0.3m.

[0144] Next, a pre-stored standard logistics box size template library is invoked. This library includes standard box type parameters commonly used in the logistics industry, and each template is labeled with precise 3D dimensions and corresponding application scenario tags. The initial size estimate is then matched one by one with all standard dimensions in the template library. The matching method involves calculating the Euclidean distance between the initial size and each template size, where (l_temp, w_temp, h_temp) are the standard dimensions in the template library. A matching threshold of 0.003m (3mm) is set; if the Euclidean distance corresponding to a template is less than this threshold, the match is considered successful.

[0145] If a matching standard template exists, for example, if the initial dimensions of 0.602m, 0.298m, and 0.301m have an Euclidean distance of 0.0021m (less than 0.003m) between them and a standard 600mm×300mm×300mm box in the template library, then the standard dimensions of the matching template are directly used as the three-dimensional dimensions of the goods. By leveraging prior knowledge of standard logistics boxes, the initial dimension estimation errors caused by factors such as sparse LiDAR point clouds and partial occlusion are effectively eliminated, significantly improving the accuracy and reliability of standard box dimension recognition.

[0146] In addition, the initial size estimate is matched against a pre-stored library of standard logistics box size templates. If a match fails, the following steps are also taken:

[0147] If the matching fails, the main color of the colored point cloud is calculated, and the color distribution of the three-dimensional points in the boundary region of the colored point cloud along each coordinate axis is analyzed.

[0148] When the difference in color distribution exceeds the color difference threshold, it is determined to be an occluded boundary. The initial size estimate is optimized by virtually expanding the bounding box boundary until the predicted color changes abruptly, and the three-dimensional size of the cargo is output.

[0149] Specifically, if the initial size estimate fails to match the standard logistics box size template library, the main color of the colored point cloud is calculated. When calculating the main color, the RGB color values ​​of all 3D points are converted to the CIELAB color space, and the mean L_avg, a_avg, b_avg and standard deviation of the color components are statistically obtained. The main color range is defined by the mean ± 1 standard deviation to ensure that the main color can accurately represent the color characteristics of the goods themselves.

[0150] Subsequently, the boundary regions of the colored point cloud along the x, y, and z axes of the object coordinate system are focused on. The color distribution of the three-dimensional points in each boundary region is statistically analyzed, the LAB value of each point and the Euclidean distance d_color between the main color are calculated, and the mean and variance of the Euclidean distance d_color in the boundary region are statistically analyzed.

[0151] Set a color difference threshold T for color distribution. The color difference threshold T is determined based on the statistical analysis of samples in the logistics scenario. For example, if the color difference threshold T is set to 3.5, and the mean Euclidean distance d_color of a certain boundary area is greater than T, it indicates that the color of the boundary area deviates from the main body and is concentrated, which means it is an obscuring object or background. In this case, the boundary area is determined to be an obscured boundary.

[0152] For occluded boundaries, a step-by-step virtual expansion strategy is adopted. Starting from the initial bounding box boundary, the expansion proceeds outward along the coordinate axis at a step size of 0.005m / step. At each expansion step, the ideal color at that location is predicted based on the main color features. For example, the ideal color at a certain location can be predicted using linear extrapolation. The main color data of five consecutive pixels inside the occluded boundary are selected, and linear functions of color variation with spatial position are fitted: L=k×x+e, a=m×x+f, b=n×x+g. Here, L is the L component of the predicted point in the CIELAB color space, a is the a component of the predicted point in the CIELAB color space, b is the b component of the predicted point in the CIELAB color space, x is the coordinate axis position, k is the slope of lightness L as a function of x, m is the slope of red-green hue a as a function of x, n is the slope of yellow-blue hue b as a function of x, and e, f, and g are the intercepts of the linear functions of L, a, and b, respectively. The ideal color value at each predicted location along the expansion path is calculated based on the linear function extrapolation.

[0153] Then, the ideal color at the current predicted position on the expansion path is calculated, along with the CIELAB Euclidean distance between the ideal color and the actual color collected in the adjacent expansion steps. When the difference between the CIELAB Euclidean distance and the previous step distance exceeds the color distribution difference threshold T, it is determined to be a drastic color change. At this point, expansion is stopped, and the point of change is the actual boundary of the cargo.

[0154] Finally, by combining the initial coordinates of the unoccluded boundary with the actual boundary coordinates determined after expansion, the three-dimensional dimensions of the cargo are recalculated. For example, if the positive x-axis direction of a cargo is the occluded boundary, the initial x_max = 0.7m, and the actual boundary after expansion is 0.85m, the final corrected length l = 0.85m - x_min = 0.75m is calculated. After simultaneously optimizing the width and height parameters, the three-dimensional dimensions of the cargo are output, effectively solving the problem of underestimation of the initial dimensions caused by sparse point clouds or local occlusion.

[0155] Based on the same general inventive concept, this invention also protects a cargo instance segmentation and nine-dimensional attitude estimation system. The cargo instance segmentation and nine-dimensional attitude estimation system provided by this invention will be described below. The cargo instance segmentation and nine-dimensional attitude estimation system described below can be referred to in correspondence with the cargo instance segmentation and nine-dimensional attitude estimation method described above.

[0156] In some embodiments, such as Figure 2 As shown, the present invention also provides a cargo instance segmentation and nine-dimensional pose estimation system, comprising:

[0157] The calibration module 210 is used to perform spatial synchronous calibration of the camera and lidar used for data acquisition, and to obtain calibration parameters including the sensor's extrinsic and intrinsic parameters.

[0158] The data acquisition module 220 is used to simultaneously acquire RGB images of the target scene and LiDAR point cloud data through the calibrated camera and LiDAR;

[0159] The instance segmentation module 230 is used to perform instance segmentation on the RGB image and the LiDAR point cloud respectively, and to fuse the semantic and geometric features of the RGB image and the LiDAR point cloud data through a cross-modal feature fusion mechanism to output the cargo instance-level segmentation result.

[0160] The color point cloud generation module 240 is used to map the color information of the RGB image to the LiDAR point cloud data corresponding to the cargo instance-level segmentation result according to the calibration parameters, and generate the color point cloud for each cargo instance.

[0161] The pose estimation module 250 is used to perform geometric processing on the color point cloud to fuse color information and output nine-dimensional pose parameters of the cargo instance-level segmentation result. The nine-dimensional pose parameters include three-dimensional position coordinates, cargo three-dimensional orientation and cargo three-dimensional dimensions.

[0162] Figure 3 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention.

[0163] like Figure 3 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions stored in the memory 330 to execute a cargo instance segmentation and nine-dimensional pose estimation method.

[0164] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0165] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the cargo instance segmentation and nine-dimensional pose estimation methods provided by the above methods.

[0166] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cargo instance segmentation and nine-dimensional pose estimation methods provided by the above methods.

[0167] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for cargo instance segmentation and nine-dimensional pose estimation, the method comprising: include: Spatial synchronous calibration is performed on the camera and lidar used for data acquisition to obtain calibration parameters, including the sensor's extrinsic and intrinsic parameters. The calibrated camera and LiDAR simultaneously acquire RGB images of the target scene and LiDAR point cloud data. The RGB image and the LiDAR point cloud are segmented into instances respectively. Then, the semantic and geometric features of the RGB image and the LiDAR point cloud data are fused through a cross-modal feature fusion mechanism to output the cargo instance-level segmentation result. Based on the calibration parameters, the color information of the RGB image is mapped onto the LiDAR point cloud data corresponding to the cargo instance-level segmentation result to generate a color point cloud for each cargo instance; The color point cloud is subjected to geometric processing that fuses color information, and the nine-dimensional pose parameters of the cargo instance-level segmentation result are output. The nine-dimensional pose parameters include three-dimensional position coordinates, cargo three-dimensional orientation and cargo three-dimensional dimensions. The geometric processing of the color point cloud, which fuses color information, to output three-dimensional position coordinates includes: Calculate the average geometric coordinates of all three-dimensional points in the color point cloud to obtain the geometric centroid; The color difference value is obtained by calculating the Euclidean distance between the color value of each three-dimensional point in the color point cloud and the average color at the geometric centroid. Based on the color difference value, a weight is assigned to each 3D point using a Gaussian function; the smaller the color difference value, the higher the weight of the 3D point. The geometric coordinates of each three-dimensional point are weighted and averaged according to the weights to calculate the three-dimensional position coordinates of the cargo instance. The color point cloud is geometrically processed to fuse color information, and the resulting three-dimensional orientation of the cargo is output, including: Principal component analysis is performed on the geometric coordinates of the colored point cloud to obtain three orthogonal principal direction vectors, and an initial orientation estimate is determined based on the principal direction vectors; the principal direction vectors are V1, V2, and V3, where V1 is greater than V2 and V3. The colored point cloud is projected onto a two-dimensional plane composed of V1 and V2, and the color information of the three-dimensional points is preserved to generate a textured two-dimensional projection image. Extract the orientation sensitivity features of the two-dimensional projection image; Based on the orientation sensitivity feature, the first similarity between the features of the two-dimensional projection image in the 0-degree rotation state and the features of the predefined positive template, and the second similarity between the features of the image in the 180-degree rotation state and the features of the predefined positive template are calculated respectively. Compare the first similarity with the second similarity, and select the rotation state direction corresponding to the higher value of the first similarity and the second similarity as the direction to eliminate ambiguity; Based on the disambiguation direction and the initial orientation estimate, output the three-dimensional orientation of the cargo; The color point cloud is subjected to geometric processing that fuses color information, and the output cargo three-dimensional dimensions include: Based on the object coordinate system determined by the three-dimensional orientation of the cargo, the axis-aligned bounding box of the colored point cloud is calculated to obtain an initial size estimate; The initial size estimate is matched with a pre-stored library of standard logistics box size templates; If a match is successful, the matched template size will be used as the three-dimensional size of the goods.

2. The method of cargo instance segmentation and nine-dimensional pose estimation of claim 1, wherein, The process involves performing instance segmentation on the RGB image and the LiDAR point cloud, and then fusing the semantic and geometric features of the RGB image and the LiDAR point cloud data through a cross-modal feature fusion mechanism to output cargo instance-level segmentation results, including: The RGB image is processed using an improved Mask R-CNN network to output image instance segmentation results. The improved Mask R-CNN network introduces a coordinate attention mechanism into the feature pyramid network. The PointGroup network is used to perform instance clustering on the LiDAR point cloud data, and the point cloud instance segmentation results are output. The image instance segmentation result and the point cloud instance segmentation result are fused through a cross-modal attention fusion mechanism to obtain fused features; Based on the fusion features, after nonmaximum suppression and cross-modal verification, the final cargo instance-level segmentation result is output.

3. The cargo instance segmentation and nine-dimensional pose estimation method according to claim 2, characterized in that, The method involves fusing the image instance segmentation result and the point cloud instance segmentation result through a cross-modal attention fusion mechanism to obtain fused features, including: Image features are extracted from the image instance segmentation results, and point cloud features are extracted from the point cloud instance segmentation results; The image features and the point cloud features are concatenated and input into a sub-network consisting of global pooling and fully connected layers for feature analysis to generate dynamic weights. The dynamic weights are normalized using Softmax to output image feature weights, point cloud feature weights, and interaction feature weights. A cross-attention mechanism is used to generate interactive features by using image features as queries and point cloud features as keys and values. Substitute the image features, point cloud features, image feature weights, point cloud feature weights, interactive features, and interactive feature weights into the fusion formula to generate fused features. The fusion formula is as follows: F_inter = ； ； in, As a feature of fusion, Image feature weights, For point cloud feature weights, For interactive feature weights, For image features, Point cloud features F_inter For interactive features.

4. The method of cargo instance segmentation and nine-dimensional pose estimation of claim 1, wherein, The step of mapping the color information of the RGB image onto the LiDAR point cloud data corresponding to the cargo instance-level segmentation result according to the calibration parameters, to generate a color point cloud for each cargo instance, includes: Based on the calibration parameters, the three-dimensional points in the LiDAR point cloud data corresponding to the cargo instance-level segmentation result are transformed from the LiDAR coordinate system to the camera coordinate system, and then projected onto the pixel plane of the RGB image to obtain the pixel coordinates corresponding to the three-dimensional points. For each pixel coordinate, a color value is calculated from the RGB image using bilinear interpolation; The color values ​​are assigned to the corresponding 3D points to generate a color point cloud for each cargo instance.

5. The method of cargo instance segmentation and nine-dimensional pose estimation of claim 1, wherein, After projecting the colored point cloud onto a two-dimensional plane composed of V1 and V2, and preserving the color information of the three-dimensional points to generate a textured two-dimensional projection image, the process further includes: When the texture direction feature discrimination of the two-dimensional projection image is lower than a preset threshold, the two-dimensional projection image is input into a pre-trained lightweight convolutional neural network. A continuous angular offset is calculated by regression using the lightweight convolutional neural network. The initial orientation estimate obtained based on principal component analysis is fine-tuned by the angular offset to obtain the three-dimensional orientation of the cargo.

6. The method of cargo instance segmentation and nine-dimensional pose estimation of claim 1, wherein, After matching the initial size estimate with a pre-stored library of standard logistics box size templates, the process also includes: If the matching fails, the main color of the colored point cloud is calculated, and the color distribution of the three-dimensional points in the boundary region of the colored point cloud along each coordinate axis is analyzed. When the difference in color distribution exceeds the color difference threshold, it is determined to be an occluded boundary. The initial size estimate is optimized by virtually expanding the bounding box boundary until the predicted color changes abruptly, and the three-dimensional size of the cargo is output.

7. A system for cargo instance segmentation and nine-dimensional pose estimation, the system comprising: include: The calibration module is used to perform spatial synchronous calibration of the camera and lidar used for data acquisition, and to obtain calibration parameters including the sensor's extrinsic and intrinsic parameters. The data acquisition module is used to simultaneously acquire RGB images of the target scene and LiDAR point cloud data through the calibrated camera and LiDAR. The instance segmentation module is used to perform instance segmentation on the RGB image and the LiDAR point cloud respectively, and to fuse the semantic and geometric features of the RGB image and the LiDAR point cloud data through a cross-modal feature fusion mechanism to output the cargo instance-level segmentation result; The color point cloud generation module is used to map the color information of the RGB image onto the lidar point cloud data corresponding to the cargo instance-level segmentation result according to the calibration parameters, so as to generate a color point cloud for each cargo instance. The pose estimation module is used to perform geometric processing on the colored point cloud by fusing color information, and output the nine-dimensional pose parameters of the cargo instance-level segmentation result. The nine-dimensional pose parameters include three-dimensional position coordinates, cargo three-dimensional orientation and cargo three-dimensional dimensions. The geometric processing of the color point cloud, which fuses color information, to output three-dimensional position coordinates includes: Calculate the average geometric coordinates of all three-dimensional points in the color point cloud to obtain the geometric centroid; The color difference value is obtained by calculating the Euclidean distance between the color value of each three-dimensional point in the color point cloud and the average color at the geometric centroid. Based on the color difference value, a weight is assigned to each 3D point using a Gaussian function; the smaller the color difference value, the higher the weight of the 3D point. The geometric coordinates of each three-dimensional point are weighted and averaged according to the weights to calculate the three-dimensional position coordinates of the cargo instance. The color point cloud is geometrically processed to fuse color information, and the resulting three-dimensional orientation of the cargo is output, including: Principal component analysis is performed on the geometric coordinates of the colored point cloud to obtain three orthogonal principal direction vectors, and an initial orientation estimate is determined based on the principal direction vectors; the principal direction vectors are V1, V2, and V3, where V1 is greater than V2 and V3. The colored point cloud is projected onto a two-dimensional plane composed of V1 and V2, and the color information of the three-dimensional points is preserved to generate a textured two-dimensional projection image. Extract the orientation sensitivity features of the two-dimensional projection image; Based on the orientation sensitivity feature, the first similarity between the features of the two-dimensional projection image in the 0-degree rotation state and the features of the predefined positive template, and the second similarity between the features of the image in the 180-degree rotation state and the features of the predefined positive template are calculated respectively. Compare the first similarity with the second similarity, and select the rotation state direction corresponding to the higher value of the first similarity and the second similarity as the direction to eliminate ambiguity; Based on the disambiguation direction and the initial orientation estimate, output the three-dimensional orientation of the cargo; The color point cloud is subjected to geometric processing that fuses color information, and the output cargo three-dimensional dimensions include: Based on the object coordinate system determined by the three-dimensional orientation of the cargo, the axis-aligned bounding box of the colored point cloud is calculated to obtain an initial size estimate; The initial size estimate is matched with a pre-stored library of standard logistics box size templates; If a match is successful, the matched template size will be used as the three-dimensional size of the goods.

Images