Multi-object overlap detection method and system based on image and point cloud fusion

Abstract

The application discloses a multi-object overlapping detection method and system based on image and point cloud fusion, and the detection method comprises the following steps: acquiring an initial image and an initial point cloud; detecting the initial image to obtain a first detection result; matching the first detection result with the initial point cloud to obtain an alignment matrix; projecting the initial point cloud onto a corresponding image by using the alignment matrix to obtain a depth image; and prewarning object overlapping according to the depth image and the first detection result. The application adds depth detection after visual judgment and performs secondary detection, so that the deficiency of two-dimensional information is effectively made up. The method not only improves the detection accuracy, but also enhances the processing capacity for object overlapping and occlusion in a complex environment, and provides a more robust solution for various application scenarios.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of object overlap detection, and particularly relates to a multi-object overlap detection method and system based on image and point cloud fusion. BACKGROUND

[0002] In robot grasping or logistics automation, a single visual system is difficult to accurately determine the front and back positions of objects, often leading to reduced grasping efficiency and accuracy. This is mainly because the depth information provided by two-dimensional images is limited, making it difficult to effectively deal with object occlusion or overlap. In addition, in unmanned aerial vehicle inspection and remote sensing image analysis, a single visual system is extremely sensitive to light changes and dynamic scenes, which can easily lead to the failure of feature point extraction and matching, thereby affecting the detection accuracy and system robustness. This instability also affects the stitching of high-resolution maps and the accurate detection of repeated coverage areas in satellite images. In wildlife protection and vegetation coverage monitoring, a single visual detection system is prone to misjudgment due to occlusion and overlap. Especially in complex natural environments, two-dimensional information cannot effectively distinguish the front and back relationships of objects, leading to deviations in monitoring species activity tracks or vegetation coverage.

[0003] In summary, the traditional single visual sensor method is limited by two-dimensional information, and the uncertainty caused by depth errors cannot be ignored. SUMMARY

[0004] To solve the above technical problems, the present application provides a multi-object overlap detection method and system based on image and point cloud fusion, which adds depth detection after visual judgment for secondary detection, effectively making up for the shortcomings of two-dimensional information. This method not only improves the accuracy of detection, but also enhances the processing capability of object overlap and occlusion in complex environments, providing a more robust solution for various application scenarios.

[0005] The present application provides a multi-object overlap detection method based on image and point cloud fusion, comprising:

[0006] obtaining an initial image and an initial point cloud;

[0007] detecting the initial image to obtain a first detection result;

[0008] matching the first detection result and the initial point cloud to obtain an alignment matrix;

[0009] projecting the initial point cloud onto the corresponding image using the alignment matrix to obtain a depth image;

[0010] warning of object overlap according to the depth image and the first detection result.

[0011] Optionally, the initial image is detected to obtain a first detection result, comprising:

[0012] The initial image is input into an improved YOLOv8 model to obtain a first detection result, wherein the improved YOLOv8 model is obtained by training a training set, and the training set is labeled image data containing object overlap.

[0013] Optionally, the improved YOLOv8 model comprises an input module, a backbone module, an auxiliary module and a prediction module.

[0014] The input module is configured to scale the initial image to a uniform size to obtain an input image.

[0015] The backbone module is configured to extract image features of the input image.

[0016] The auxiliary module is configured to perform a sampling connection operation on the image features to fuse the image features.

[0017] The prediction module is configured to predict the first detection result according to the fused image features.

[0018] Optionally, extracting image features of the input image comprises:

[0019] The input image is divided into a plurality of data feature vectors.

[0020] The plurality of data feature vectors are added to obtain a feature vector.

[0021] The feature vector is processed to obtain the image features.

[0022] Optionally, the first detection result comprises: a target overlap image and a target non-overlap image.

[0023] Optionally, matching the first detection result with the initial point cloud to obtain an alignment matrix comprises:

[0024] 2D-3D cross-modal features are obtained according to the target overlap image and the initial point cloud.

[0025] The 2D-3D cross-modal features are fused, and the fused features are intersected detected.

[0026] According to the result of the intersection detection, abnormal features are removed to obtain the alignment matrix.

[0027] Optionally, obtaining 2D-3D cross-modal features according to the target overlap image and the initial point cloud comprises:

[0028] Obtaining pixel features of the target overlap image by using an upsampling module and a skip connection;

[0029] Dividing the initial point cloud into a uniform voxel grid, and distributing each point in the initial point cloud to a corresponding voxel according to three-dimensional coordinates, retaining voxels containing at least one point, and obtaining sparse voxels;

[0030] Performing sparse convolution on the sparse voxels to obtain voxel features;

[0031] Converting the voxel features into first point-level features through feature conversion;

[0032] Directly obtaining second point-level features of the initial point cloud by using a PointNet++ network model;

[0033] Adding the first point-level features and the second point-level features to obtain third point-level features;

[0034] Obtaining 2D-3D cross-modal features according to the third point-level features and the pixel features.

[0035] Optionally, projecting the initial point cloud onto a corresponding image by using the alignment matrix to obtain a depth image includes:

[0036] Extracting three-dimensional coordinates of the initial point cloud, and transforming the three-dimensional coordinates by using the alignment matrix to obtain pixel positions corresponding to the three-dimensional coordinates;

[0037] Recording depth values of each point cloud according to the pixel positions;

[0038] Integrating the pixel positions and the depth values to obtain the depth image.

[0039] Optionally, prewarning an object overlap according to the depth image and a first detection result includes:

[0040] Extracting depth values of overlapping objects in the target overlap image, and calculating average depth values of the overlapping objects respectively;

[0041] Comparing differences of the average depth values, and prewarning the overlapping objects according to a comparison result.

[0042] The application also provides a multi-object overlap detection system based on image and point cloud fusion, which includes a visual overlap detection module, a point cloud and image accurate registration module, a depth map generation module and a prewarning module.

[0043] The visual overlap detection module is used for target detection on an image.

[0044] The point cloud and image accurate registration module is configured to match the point cloud and the image according to the detection result, and obtain an alignment matrix;

[0045] The depth map generation module is configured to generate a depth image according to the point cloud according to the alignment matrix;

[0046] The early warning module is configured to perform overlap early warning according to the depth image and the detection result.

[0047] Compared with the prior art, the present application has the following advantages and technical effects:

[0048] 1. The present application adds depth detection after visual judgment, performs secondary detection, effectively makes up for the deficiency of two-dimensional information. This method not only improves the accuracy of detection, but also enhances the processing ability of objects overlapping and occlusion in complex environment, providing a more robust solution for various application scenarios.

[0049] 2. The present application integrates YOLOv8 algorithm, which can significantly reduce the target range that needs further analysis, thereby reducing the computational complexity and resource consumption, and accurately detecting and segmenting multiple targets in the scene. Traditional pure visual algorithm is often affected by perspective, occlusion, light change and complex background, resulting in target misplacement, information loss or false detection. After adding three-dimensional distance information, these problems can be effectively solved. Three-dimensional distance information helps the system to more accurately judge the spatial relationship between objects, overcoming the limitations of pure visual methods lacking depth perception. In the environment with occlusion, light change and complex background, YOLOv8 algorithm shows stable detection and segmentation effect, and three-dimensional distance information further enhances the accuracy and robustness of the system in the case of multiple object overlapping and occlusion. Overall, the present application not only improves the anti-interference ability of the detection process, but also has good adaptability, providing a more efficient and robust solution for target detection and segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0050] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application and their description are used to explain the application and are not intended to limit the application. In the drawings:

[0051] Figure 1 is a flowchart of the multi-object overlapping detection method based on image and point cloud fusion of the embodiment of the present application;

[0052] Figure 2 is a labeling example diagram of the embodiment of the present application;

[0053] Figure 3 is a main network structure diagram in the improved YOLOv8 model of the embodiment of the present application;

[0054] Figure 4 is a schematic diagram of a path aggregation network in an improved YOLOv8 model of an embodiment of the present application;

[0055] Figure 5 is a schematic diagram of a tri-net for cross-modal learning of an embodiment of the present application;

[0056] Figure 6 is a schematic diagram of an intersection detection module of an embodiment of the present application;

[0057] Figure 7 is a schematic diagram of an end-to-end pose solving module of an embodiment of the present application;

[0058] Figure 8 is a structural diagram of a multi-object overlap detection system based on image and point cloud fusion of an embodiment of the present application. DETAILED DESCRIPTION

[0059] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0060] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0061] The present application proposes a multi-object overlap detection method based on image and point cloud fusion, as shown in Figure 1 , which specifically includes the following steps:

[0062] Obtain an initial image and an initial point cloud;

[0063] Detect the initial image to obtain a first detection result;

[0064] Match the first detection result and the initial point cloud to obtain an alignment matrix;

[0065] Project the initial point cloud onto the corresponding image using the alignment matrix to obtain a depth image;

[0066] According to the depth image and the first detection result, a warning is given for object overlap.

[0067] Specifically, the present application mainly involves four steps. The first step is to use the improved YOLOv8 algorithm for target detection and segmentation, and to preliminarily warn two targets with overlapping areas. The second step is to use the VP2P-based algorithm to match the point cloud and the image, and then generate an alignment matrix. The third step is to project the point cloud onto the corresponding image according to the matrix generated by the VP2P-based algorithm, and generate a depth image according to the depth information of the point cloud. The fourth step is to extract the depth information of the corresponding bounding box area from the depth image for the visually overlapping modules warned by the first module, calculate the average depth of the two objects respectively, and set different warning thresholds according to different requirements. If the difference between the depth value of object A and the depth value of object B is less than the warning threshold, a warning will be given.

[0068] Further, the initial image is detected to obtain a first detection result, including:

[0069] The initial image is input into the improved YOLOv8 model to obtain the first detection result, wherein the improved YOLOv8 model is obtained by training a training set, and the training set is labeled image data containing object overlap.

[0070] Specifically, in this step, the improved YOLOv8 algorithm is used to detect and segment the required target in the image. The improved YOLOv8 algorithm can effectively identify multiple targets in the image and distinguish them by category and contour. This algorithm can generate corresponding semantic segmentation results according to different categories of targets and accurately mark the boundaries of each target object. No matter how complex or variable the image background is, the entire detection and segmentation process can ensure high accuracy. This process can adapt to different scenes and object categories, ensuring accurate detection of the required target in a complex background.

[0071] In addition, this step specially integrates a visual overlap preliminary detection function, which is used to judge the overlap between objects in the image. This function analyzes the relative position relationship of objects in the two-dimensional plane to preliminarily detect whether there is occlusion or overlap phenomenon.

[0072] Suppose the bounding box of the i-th target object is B i =(x i ,y i ,w i ,h i ), then the overlapping area between the two objects is If the intersection-over-union IoU(B i , B j ) exceeds a certain threshold T, it is preliminarily considered that there is an overlap phenomenon. The system will enter the subsequent more in-depth detection process, and the depth information in the three-dimensional point cloud data will be introduced to detect the overlapping area with higher accuracy.

[0073] Further, the improved YOLOv8 model comprises an input module, a backbone module, an auxiliary module and a prediction module;

[0074] The input module is configured to scale the initial image to a uniform size and obtain an input image;

[0075] The backbone module is configured to extract image features of the input image;

[0076] The auxiliary module is configured to perform a sampling connection operation on the image features and fuse the image features;

[0077] The prediction module is configured to predict a first detection result based on the fused image features.

[0078] Further, the extraction of the image features of the input image comprises:

[0079] The input image is divided into a plurality of data feature vectors;

[0080] The plurality of data feature vectors are added to obtain a feature vector;

[0081] The feature vector is processed to obtain the image features.

[0082] Further, the first detection result comprises a target overlapping image and a target non-overlapping image.

[0083] Specifically, if the visual detection does not find overlapping, the system will not need to continue the subsequent complex operation, thereby effectively saving the computing resources and improving the overall operation efficiency. Through such a design, the system optimizes the use of computing power while ensuring accuracy, making it more efficient.

[0084] Further, the matching of the first detection result and the initial point cloud to obtain an alignment matrix comprises:

[0085] 2D-3D cross-modal features are obtained based on the target overlapping image and the initial point cloud;

[0086] Feature fusion is performed on the 2D-3D cross-modal features, and intersection detection is performed on the fused features;

[0087] Based on the result of the intersection detection, abnormal features are removed to obtain the alignment matrix.

[0088] Specifically, the core task of this step is to accurately register the three-dimensional point cloud data and the two-dimensional image based on the VP2P algorithm. The VP2P algorithm can process data generated by different source sensors and perform spatial matching and fusion on them, so that the three-dimensional point cloud and the two-dimensional image data are seamlessly connected in the same coordinate system.

[0089] In this registration process, the system will calculate the best alignment matrix according to the pixel information in the image and the three-dimensional coordinate points in the point cloud. This alignment matrix not only ensures the accurate correspondence of point cloud data and image data in geometric space, but also provides a basis for subsequent steps of point cloud projection and depth map generation. Through high-precision registration, the system can accurately determine the relative positions of multiple objects in complex scenes, avoiding recognition errors caused by coordinate misplacement or mismatch.

[0090] Further, the 2D-3D cross-modal feature is obtained according to the target overlapping image and the initial point cloud, comprising:

[0091] The pixel feature of the target overlapping image is obtained by using the upsampling module and the skip connection;

[0092] The initial point cloud is divided into a uniform voxel grid, and each point in the initial point cloud is assigned to the corresponding voxel according to the three-dimensional coordinates, and the voxels containing at least one point are retained to obtain a sparse voxel;

[0093] The sparse voxel is subjected to sparse convolution to obtain voxel features;

[0094] The voxel features are converted into first point-level features through feature conversion;

[0095] The second point-level features of the initial point cloud are directly obtained by using the PointNet++ network model;

[0096] The first point-level features and the second point-level features are added element by element to obtain third point-level features;

[0097] The 2D-3D cross-modal feature is obtained according to the third point-level features and the pixel features.

[0098] Further, the initial point cloud is projected onto the corresponding image by using the alignment matrix to obtain a depth image, comprising:

[0099] The three-dimensional coordinates of the initial point cloud are extracted, and the three-dimensional coordinates are transformed using the alignment matrix to obtain the pixel positions corresponding to the three-dimensional coordinates;

[0100] According to the pixel positions, the depth values of each point cloud are recorded;

[0101] The pixel positions and depth values are integrated to obtain a depth image.

[0102] Specifically, after the registration of the point cloud and the image is completed, the system will enter this module, i.e. projecting the three-dimensional point cloud data onto the corresponding two-dimensional image plane. Through this projection process, the system can generate a high-precision depth map reflecting the relative depth between objects. The three-dimensional coordinates of the point cloud data are accurately mapped onto the two-dimensional image, so that each pixel point can obtain its corresponding depth information.

[0103] The generated depth map can not only reflect the relative position and distance between objects, but also capture the subtle depth differences of objects in a complex scene. These information plays a crucial role in subsequent detection and warning, especially in scenes with complex occlusion and overlap, the three-dimensional spatial information provided by the depth map can effectively help the system to judge the front and back relationship and the real physical distribution of the objects. This module provides important data support for the overall accuracy of the system.

[0104] Further, according to the depth image and the first detection result, the object overlap is warned, comprising:

[0105] Extracting the depth values of the overlapping objects in the target overlap image, and calculating the average depth values of the overlapping objects respectively;

[0106] Comparing the difference of the average depth values, and according to the comparison result, the overlapping objects are warned.

[0107] Specifically, this step is the core detection link of the whole system, which is specially used for further accurate detection and warning of the overlap of objects after the preliminary detection is completed and the preliminary warning is triggered. At this time, the system will analyze the depth map generated by the third module, especially for the object area which has been found to overlap in the preliminary detection process, and further overlap detection is carried out.

[0108] In this step, the system will calculate the average depth value of the object through the depth information, and further refine the preliminary judgment. Assuming that the average depth of two objects is Z i and Z j , if the depth difference |Z i -Z j | is less than the preset threshold value ΔZ, the system will trigger the accurate warning, indicating that there may be actual overlap or occlusion between the objects. Through this secondary overlap detection, the system can significantly improve the judgment accuracy of the relative position of the objects in complex scenes, and avoid false warnings due to the accumulation of errors in the preliminary detection.

[0109] In addition, this step also has high flexibility, which can adjust the warning threshold according to the needs of different application scenarios. For example, in the robot grabbing task, the sensitivity of the accurate warning can be adjusted according to the grabbing needs of the objects; while in the remote sensing image analysis, the system can set more rigorous warning standards according to different depth information. This flexible warning mechanism enables the system to cope with various complex environments and ensure high-precision and high-robustness detection results.

[0110] The application also provides a multi-object overlap detection system based on image and point cloud fusion, which comprises: Figure 8As shown, specifically includes: visual overlap detection module, point cloud and image accurate registration module, depth map generation module and early warning module;

[0111] The visual overlap detection module is configured to perform target detection on the image.

[0112] The point cloud and image accurate registration module is configured to match the point cloud and the image according to the detection result to obtain an alignment matrix.

[0113] The depth map generation module is configured to generate a depth image according to the point cloud according to the alignment matrix.

[0114] The early warning module is configured to perform overlap early warning according to the depth image and the detection result.

[0115] The embodiment will be described in detail below with reference to the accompanying drawings:

[0116] Step one: target detection and segmentation and visual overlap preliminary detection

[0117] Before model training, first, the labelimg tool needs to be used to label the target in the image. Open labelimg, import the image to be labeled, use the rectangular frame tool to mark each target object, and specify the class label for each frame. The labeling example is shown in Figure 2 .

[0118] The embodiment proposes to use an improved YOLOv8 network for 2D image target detection and instance segmentation.

[0119] In this embodiment, the improved YOLOv8 module mainly improves the performance by optimizing the backbone network and the up-sampling module. First, the original YOLOv8 backbone network is replaced by a PoolFormer backbone network with stronger feature extraction capability. PoolFormer uses a simplified non-parametric pooling operation, which enhances the extraction of key features in the image and reduces the parameter amount of the network. This improvement enables the model to more accurately segment the edge area of stacked parts, reduces the misidentification of the background, and improves the segmentation accuracy.

[0120] In the up-sampling module, the simple nearest neighbor interpolation method originally used by YOLOv8 is replaced by a more powerful CARAFE up-sampling module. The CARAFE module uses a larger receptive field and more detailed feature reconstruction to more effectively utilize the semantic information of the image, improving the accuracy and detail processing capability of target segmentation. After these two optimizations, the segmentation and detection accuracy of the model is significantly improved, especially in complex stacked part recognition, showing stronger robustness while maintaining real-time processing capability.

[0121] The improved YOLOv8 mainly consists of an input layer, a backbone framework, an auxiliary framework, and a prediction framework.

[0122] Input layer: The image to be detected is scaled to a uniform size by data enhancement and adaptive image scaling methods.

[0123] The main framework is as shown in Figure 3 The embodiment proposes that the main framework extracts image features using the self-attention mechanism of PoolFormer. First, the input image I is divided into N block data feature vectors, as shown in the following formula:

[0124] X = InputEmb(I)

[0125] where X is the feature vector, X ∈ R N*C , N is the sequence length, and C is the feature dimension. X is input into two residual sub-blocks.

[0126] The first sub-block is composed of normalization (Norm) and pooling (Pool) modules, and the result is added to the input X to obtain the feature Y:

[0127] Y = Pool(Norm(X)) + X

[0128] The second sub-block is composed of normalization and multi-layer perceptron modules, which are composed of normalization, linear transformation and nonlinear activation operations with Y to form the final output Z:

[0129] Y = σ(Norm(Y)W1)W2 + Y

[0130] where W1 and W2 are learnable parameters in the multi-layer perceptron, and σ is the activation function. Optionally, the main framework of the embodiment can be stacked by 4 PoolFormer blocks, and the stacking ratio is [4, 4, 12, 4].

[0131] Auxiliary framework: A module composed of path aggregation network (PAN) can be used to sample and connect the features extracted by the main framework, realizing the feature fusion of the security picture. In Figure 3 , the C2f module mainly increases the depth and receptive field of the network, improves the feature extraction capability, and CBS is mainly composed of convolution, batch normalization and activation function, which are respectively used for feature extraction, output normalization and enhancement of the nonlinear ability of the model. Whether to parameter.

[0132] The path aggregation network used in the embodiment is shown in Figure 4As shown, the path aggregation network (PAN) module adopts a bottom-up enhanced path to shorten the information transmission path, and stores the precise positioning signal in the low-level features, and improves the structure of the feature pyramid. In the enhancement process, three different levels of features are enhanced by shortening the path {M2, M3, M4} to {N2, N3, N4}. After 3*3 convolution operation, the features of different levels are added to the path information to obtain the feature enhancement process of different levels:

[0133]

[0134] wherein, represents the feature addition.

[0135] Prediction framework: responsible for generating the final detection results. This framework includes three main outputs, which are boundary box regression, confidence score, and class probability. Boundary box regression is used to predict the position and size of the target. Usually, four values are output, corresponding to the center coordinates (x, y) and width, height (w, h) of the boundary box. Confidence score predicts whether each boundary box contains a target and the confidence of the target. Class probability is used to predict the probability of the target belonging to each class.

[0136] Then create a folder named mydata in the data directory in the YOLOv8 folder, and put the annotated xml files and images into the corresponding directories. The images folder stores the images, and the xml folder stores the corresponding xml files of the images. Then generate train.txt, val.txt, test.txt and trainval.txt four files in the Main folder and store them in the dataSet folder. Then start training.

[0137] After training, the system uses the YOLOv8 model to perform target detection and segmentation on the input image. Through YOLOv8 inference, the boundary box and class information of each detected target are obtained, and the results are output, including segmentation mask (if applicable) and the precise boundary of each target. No matter how complex or variable the image background is, the entire detection and segmentation process can ensure high accuracy.

[0138] On this basis, the system integrates a visual overlap preliminary detection function. The function can analyze the spatial relationship of the detected targets, and judge whether there is an overlap or occlusion phenomenon between the targets in the image. The visual overlap preliminary detection mainly analyzes the relative position of the object in the two-dimensional image, identifies the boundary overlap region, and generates a preliminary overlap judgment result. If it is detected that there is an overlap between the targets, the system will further enter the depth analysis module for more accurate three-dimensional point cloud depth information analysis; on the contrary, if the visual preliminary detection does not find overlap, the system will skip the subsequent process to avoid unnecessary calculation, save resources and improve processing efficiency.

[0139] Step two: accurate registration of point cloud and image

[0140] The embodiment is based on a differentiable registration method of VP2P image and radar point cloud when realizing the registration of point cloud and image.

[0141] VP2P is a multi-modal learning framework, mainly composed of a cross-modal learning part, an intersection detection part, and an end-to-end pose solving part.

[0142] The cross-modal learning part is as shown in Figure 5 The cross-modal learning part: taking a pair of incorrectly registered image and point cloud as input, using a U-shaped convolutional network for image feature extraction for the pixel branch, and using a residual network for the base layer of the U-shaped network to obtain the feature f of each pixel through the upsampling module and the jump connection. pixel ∈R H×W×C .

[0143] For the input point cloud, the three-dimensional space is divided into a uniform voxel grid according to the size, each point in the point cloud is assigned to the corresponding voxel according to its three-dimensional coordinates, and then only the voxels containing at least one point are retained, and the empty voxels are not stored. Sparse convolution operation is performed on the sparse voxels to obtain voxel features f voxel ∈R N×C .

[0144] The original point cloud is also input into the PointNet++ based network, and the self-attention module is used to perform hierarchical set abstraction to obtain global features f and further apply the feature propagation module to learn the point-by-point feature f point ∈R N×C .

[0145] Then the voxel-level feature f voxel ∈R N×C is converted into a point-level feature through feature conversion, and is added element by element with the point set feature directly extracted from the point cloud, and the result is further subjected to adaptive weighted loss with the pixel-level feature.

[0146] The intersection detection module is as shown inFigure 6 The intersection detection part is composed of four subparts: feature fusion, probability prediction, abnormal region removal, and loss function.

[0147] Feature fusion is to fuse the global 2D feature g 2D extracted in the cross-modal learning part with the feature f 3D of each 3D point to form a combined feature vector [g 2D : f 3D ].

[0148] Probability prediction uses a multi-layer perceptron to learn the combined feature vector, outputting the probability of each 3D point being in the 2D-3D intersection region Similarly, for the abnormal region of the 2D image space, the probability of each pixel point being in the intersection region is predicted

[0149] In the abnormal region removal, a probability threshold σ is defined, and 2D or 3D elements with a probability lower than σ are considered as outliers and removed.

[0150] Finally, Z in and Z out are sampled from the intersection region and the abnormal region respectively.

[0151]

[0152] The end-to-end pose solving part is shown in Figure 7 To align the point cloud and the image, the distance information of the three-dimensional point cloud is used for secondary judgment. The end-to-end pose solving part searches for the point with the highest similarity to the feature of the remaining 3D points for each 2D pixel point, estimates the gradient using Gumbel Softmax, and ensures the differentiability during training.

[0153] Then, PnP is solved in the form of probability distribution, the output is defined as the probability distribution p(T|X), and the KL divergence loss is used for optimization:

[0154]

[0155] In the inference stage, to improve efficiency, the EPnP method with a time complexity of O(n) is used, and the RANSAC algorithm is combined for more robust pose estimation.

[0156] Finally, the Gauss-Newton method is used to further solve the accurate pose

[0157] T' = [R'|t'], and the pose loss is calculated:

[0158]

[0159] If overlap is detected in the first module, the system will enter the precise registration phase of the point cloud and image. In this step, the image and point cloud data are input, and the system processes sparse voxels to generate layer sparse voxels V. Then, a ternary network is applied to extract features from the three modalities. Specifically, two-dimensional modalities are represented as pixel features, and three-dimensional modalities are represented as a combination of voxel and point features, and adaptive weighted loss is used to learn unique 2D-3D cross-modal patterns. After extracting features, the system performs cross-modal feature fusion to detect intersection regions in 2D and 3D spaces. According to the results of intersection detection, abnormal regions are removed, and 2D-3D feature matching is used to establish 2D-3D correspondence. If the intersection in the intersection detection is greater than the set threshold, the pose solution is linked, and if no overlap is detected in the visual detection, i.e., the intersection is less than the threshold, the pose solution is not linked.

[0160] Specifically, the system first acquires three-dimensional point cloud data in the scene and matches it with the corresponding two-dimensional image. Then, the VP2P algorithm is used to calculate the geometric transformation relationship between the point cloud data and the image, generating an alignment matrix. This matrix ensures that each three-dimensional point in the point cloud can be mapped to the corresponding pixel point in the image.

[0161] When performing this registration step, the system checks spatial consistency to ensure that the registration result accurately reflects the three-dimensional relationship between objects. Finally, the generated alignment matrix provides an important foundation for subsequent steps, enabling the system to obtain depth information for each target object on the two-dimensional image. This process ensures the accuracy and reliability of the data, laying a solid foundation for subsequent depth analysis and detection.

[0162] Step three: Point cloud projection and depth map generation

[0163] After completing the registration of point cloud and image through the second step, the system will enter the point cloud projection and depth map generation module. In this phase, the system uses the alignment matrix generated by registration to project the three-dimensional point cloud data onto the corresponding two-dimensional image.

[0164] In this process, the system iterates through each three-dimensional point cloud point, extracts its three-dimensional coordinates (x, y, z), and uses the alignment matrix to transform each three-dimensional coordinate to calculate its corresponding pixel position on the two-dimensional image. This process converts three-dimensional coordinates to homogeneous coordinates:

[0165]

[0166] In the formula for converting 3D coordinates to homogeneous coordinates, w is the homogeneity factor, which is usually set to 1, and the corresponding projection formula is applied to obtain the image coordinates (u,v):

[0167]

[0168] Where K is the intrinsic parameter matrix:

[0169]

[0170] f x ,f y These are the camera's focal lengths along the x and y axes, respectively, and c x ,c y R is the coordinate of the optical center of the image on the x and y axes, respectively. R is the rotation matrix, t is the translation vector, and I is a 3×3 identity matrix used to concatenate the translation vector and the rotation matrix into a 3×4 matrix.

[0171] Next, at the corresponding pixel location in the image, the system records the depth value Z of each point cloud point, forming a depth map. If multiple point cloud points are projected onto the same pixel location, the system selects the one with the smallest depth value to ensure that it represents the object closest to the camera.

[0172] After this process, the system integrates all processed pixel positions and their corresponding depth values ​​to generate a complete depth map. The value of each pixel in this depth map represents the relative depth of an object, reflecting the relative distance and three-dimensional spatial relationships between objects. Finally, the system performs a quality check on the generated depth map to ensure its accuracy, especially when multiple objects overlap. Visualization tools can be used to view the quality of the depth map and adjust parameters during the projection process accordingly. Through this series of steps, the system effectively generates depth maps, helping to accurately identify the relative depths between objects and ensuring the accuracy of subsequent detection.

[0173] Step 4: Secondary Overlap Detection and Precise Early Warning

[0174] After generating the depth map, the system enters the secondary overlap detection and precise early warning module. This module only starts after the initial visual overlap detection is successful and an initial early warning is triggered. The system first extracts the depth information corresponding to the overlapping areas found in the initial detection; this information will be used for subsequent depth analysis. In practice, the system filters points within the overlapping area, extracts the depth values ​​of two objects, and then calculates the average depth value of each object within that area. By comparing the difference in average depth between the two objects, the system can determine their actual 3D spatial relationship. If the difference in average depth values ​​is less than a preset early warning threshold, the system will further trigger a precise early warning, indicating that there is an actual overlap or occlusion between the objects.

[0175] In this case, the system outputs a warning signal to prompt the user or the system to handle or optimize the relationship between objects. In different application scenarios, the warning threshold can be adjusted according to actual needs to adapt to the accuracy requirements of different tasks. For example, in a robot grasping task, the system can set a higher precision warning sensitivity to ensure the safety and effectiveness of object grasping, while in remote sensing image stitching, a stricter warning standard can be set according to the depth information of different targets to improve the accuracy of stitching. Through these steps, the system can achieve accurate overlap detection and effective warning mechanism, providing important support for subsequent processing and decision-making.

[0176] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A multi-object overlap detection method based on image and point cloud fusion, characterized in that, The method comprises the following steps: acquiring an initial image and an initial point cloud; detecting the initial image to obtain a first detection result; the detection comprises: inputting the initial image into an improved YOLOv8 model to obtain the first detection result; the improved YOLOv8 model comprises an input module, a backbone module, an auxiliary module and a prediction module; the input module is used for scaling the initial image to a uniform size to obtain an input image; the backbone module is used for extracting image features of the input image, and the image features are extracted by using a self-attention mechanism of a PoolFormer; the auxiliary module is used for performing a sampling connection operation on the image features, fusing the image features, using a module composed of a path aggregation network (PAN) to perform a sampling connection operation on the features extracted by the backbone framework, and using a CARAFE up-sampling module in the up-sampling; the prediction module is used for predicting the first detection result according to the fused image features; the first detection result comprises a target overlapping image; matching the first detection result and the initial point cloud to obtain an alignment matrix; the matching comprises: acquiring 2D-3D cross-modal features according to the target overlapping image and the initial point cloud, which comprises: obtaining pixel features of the target overlapping image by using an up-sampling module and a jump connection; dividing the initial point cloud into uniform voxel grids, and assigning each point in the initial point cloud to a corresponding voxel according to three-dimensional coordinates to retain voxels containing at least one point and obtain sparse voxels; performing sparse convolution on the sparse voxels to obtain voxel features; converting the voxel features into first point-level features by feature conversion; directly obtaining second point-level features of the initial point cloud by using a PointNet++ network model; performing element addition on the first point-level features and the second point-level features to obtain third point-level features; and acquiring 2D-3D cross-modal features according to the third point-level features and the pixel features; performing feature fusion on the 2D-3D cross-modal features and performing intersection detection on the fused features; removing abnormal features according to the result of the intersection detection to obtain the alignment matrix; projecting the initial point cloud onto a corresponding image by using the alignment matrix to obtain a depth image; warning an object overlap according to the depth image and the first detection result. 2.The multi-object overlap detection method based on image and point cloud fusion according to claim 1, wherein, The detection of the initial image to obtain the first detection result comprises: The improved YOLOv8 model is obtained by training a training set, and the training set is labeled image data containing an object overlap. 3.The multi-object overlap detection method based on image and point cloud fusion according to claim 2, characterized in that, The extraction of the image features of the input image comprises: cutting the input image into a plurality of data feature vectors; adding the plurality of data feature vectors to obtain a feature vector; processing the feature vector to obtain the image features. 4.The multi-object overlap detection method based on image and point cloud fusion according to claim 1, wherein, The first detection result further comprises a target non-overlapping image. 5.The multi-object overlap detection method based on image and point cloud fusion according to claim 1, wherein, The projection of the initial point cloud onto a corresponding image by using the alignment matrix to obtain a depth image comprises: extracting three-dimensional coordinates of the initial point cloud, and transforming the three-dimensional coordinates by using the alignment matrix to obtain pixel positions corresponding to the three-dimensional coordinates. According to the pixel position, record the depth value of each point cloud; Integrate the pixel position and the depth value to obtain the depth image.

6. The method of claim 5, wherein the image-based and point cloud fusion-based multi-object overlap detection method further comprises: According to the depth image and the first detection result, the object overlap is pre-warned, including: Extracting the depth value of the overlapping object in the target overlap image, respectively calculating the average depth value of the overlapping object; Comparing the difference of the average depth value, and according to the comparison result, the overlapping object is pre-warned.

7. A multi-object overlap detection system based on image and point cloud fusion, for implementing the multi-object overlap detection method based on image and point cloud fusion according to any one of claims 1-6, characterized in that, Including: Visual overlap detection module, point cloud and image accurate registration module, depth map generation module and warning module; The visual overlap detection module is used for target detection of the image; The point cloud and image accurate registration module is used for matching the point cloud and the image according to the detection result to obtain an alignment matrix; The depth map generation module is used for generating a depth image according to the point cloud according to the alignment matrix; The warning module is used for overlap warning according to the depth image and the detection result.

Images