Multi-object overlapping detection method and system based on image and point cloud fusion

Abstract

The invention discloses a multi-object overlapping detection method and system based on image and point cloud fusion. The detection method comprises the following steps: acquiring an initial image and an initial point cloud; detecting the initial image to obtain a first detection result; matching the first detection result with the initial point cloud to obtain an alignment matrix; projecting the initial point cloud to a corresponding image by using the alignment matrix to obtain a depth image; and performing early warning on object overlapping according to the depth image and the first detection result. Depth detection is added after visual judgment, secondary detection is carried out, and the defect of two-dimensional information is effectively overcome. The method not only improves the accuracy of detection, but also enhances the capability of processing overlapping and shielding of objects in a complex environment, and provides a more robust solution for various application scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of object overlap detection, and in particular to a multi-object overlap detection method and system based on image and point cloud fusion. Background Art

[0002] In robot grasping or logistics automation, a single visual system has difficulty accurately judging the front and back positions of objects, often resulting in reduced grasping efficiency and accuracy. This is mainly because the depth information provided by two-dimensional images is limited, and it is difficult to effectively deal with situations where objects are occluded or overlapped. In addition, in drone inspections and remote sensing image analysis, a single visual system is extremely sensitive to lighting changes and dynamic scenes, which can easily lead to failures in feature point extraction and matching, thus affecting the accuracy of detection and the robustness of the system. This instability also affects the stitching of high-resolution maps and the accurate detection of repeated coverage areas in satellite images. In wildlife protection and vegetation coverage monitoring, a single visual detection system is prone to misjudgment due to occlusion and overlap problems. Especially in complex natural environments, two-dimensional information cannot effectively distinguish the front and back relationships of objects, resulting in deviations when monitoring species activity trajectories or vegetation coverage.

[0003] In summary, the traditional single vision sensor approach is constrained by the limitation of two-dimensional information, and the uncertainty caused by depth error is an issue that cannot be ignored. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a multi-object overlap detection method and system based on image and point cloud fusion, which effectively makes up for the lack of two-dimensional information by adding depth detection after visual judgment and performing secondary detection. This method not only improves the accuracy of detection, but also enhances the ability to handle overlapping and occluded objects in complex environments, providing a more robust solution for various application scenarios.

[0005] The present invention provides a multi-object overlapping detection method based on image and point cloud fusion, comprising:

[0006] Obtain initial image and initial point cloud;

[0007] Detecting the initial image to obtain a first detection result;

[0008] Matching the first detection result with the initial point cloud to obtain an alignment matrix;

[0009] Using the alignment matrix, projecting the initial point cloud onto a corresponding image to obtain a depth image;

[0010] An early warning is issued for object overlap based on the depth image and the first detection result.

[0011] Optionally, detecting the initial image to obtain a first detection result includes:

[0012] The initial image is input into an improved YOLOv8 model to obtain a first detection result, wherein the improved YOLOv8 model is obtained by training a training set, and the training set is annotated image data containing overlapping objects.

[0013] Optionally, the improved YOLOv8 model includes an input module, a backbone module, an auxiliary module and a prediction module;

[0014] The input module is used to scale the initial image to a uniform size to obtain an input image;

[0015] The backbone module is used to extract image features of the input image;

[0016] The auxiliary module is used to perform sampling and connection operations on the image features to fuse the image features;

[0017] The prediction module is used to predict the first detection result according to the fused image features.

[0018] Optionally, extracting image features of the input image includes:

[0019] Dividing the input image into a plurality of data feature vectors;

[0020] Adding the plurality of data feature vectors to obtain a feature vector;

[0021] The feature vector is processed to obtain the image feature.

[0022] Optionally, the first detection result includes: a target overlapping image and a target non-overlapping image.

[0023] Optionally, matching the first detection result with the initial point cloud to obtain an alignment matrix includes:

[0024] Acquire 2D-3D cross-modal features according to the target overlapping image and the initial point cloud;

[0025] Performing feature fusion on the 2D-3D cross-modal features, and performing intersection detection on the fused features;

[0026] According to the result of the intersection detection, abnormal features are removed to obtain the alignment matrix.

[0027] Optionally, acquiring 2D-3D cross-modal features according to the target overlapping image and the initial point cloud includes:

[0028] Obtaining pixel features of the target overlapping image using an upsampling module and a skip connection;

[0029] Dividing the initial point cloud into a uniform voxel grid, and assigning each point in the initial point cloud to a corresponding voxel according to the three-dimensional coordinates, retaining the voxel containing at least one point, and obtaining sparse voxels;

[0030] Performing sparse convolution on the sparse voxels to obtain voxel features;

[0031] Converting the voxel feature into a first point-level feature through feature conversion;

[0032] Using the PointNet++ network model, directly obtaining the second point-level features of the initial point cloud;

[0033] Performing element-wise addition on the first point-level feature and the second point-level feature to obtain a third point-level feature;

[0034] A 2D-3D cross-modal feature is obtained according to the third point-level feature and the pixel feature.

[0035] Optionally, using the alignment matrix to project the initial point cloud onto a corresponding image, and obtaining a depth image includes:

[0036] Extracting the three-dimensional coordinates of the initial point cloud, transforming the three-dimensional coordinates using the alignment matrix, and obtaining pixel positions corresponding to the three-dimensional coordinates;

[0037] According to the pixel position, record the depth value of each point cloud;

[0038] The pixel position and the depth value are integrated to obtain the depth image.

[0039] Optionally, issuing an early warning for object overlap according to the depth image and the first detection result includes:

[0040] Extracting the depth values ​​of the overlapping objects in the target overlapping image, and calculating the average depth values ​​of the overlapping objects respectively;

[0041] The differences in the average depth values ​​are compared, and based on the comparison result, an early warning is issued for the overlapping objects.

[0042] The present invention also provides a multi-object overlap detection system based on image and point cloud fusion, including: a visual overlap detection module, a point cloud and image precise registration module, a depth map generation module and an early warning module;

[0043] The visual overlap detection module is used to perform target detection on the image;

[0044] The point cloud and image precise registration module is used to match the point cloud and the image according to the detection results to obtain the alignment matrix;

[0045] The depth map generation module is used to generate a depth image according to the point cloud based on the alignment matrix;

[0046] The warning module is used to issue overlapping warnings based on the depth image and the detection results.

[0047] Compared with the prior art, the present invention has the following advantages and technical effects:

[0048] 1. The present invention adds depth detection after visual judgment to perform secondary detection, which effectively makes up for the lack of two-dimensional information. This method not only improves the accuracy of detection, but also enhances the ability to handle overlapping and occluded objects in complex environments, providing a more robust solution for various application scenarios.

[0049] 2. By integrating the YOLOv8 algorithm, the present invention can significantly reduce the scope of targets that need further analysis, thereby reducing computational complexity and resource consumption, while accurately detecting and segmenting multiple targets in the scene. Traditional pure visual algorithms are often affected by perspective, occlusion, illumination changes and complex backgrounds, resulting in target dislocation, information loss or misdetection, and these problems can be effectively solved by adding three-dimensional distance information. Three-dimensional distance information helps the system to more accurately judge the spatial relationship between objects, overcoming the limitation of the pure visual method that lacks depth perception. In an environment with occlusion, illumination changes and complex backgrounds, the YOLOv8 algorithm demonstrates stable detection and segmentation effects, while the three-dimensional distance information further enhances the accuracy and robustness of the system in the case of multiple object overlap and occlusion. Overall, the present invention not only improves the anti-interference ability of the detection process, but also has good adaptability, providing a more efficient and robust solution for target detection and segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0051] Figure 1 is a flow chart of a multi-object overlapping detection method based on image and point cloud fusion according to an embodiment of the present invention;

[0052] Figure 2 is a labeled example diagram of an embodiment of the present invention;

[0053] Figure 3 is a diagram of the backbone network structure in the improved YOLOv8 model of an embodiment of the present invention;

[0054] Figure 4 is a schematic diagram of a path aggregation network in an improved YOLOv8 model according to an embodiment of the present invention;

[0055] Figure 5 is a schematic diagram of a ternary network for cross-modal learning according to an embodiment of the present invention;

[0056] Figure 6 is a schematic diagram of an intersection detection module according to an embodiment of the present invention;

[0057] Figure 7 is a schematic diagram of an end-to-end posture solution module in an embodiment of the present invention;

[0058] Figure 8 It is a structural diagram of a multi-object overlapping detection system based on image and point cloud fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0060] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0061] The present invention proposes a multi-object overlapping detection method based on image and point cloud fusion, such as Figure 1 As shown, the specific steps include:

[0062] Obtain initial image and initial point cloud;

[0063] Detecting the initial image to obtain a first detection result;

[0064] Match the first detection result with the initial point cloud to obtain an alignment matrix;

[0065] Using the alignment matrix, the initial point cloud is projected onto the corresponding image to obtain the depth image;

[0066] According to the depth image and the first detection result, an early warning is issued for object overlap.

[0067] Specifically, the present invention mainly involves four steps. The first step is to use the improved YOLOv8 algorithm to detect and segment the target, and to issue a preliminary warning for two targets with overlapping areas. The second step is to use the VP2P-based algorithm to match the point cloud and the image, and then generate an alignment matrix. The third step is to project the point cloud onto the corresponding image according to the moment generated by the VP2P-based algorithm, and generate a depth image according to the depth information of the point cloud. The fourth step is to extract the depth information of the corresponding bounding box area from the depth map for the visually overlapping modules warned by the first module, calculate the average depth of the two objects respectively, set different warning thresholds according to different needs, and issue a warning if the difference between the depth value of object A and the depth value of object B is less than the warning threshold.

[0068] Further, detecting the initial image to obtain a first detection result includes:

[0069] The initial image is input into the improved YOLOv8 model to obtain a first detection result, wherein the improved YOLOv8 model is obtained by training with a training set, and the training set is annotated image data containing overlapping objects.

[0070] Specifically, in this step, the improved YOLOv8 algorithm is used to detect and segment the required targets in the image. The improved YOLOv8 algorithm can effectively identify multiple targets in the image and distinguish them by category and contour. The algorithm can generate corresponding semantic segmentation results based on targets of different categories and accurately calibrate the boundaries of each target object. No matter how complex or changing the image background is, the entire detection and segmentation process can ensure high accuracy. This process can adapt to different scenes and object categories, ensuring that the required targets can be accurately detected even in complex backgrounds.

[0071] In addition, this step specially integrates the visual overlap initial detection function, which is specifically used to determine the overlap between objects in the image. This function preliminarily detects whether there is occlusion or overlap by analyzing the relative position relationship of the objects on the two-dimensional plane.

[0072] Assume that the bounding box of the i-th target object is B i =(x i ,y i ,w i ,h i ), then the overlap area between the two objects is If the intersection over union (IoU) i ,B j ) exceeds a certain threshold T, it is preliminarily considered that there is overlap. The system will enter a subsequent more in-depth detection process and perform a more accurate detection of the overlapping area by introducing the depth information in the 3D point cloud data.

[0073] Furthermore, the improved YOLOv8 model includes an input module, a backbone module, an auxiliary module, and a prediction module;

[0074] An input module, used to scale the initial image to a uniform size and obtain an input image;

[0075] The backbone module is used to extract image features of the input image;

[0076] Auxiliary module, used to perform sampling and connection operations on image features and fuse image features;

[0077] The prediction module is used to predict the first detection result according to the fused image features.

[0078] Furthermore, extracting image features of the input image includes:

[0079] Divide the input image into several data feature vectors;

[0080] Add several data feature vectors to obtain a feature vector;

[0081] Process the feature vector to obtain image features.

[0082] Furthermore, the first detection result includes: a target overlapping image and a target non-overlapping image.

[0083] Specifically, if visual inspection does not find overlap, the system will not need to continue with subsequent complex calculations, thereby effectively saving computing resources and improving overall operating efficiency. Through this design, the system optimizes the use of computing power while ensuring accuracy, making it more efficient.

[0084] Furthermore, matching the first detection result with the initial point cloud to obtain an alignment matrix includes:

[0085] Obtain 2D-3D cross-modal features based on the target overlapping image and the initial point cloud;

[0086] Perform feature fusion on 2D-3D cross-modal features and perform intersection detection on the fused features;

[0087] According to the results of intersection detection, abnormal features are removed and the alignment matrix is ​​obtained.

[0088] Specifically, the core task of this step is to accurately register the 3D point cloud data with the 2D image through the VP2P-based algorithm. The VP2P-based algorithm can process the data generated by different source sensors and perform spatial matching and fusion on them, so that the 3D point cloud and 2D image data can be seamlessly connected in the same coordinate system.

[0089] During this registration process, the system will calculate the best alignment matrix based on the pixel information in the image and the three-dimensional coordinate points in the point cloud. This alignment matrix not only ensures the precise correspondence between the point cloud data and the image data in the geometric space, but also provides a basis for point cloud projection and depth map generation in subsequent steps. Through high-precision registration, the system can accurately determine the relative positions of multiple objects in complex scenes and avoid recognition errors caused by coordinate misalignment or mismatch.

[0090] Furthermore, according to the target overlapping image and the initial point cloud, the 2D-3D cross-modal features are obtained, including:

[0091] The pixel features of the target overlapping image are obtained by using upsampling modules and skip connections;

[0092] Divide the initial point cloud into a uniform voxel grid, and assign each point in the initial point cloud to a corresponding voxel according to the three-dimensional coordinates, retain the voxels containing at least one point, and obtain sparse voxels;

[0093] Perform sparse convolution on sparse voxels to obtain voxel features;

[0094] Convert the voxel features into the first point-level features through feature transformation;

[0095] Use the PointNet++ network model to directly obtain the second point-level features of the initial point cloud;

[0096] Perform element-wise addition of the first point-level feature and the second point-level feature to obtain a third point-level feature;

[0097] According to the third point-level features and pixel features, 2D-3D cross-modal features are obtained.

[0098] Furthermore, the initial point cloud is projected onto the corresponding image using the alignment matrix, and the depth image is obtained by:

[0099] Extract the 3D coordinates of the initial point cloud, transform the 3D coordinates using the alignment matrix, and obtain the pixel positions corresponding to the 3D coordinates;

[0100] According to the pixel position, record the depth value of each point cloud;

[0101] Integrate the pixel position and depth value to obtain a depth image.

[0102] Specifically, after the point cloud and image are registered, the system will enter this module, which projects the 3D point cloud data onto the corresponding 2D image plane. Through this projection process, the system can generate a high-precision depth map that reflects the relative depth between objects. The 3D coordinates of the point cloud data are accurately mapped to the 2D image, so that each pixel can obtain its corresponding depth information.

[0103] The generated depth map can not only reflect the relative position and distance between objects, but also capture the subtle depth differences of objects in complex scenes. This information plays a vital role in subsequent detection and early warning, especially in scenes with complex occlusion and overlapping phenomena. The three-dimensional spatial information provided by the depth map can effectively help the system determine the front and back relationship and the real physical distribution of objects. This module provides important data support for the overall accuracy of the system.

[0104] Further, according to the depth image and the first detection result, issuing an early warning for object overlap includes:

[0105] Extract the depth values ​​of overlapping objects in the target overlapping image, and calculate the average depth values ​​of the overlapping objects respectively;

[0106] The difference in average depth values ​​is compared, and based on the comparison results, an early warning is issued for overlapping objects.

[0107] Specifically, this step is the core detection link of the entire system, which is specifically used to perform further accurate detection and warning of object overlap after the initial detection is completed and the initial warning is triggered. At this time, the system will analyze the depth map generated by the third module, especially for the overlapping object areas found in the initial detection process, and perform more in-depth overlap detection.

[0108] In this step, the system will calculate the average depth value of the object through the depth information to further refine the initial judgment. Assume that the average depths of the two objects are Z i and Z j , if their depth difference |Z i -Z j | is less than the preset threshold ΔZ, the system will trigger an accurate warning, indicating that there may be actual overlap or occlusion between objects. Through this secondary overlap detection, the system can significantly improve the accuracy of judging the relative position of objects in complex scenes and avoid false warnings caused by the accumulation of errors in the initial detection.

[0109] In addition, this step is highly flexible and can adjust the warning threshold according to the needs of different application scenarios. For example, in robot grasping tasks, the sensitivity of accurate warning can be adjusted according to the grasping requirements of the object; in remote sensing image analysis, the system can set more rigorous warning standards based on different depth information. This flexible warning mechanism enables the system to cope with various complex environments and ensure high-precision and high-robustness detection results.

[0110] The present invention also provides a multi-object overlap detection system based on image and point cloud fusion, such as Figure 8As shown, it specifically includes: visual overlap detection module, point cloud and image precise registration module, depth map generation module and early warning module;

[0111] Visual overlap detection module, used to detect objects in images;

[0112] Point cloud and image precise registration module, used to match point cloud and image according to the detection results and obtain the alignment matrix;

[0113] A depth map generation module, used to generate a depth image from the point cloud according to the alignment matrix;

[0114] The early warning module is used to issue overlapping early warnings based on the depth image and detection results.

[0115] The present embodiment is described in detail below with reference to the accompanying drawings:

[0116] Step 1: Target detection and segmentation and initial visual overlap detection

[0117] Before training the model, you first need to use the labelimg tool to label the objects in the image. Open labelimg, import the image to be labeled, use the rectangular box tool to mark each target object, and assign a category label to each box. The labeling example is as follows: Figure 2 shown.

[0118] This embodiment proposes to use an improved YOLOv8 network to perform object detection and instance segmentation of 2D images.

[0119] In this embodiment, the improved YOLOv8 module mainly improves performance by optimizing the backbone network and upsampling module. First, the original YOLOv8 backbone network is replaced by the PoolFormer backbone network with stronger feature extraction capabilities. PoolFormer uses a simplified non-parametric pooling operation to enhance the ability to extract key features in the image while reducing the number of network parameters. This improvement enables the model to more accurately segment the edge areas of stacked parts, reduce misidentification of the background, and improve segmentation accuracy.

[0120] In the upsampling module, the simple nearest neighbor interpolation method originally used by YOLOv8 is replaced by the more powerful CARAFE upsampling module. The CARAFE module uses the semantic information of the image more effectively through a larger receptive field and more refined feature reorganization, improving the accuracy of target segmentation and detail processing capabilities. After these two optimizations, the segmentation and detection accuracy of the model has been significantly improved, especially in the recognition of complex stacked parts, showing stronger robustness while maintaining real-time processing capabilities.

[0121] The improved YOLOv8 mainly consists of input layer, backbone framework, auxiliary framework and prediction framework.

[0122] Input layer: The image to be detected is scaled to a uniform size through data enhancement and adaptive image scaling methods.

[0123] The backbone framework is Figure 3 As shown: This embodiment proposes a backbone framework to extract image features using PoolFomer's self-attention mechanism. First, the input image I is divided into N blocks of data feature vectors, as shown in the following formula:

[0124] X=InputEmb(I)

[0125] Among them, X is the feature vector, X∈R N*C , N is the sequence length, C is the feature dimension. X is input into two residual sub-blocks.

[0126] The first sub-block consists of a normalization (Norm) and a pooling (Pool) module, and the result is added to the input X to obtain the feature Y:

[0127] Y=Pool(Norm(X))+X

[0128] The second sub-block consists of normalization and multi-layer perceptron modules, which form the final output Z with Y through normalization, linear transformation, and nonlinear activation operations:

[0129] Y=σ(Norm(Y)W1)W2+Y

[0130] Wherein, W1 and W2 are learnable parameters in the multilayer perceptron, and σ is an activation function. Optionally, the backbone framework of this embodiment can be formed by stacking 4 PoolFormer blocks, and the stacking ratio is [4, 4, 12, 4].

[0131] Auxiliary framework: A module composed of a path aggregation network (PAN) can be used to perform sampling and connection operations on the features extracted by the backbone framework to achieve feature fusion of security images. Figure 3 In the C2f module, the main function is to increase the depth and receptive field of the network and improve the feature extraction capability. CBS is mainly composed of convolution, batch normalization and activation functions, which are used to extract features, normalize output and enhance the nonlinear ability of the model.

[0132] The schematic diagram of the path aggregation network used in this embodiment is as follows Figure 4As shown in the figure, the path aggregation network (PAN) module uses a bottom-up enhancement path to shorten the information transmission path to store accurate positioning signals in low-level features and improve the structure of the feature pyramid. In the enhancement process, the features of three different levels are enhanced by shortening the path {M2, M3, M4}, represented by {N2, N3, N4}. The feature enhancement process of different levels can be obtained by adding the path information after the 3*3 convolution operation:

[0133]

[0134] in, represents feature addition.

[0135] Prediction framework: responsible for generating the final detection results. The framework includes three main outputs: bounding box regression, confidence score, and category probability. Bounding box regression is used to predict the location and size of the target. Usually four values ​​are output, corresponding to the center coordinates (x, y) and width and height (w, h) of the bounding box. The confidence score predicts whether each bounding box contains the target and the confidence of the target. Category probability is used to predict the probability that the target belongs to each category.

[0136] Then create a folder named mydata in the data directory in the YOLOv8 folder, put the labeled xml files and pictures in the corresponding directories, store the pictures in the images folder, and then store the xml files corresponding to the images in the xml folder. After that, four files, train.txt, val.txt, test.txt and trainval.txt, are automatically generated in the Main folder and stored in the dataSet folder, and then training begins.

[0137] After training, the system uses the YOLOv8 model to detect and segment the input image. Through YOLOv8 reasoning, the bounding box and category information of each detected object are obtained, and the results are output, including the segmentation mask (if applicable) and the precise boundaries of each object. No matter how complex or changing the image background is, the entire detection and segmentation process can ensure high accuracy.

[0138] On this basis, the system integrates the visual overlap initial detection function. This function can analyze the spatial relationship of the detected targets and determine whether there is overlap or occlusion between targets in the image. Visual overlap initial detection mainly analyzes the relative position of objects in the two-dimensional image, identifies their boundary overlap areas, and generates preliminary overlap judgment results. If overlap is detected between targets, the system will further enter the depth analysis module to perform more accurate three-dimensional point cloud depth information analysis; on the contrary, if no overlap is found in the initial visual detection, the system will skip the subsequent process to avoid unnecessary calculations, save resources and improve processing efficiency.

[0139] Step 2: Accurate registration of point cloud and image

[0140] This embodiment implements the registration of point cloud and image based on a differentiable registration method of VP2P image and radar point cloud.

[0141] VP2P is a multimodal learning framework, which mainly consists of three parts: cross-modal learning part, intersection detection part, and end-to-end posture solution part.

[0142] The cross-modal learning part is as follows Figure 5 As shown in the figure, the cross-modal learning part: a pair of improperly registered images and point clouds are used as input, and a U-shaped convolutional network is used for the pixel branch to extract image features. The base layer of the U-shaped network used here adopts a residual network, and the pixel-by-pixel features f are obtained through upsampling modules and jump connections. pixel ∈R H×W×C .

[0143] For the input point cloud, the three-dimensional space is divided into uniform voxel grids according to size, and each point in the point cloud is assigned to the corresponding voxel according to its three-dimensional coordinates. Then only voxels containing at least one point are retained, and empty voxels are not stored. Sparse convolution operations are performed on these sparse voxels to obtain the voxel features f voxel ∈R N×C .

[0144] At the same time, the original point cloud is input into the PointNet++-based network, and the self-attention module is used to perform hierarchical set abstraction to obtain global features. And further apply the feature propagation module to learn the point-by-point features f point ∈R N×C .

[0145] Then the voxel-level feature f voxel ∈R N×C Through feature transformation, it is converted into point-level features, and then element-wise added with the point set features directly extracted from the point cloud. The result is then adaptively weighted with the pixel-level features.

[0146] The intersection detection module is as follows Figure 6 As shown in the figure, the intersection detection part consists of four small parts: feature fusion, probability prediction, abnormal area removal and loss function.

[0147] Feature fusion is the global 2D feature g extracted from the cross-modal learning part 2D With the feature f of each 3D point 3D Concatenate to form a combined feature vector [g 2D :f 3D ].

[0148] The probability prediction uses a multi-layer perceptron to learn the combined feature vector and output the probability that each 3D point is located in the 2D-3D intersection area. Similarly, for the abnormal area in the 2D image space, the probability of each pixel being located in the intersection area is predicted

[0149] In the abnormal region removal, a probability threshold σ is defined, and the 2D or 3D elements with a probability lower than σ will be considered as outliers and removed.

[0150] Finally, by sampling Z from the intersection area and the abnormal area respectively in and Z out For 2D or 3D elements, the detection loss is calculated:

[0151]

[0152] The end-to-end posture solution part is as follows Figure 7 As shown in the figure, in order to align the point cloud and the image, the distance information of the 3D point cloud is used for secondary judgment. The end-to-end posture solution part is as follows: for each 2D pixel point, the point with the greatest similarity to the remaining 3D point features in the feature space is searched, and the gradient is estimated using Gumbel Softmax to ensure differentiability during training.

[0153] Then, we solve the PnP problem in the form of probability distribution, define the output as probability distribution p(T|X), and use KL divergence loss optimization:

[0154]

[0155] In the inference stage, in order to improve efficiency, the EPnP method with a time complexity of O(n) is used, and combined with the RANSAC algorithm for more robust pose estimation.

[0156] Finally, the Gauss-Newton method is used to further solve the accurate posture

[0157] T'=[R'|t'], and calculate the posture loss:

[0158]

[0159] If overlap is detected in the first module, the system will enter the stage of precise registration of point cloud and image. In this step, the image and point cloud data are used as input, and the system processes the sparse voxels to generate a layer of sparse voxels V. Next, the ternary network is applied to extract features from the three modalities. The specific operation is to represent the two-dimensional pattern as a pixel feature, and the three-dimensional pattern as a combination of voxel and point features, and use adaptive weighted losses to learn unique 2D-3D cross-modal patterns. After extracting the features, the system performs cross-modal feature fusion to detect the intersection area in 2D and 3D space. According to the results of the intersection detection, the abnormal area is removed, and the 2D-3D feature matching is used to establish the 2D-3D correspondence. If the intersection in the intersection detection is greater than the set threshold, the posture solution must be linked. If no overlap is detected by visual detection, that is, the intersection is less than the threshold, there is no need to link the posture solution.

[0160] Specifically, the system first obtains the 3D point cloud data in the scene and matches it with the corresponding 2D image. Then, the VP2P algorithm is used to calculate the geometric transformation relationship between the point cloud data and the image to generate an alignment matrix. This matrix ensures that each 3D point in the point cloud can be mapped to the corresponding pixel in the image.

[0161] When performing this registration step, the system checks spatial consistency to ensure that the registration result accurately reflects the three-dimensional relationship between objects. Finally, the generated alignment matrix provides an important basis for subsequent steps, enabling the system to obtain depth information for each target object on the two-dimensional image. This process ensures the accuracy and reliability of the data, laying a solid foundation for subsequent depth analysis and detection.

[0162] Step 3: Point cloud projection and depth map generation

[0163] After completing the registration of the point cloud and the image in the second step, the system will enter the point cloud projection and depth map generation module. In this stage, the system uses the alignment matrix generated by the registration to project the 3D point cloud data onto the corresponding 2D image.

[0164] In this process, the system will traverse each 3D point cloud point, extract its 3D coordinates (x, y, z), and use the alignment matrix to transform each 3D coordinate to calculate its corresponding pixel position on the 2D image. This process converts the 3D coordinates into homogeneous coordinates:

[0165]

[0166] In the formula for converting three-dimensional coordinates to homogeneous coordinates, w is the homogeneous factor, which is usually set to 1, and the corresponding projection formula is applied to obtain the image coordinates (u, v):

[0167]

[0168] Where K is the internal parameter matrix:

[0169]

[0170] f x ,f y are the focal lengths of the camera in the x and y axes, respectively, and c x ,c y are the coordinates of the optical center of the image on the x and y axes, respectively, R is the rotation matrix, t is the translation vector, and I is a 3×3 identity matrix, which is used to concatenate the translation vector and the rotation matrix into a 3×4 matrix.

[0171] Next, at the corresponding pixel position of the image, the system records the depth value Z of each point cloud point to form a depth map. If multiple point cloud points are projected to the same pixel position, the system will select the one with the smallest depth value to ensure that it represents the object closest to the camera.

[0172] After completing this process, the system integrates all processed pixel positions and corresponding depth values ​​to generate a complete depth map. The value of each pixel in the depth map represents the relative depth of the object, which can reflect the relative distance and three-dimensional spatial relationship between objects. Finally, the system will perform a quality check on the generated depth map to ensure its accuracy, especially when multiple objects overlap. The quality of the depth map can be viewed through visualization tools, and the parameters in the projection process can be adjusted in time. Through this series of steps, the system can effectively generate a depth map, which helps to accurately identify the relative depth between objects and ensure the accuracy of subsequent detection.

[0173] Step 4: Secondary overlap detection and accurate warning

[0174] After the depth map is generated, the system enters the secondary overlap detection and precise warning module. This module is only started after the initial visual overlap detection is successful and the initial warning is triggered. The system first extracts the depth information corresponding to the overlapping area found in the initial detection, which will be used for subsequent depth analysis. In specific operations, the system will screen the points in the overlapping area, extract the depth values ​​of the two objects, and then calculate the average depth values ​​of the two objects in the area respectively. By comparing the average depth difference of the two objects, the system can determine the actual three-dimensional spatial relationship between them. If the difference in the average depth values ​​is less than the preset warning threshold, the system will further trigger a precise warning, indicating that there is actual overlap or occlusion between the objects.

[0175] In this case, the system will output a warning signal to prompt the user or system to process or optimize the relationship between objects. In different application scenarios, the warning threshold can be adjusted according to actual needs to adapt to the accuracy requirements of different tasks. For example, in robot grasping tasks, the system can set a higher precision warning sensitivity to ensure the safety and effectiveness of object grasping, while in remote sensing image stitching, stricter warning standards can be set according to the depth information of different targets to improve the accuracy of stitching. Through this series of steps, the system can achieve accurate overlap detection and an effective warning mechanism, providing important support for subsequent processing and decision-making.

[0176] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multi-object overlapping detection method based on image and point cloud fusion, characterized in that: include: Obtain initial image and initial point cloud; Detecting the initial image to obtain a first detection result; Matching the first detection result with the initial point cloud to obtain an alignment matrix; Using the alignment matrix, projecting the initial point cloud onto a corresponding image to obtain a depth image; An early warning is issued for object overlap based on the depth image and the first detection result.

2. The multi-object overlapping detection method based on image and point cloud fusion according to claim 1, characterized in that: Detecting the initial image to obtain a first detection result includes: The initial image is input into an improved YOLOv8 model to obtain a first detection result, wherein the improved YOLOv8 model is obtained by training a training set, and the training set is annotated image data containing overlapping objects.

3. The multi-object overlapping detection method based on image and point cloud fusion according to claim 2 is characterized in that: The improved YOLOv8 model includes an input module, a backbone module, an auxiliary module and a prediction module; The input module is used to scale the initial image to a uniform size to obtain an input image; The backbone module is used to extract image features of the input image; The auxiliary module is used to perform sampling and connection operations on the image features to fuse the image features; The prediction module is used to predict the first detection result according to the fused image features.

4. The multi-object overlapping detection method based on image and point cloud fusion according to claim 3 is characterized in that: Extracting image features of the input image includes: Dividing the input image into a plurality of data feature vectors; Adding the plurality of data feature vectors to obtain a feature vector; The feature vector is processed to obtain the image feature.

5. The multi-object overlapping detection method based on image and point cloud fusion according to claim 1, characterized in that: The first detection result includes: a target overlapping image and a target non-overlapping image.

6. The multi-object overlapping detection method based on image and point cloud fusion according to claim 5, characterized in that: Matching the first detection result with the initial point cloud to obtain an alignment matrix includes: Acquire 2D-3D cross-modal features according to the target overlapping image and the initial point cloud; Performing feature fusion on the 2D-3D cross-modal features, and performing intersection detection on the fused features; According to the result of the intersection detection, abnormal features are removed to obtain the alignment matrix.

7. The multi-object overlapping detection method based on image and point cloud fusion according to claim 6, characterized in that: Acquiring 2D-3D cross-modal features according to the target overlapping image and the initial point cloud includes: Obtaining pixel features of the target overlapping image using an upsampling module and a skip connection; Dividing the initial point cloud into a uniform voxel grid, and assigning each point in the initial point cloud to a corresponding voxel according to the three-dimensional coordinates, retaining the voxel containing at least one point, and obtaining sparse voxels; Performing sparse convolution on the sparse voxels to obtain voxel features; Converting the voxel feature into a first point-level feature through feature conversion; Using the PointNet++ network model, directly obtaining the second point-level features of the initial point cloud; Performing element-wise addition on the first point-level feature and the second point-level feature to obtain a third point-level feature; A 2D-3D cross-modal feature is obtained according to the third point-level feature and the pixel feature.

8. The multi-object overlapping detection method based on image and point cloud fusion according to claim 1, characterized in that: Using the alignment matrix, projecting the initial point cloud onto a corresponding image, and acquiring a depth image includes: Extracting the three-dimensional coordinates of the initial point cloud, transforming the three-dimensional coordinates using the alignment matrix, and obtaining pixel positions corresponding to the three-dimensional coordinates; According to the pixel position, record the depth value of each point cloud; The pixel position and the depth value are integrated to obtain the depth image.

9. The multi-object overlapping detection method based on image and point cloud fusion according to claim 5, characterized in that: Providing an early warning for object overlap according to the depth image and the first detection result includes: Extracting the depth values ​​of the overlapping objects in the target overlapping image, and calculating the average depth values ​​of the overlapping objects respectively; The differences in the average depth values ​​are compared, and based on the comparison result, an early warning is issued for the overlapping objects. 10.Multi-object overlapping detection system based on image and point cloud fusion, characterized in that: include: Visual overlap detection module, point cloud and image precise registration module, depth map generation module and early warning module; The visual overlap detection module is used to perform target detection on the image; The point cloud and image precise registration module is used to match the point cloud and the image according to the detection results to obtain the alignment matrix; The depth map generation module is used to generate a depth image according to the alignment matrix and the point cloud; The warning module is used to issue overlapping warnings based on the depth image and the detection results.

Images