A laser point cloud occluded vehicle completion method and device based on a radar and vision fusion deep learning framework

Abstract

The application discloses a kind of automatic driving environment perception technical field's based on laser vision fusion deep learning framework's laser point cloud occluded vehicle completion method and device, comprising: obtaining original laser point cloud data and aerial photograph image;Fan-shaped occlusion area is constructed based on original laser point cloud data;Identify the two-dimensional boundary box, orientation and category label of all vehicle targets in image;Fan-shaped occlusion area and the two-dimensional boundary box, orientation and category label of all vehicle targets are input to pre-trained double-branch deep learning network, whether there is existence occluded vehicle and the position, orientation and category information of occluded vehicle are predicted;Based on the category information of occluded vehicle, select vehicle point cloud template, then generate vehicle point cloud after completion scene point cloud as the final result of occlusion area vehicle completion.The application introduces image-point cloud space mapping mechanism based on laser vision fusion deep learning framework, significantly improves the automatic identification and positioning ability to occluded vehicle.

Description

Technical Field

[0001] This invention relates to a method and device for completing vehicle occlusion caused by laser point clouds based on a laser-visual fusion deep learning framework, belonging to the field of autonomous driving environmental perception technology. Background Technology

[0002] With the rapid increase in the number of motor vehicles, especially cars, road traffic pressure is intensifying, and traffic accidents are becoming more frequent, diverse, and complex. Although the total number of accidents has stabilized in recent years, the sheer number of motor vehicles and drivers means that traffic safety management still faces enormous challenges. Road traffic accidents not only cause significant property damage and personal injury, but also pose a serious threat to urban operational efficiency and public safety. Therefore, improving road traffic perception capabilities, especially in areas such as accident monitoring, early warning, and causal analysis, has become a core direction for the development of Intelligent Transportation Systems (ITS) and traffic management systems.

[0003] In current environmental perception systems, LiDAR (Light Detection and Ranging) has become the primary sensor for autonomous driving and roadside perception due to its high-precision 3D reconstruction capabilities. However, in actual deployment, LiDAR suffers from an inherent occlusion problem: when a vehicle in front completely obscures the laser path of a vehicle behind, the vehicle behind cannot be scanned, resulting in missed detections. This problem is particularly prominent in typical traffic scenarios such as dense traffic flow or stationary congestion. Existing solutions include increasing the number of LiDARs, deploying multi-view perception networks, or using motion trajectory prediction algorithms for target estimation, but these methods suffer from high costs, susceptibility to accumulated errors from occlusion, and limited applicability. In contrast, drones have advantages such as a wide top-down view, flexible deployment, and high image resolution, enabling them to capture information in traditional roadside perception blind spots, becoming a new sensing method for supplementing blind spots. However, the current technology system still lacks a method to complete the point cloud data of occluded vehicles through laser-view fusion, and a high-precision completion mechanism for occluded areas has not yet been formed, which restricts the overall environmental modeling quality and the perception integrity of the system. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a method and device for laser point cloud occlusion vehicle completion based on the laser-visual fusion deep learning framework.

[0005] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution.

[0006] In a first aspect, this invention discloses a method for completing vehicle occlusion caused by laser point clouds based on a laser-visual fusion deep learning framework, comprising: Acquire raw laser point cloud data and aerial images of road scenes that have been time-aligned and collected synchronously by LiDAR and UAV; The original laser point cloud data is processed to identify vehicles ahead, and a fan-shaped occlusion area is constructed based on the geometric contour of the vehicle and the positional relationship between the laser radar and the laser point cloud. The aerial images are processed to identify the two-dimensional bounding boxes, orientations, and category labels of all vehicle targets in the images; The two-dimensional bounding boxes, orientations, and category labels of the fan-shaped occlusion region and all vehicle targets are input into a pre-trained dual-branch deep learning network to predict whether there are occluded vehicles and the location, orientation, and category information of the occluded vehicles. Based on the category information of the occluded vehicle, a matching vehicle point cloud template is selected from a pre-built vehicle point cloud template library. Based on the position and orientation of the occluded vehicle, the vehicle point cloud template is placed at the corresponding position in the original laser point cloud data to generate a scene point cloud after vehicle point cloud completion, which is the final result of vehicle completion in the occluded area.

[0007] Furthermore, the process of identifying the vehicle ahead by processing the original laser point cloud data includes: The original laser point cloud data is preprocessed by sequentially performing denoising, ground segmentation, and Euclidean clustering to obtain clustering results, and vehicles ahead are identified based on the clustering results.

[0008] Furthermore, the aerial image recognition processing, which identifies the two-dimensional bounding boxes, orientations, and category labels of all vehicle targets in the image, includes: The aerial image is input into a pre-trained YOLOv8 deep learning model, which outputs the target's two-dimensional bounding box, orientation, and category label.

[0009] Furthermore, the dual-branch deep learning network includes: a dual-branch deep learning network comprising an image feature extraction module, a point cloud feature extraction module, a feature fusion module, and a mapping regression module; The image feature extraction module is used to extract features from the two-dimensional bounding box, orientation, and category label of the vehicle target using ConvNeXt, thereby obtaining the semantic features of the detected target in the aerial image. ; The point cloud feature extraction module is used to extract the three-dimensional spatial structure features of the fan-shaped occlusion region using PointTransformer. ; The feature fusion module is used to integrate the semantic features of the detected targets in the aerial image based on a pre-constructed projection mapping matrix 𝑃. The point cloud space projected onto the fan-shaped occlusion region Ω, and the three-dimensional spatial structure features of the fan-shaped occlusion region. Alignment, assisted by cross-attention mechanism to generate joint features of image and point cloud The projection mapping matrix P The representation of is: ; ; in, f x , f y The focal length in the horizontal and vertical directions of the image; c x , c y Principal point of the image; K This is the camera intrinsic parameter matrix; R This is a rotation matrix used to align the point cloud coordinate system with the camera coordinate system; T This is a translation vector used to represent the relative position between the lidar and the UAV. R and T Perform coordinate transformation between point cloud data and image data; The mapping and regression module is used to determine the joint features of the image and the point cloud. Predict whether there are any obscured vehicles, as well as the location, orientation, and category of the obscured vehicles.

[0010] Furthermore, the training process of the dual-branch deep learning network includes: Acquire historical datasets collected synchronously by LiDAR and UAV, and use the historical datasets to construct a training set; The feature fusion module employs an attention mechanism to achieve cross-modal information alignment, as shown below: ; in, A This represents the attention matrix between the image and the point cloud. W 1. W 2 represents the feature map weight matrix; The impairment function used in the dual-branch deep learning network for: ; in, Represents the point cloud coordinates predicted from image features; This represents the coordinates of the center point of a vehicle in a real point cloud; Represents the weighting coefficient of the loss term; The dual-branch deep learning network is trained using the training set, the attention mechanism, and the impairment function to obtain a trained dual-branch deep learning network.

[0011] Furthermore, the construction of the vehicle point cloud template library includes: Point cloud data of typical vehicles are extracted from historical LiDAR data. Euclidean clustering is used to separate the vehicle point cloud from the background based on the spatial distribution of the point cloud data of typical vehicles. The separated vehicle point clouds are then labeled with categories and stored in the vehicle point cloud template library.

[0012] Further, the step of placing the vehicle point cloud template into the corresponding position in the original laser point cloud data based on the position and orientation of the occluded vehicle to generate the scene point cloud after vehicle point cloud completion includes: The vehicle point cloud template is spatially registered with the original laser point cloud data by an iterative nearest point algorithm. After spatial registration, a probabilistic fusion algorithm is used to eliminate noise in the overlapping area, forming a scene point cloud after the vehicle point cloud is completed.

[0013] Furthermore, the step of spatially registering the vehicle point cloud template with the original laser point cloud data using an iterative nearest-point algorithm includes: (1) Based on the position and orientation of the obscured vehicle, the vehicle point cloud template is initially aligned to the corresponding area of ​​the original laser point cloud data as the initial input of the iterative nearest point algorithm; (2) For each point in the vehicle point cloud template, search for the point with the closest Euclidean distance in the original laser point cloud data and establish a set of point-to-point correspondences; (3) Calculate the optimal rotation matrix by minimizing the objective function. R 1. Translation vector T 1. The expression for minimizing the objective function is: ; in, q i This represents the i-th point in the vehicle point cloud template; p i Indicates the original laser point cloud data and q i The nearest point; This represents the square of the Euclidean distance. min Indicates minimization; (4) Solve the R 1 and T 1. Apply to the vehicle point cloud template to update its spatial position; (5) Repeat steps (2) to (4) until one of the following convergence conditions is met: The difference between the transformation matrices of two adjacent iterations is less than a threshold. The transformation matrix is ​​obtained by solving... R 1 and T A matrix consisting of 1; The change in the value of the objective function is less than the preset tolerance; The maximum number of iterations has been reached.

[0014] Furthermore, the step of using a probabilistic fusion algorithm to eliminate noise in overlapping regions and form a scene point cloud after vehicle point cloud completion includes: Probabilistic fusion is performed on the overlapping areas of the registered vehicle point cloud template and the original laser point cloud data, including: Based on the point cloud density and registration error, calculate any one of the vehicle point cloud template and the original laser point cloud data. i Confidence level of point assignment ω i ; ,in, D i It is the first i Point cloud density factor of each point E i It is the first i Registration error factor for each point α , β It is the adjustment coefficient; Points with confidence levels below the threshold in the overlapping region are considered outliers, while points with confidence levels above the threshold in the overlapping region are used as high-confidence vehicle point cloud templates and original laser point cloud data. Remove the outliers, retain the high-confidence vehicle point cloud template and the original laser point cloud data, and generate the scene point cloud after vehicle completion.

[0015] Secondly, this invention also discloses a laser point cloud occlusion vehicle completion device based on a laser-visual fusion deep learning framework, comprising: The acquisition module is used to acquire raw laser point cloud data and aerial images of the road scene that have been time-aligned and collected synchronously by LiDAR and UAV. The first processing module is used to identify the original laser point cloud data, identify the vehicle in front, and construct a fan-shaped occlusion area based on the geometric contour of the vehicle and the positional relationship of the laser radar. The second processing module is used to perform recognition processing on the aerial image, and to identify the two-dimensional bounding box, orientation and category label of all vehicle targets in the image; The prediction module is used to input the two-dimensional bounding boxes, orientations, and category labels of the fan-shaped occlusion area and all vehicle targets into a pre-trained dual-branch deep learning network to predict whether there are occluded vehicles and the location, orientation, and category information of the occluded vehicles. The generation module is used to select a matching vehicle point cloud template from a pre-built vehicle point cloud template library based on the category information of the occluded vehicle; and to place the vehicle point cloud template into the corresponding position in the original laser point cloud data based on the position and orientation of the occluded vehicle, thereby generating a scene point cloud after vehicle point cloud completion as the final result of vehicle completion in the occluded area.

[0016] The beneficial effects achieved by this invention are as follows: The laser point cloud-based vehicle compensation method based on a laser-eye fusion deep learning framework, as described in this invention, achieves high-precision target correspondence between UAV aerial images and ground laser point clouds by introducing an image-point cloud spatial mapping mechanism based on this framework. This significantly improves the automatic identification and localization capabilities of occluded vehicles. Simultaneously, a vehicle point cloud template library is constructed. Based on model detection results, a matching vehicle point cloud template can be selected from this library to achieve complete reconstruction of vehicles in occluded areas, enhancing the environmental perception integrity and robustness of the intelligent transportation system in complex traffic scenarios. Furthermore, compared to traditional solutions relying on multiple high-line-count laser radars, this invention fully utilizes the advantages of UAVs—wide field of view, high mobility, and controllable cost—effectively reducing the overall deployment and operating costs of the system while ensuring effective perception coverage. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a point cloud diagram of the obscured vehicle. Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0019] The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature.

[0020] Example 1: This example introduces a method for completing vehicle occlusion caused by laser point clouds based on a laser-visual fusion deep learning framework, including: Acquire raw laser point cloud data and aerial images of road scenes that have been time-aligned and collected synchronously by LiDAR and UAV; The original laser point cloud data is processed to identify vehicles ahead, and a fan-shaped occlusion area is constructed based on the geometric contour of the vehicle and the positional relationship between the laser radar and the laser point cloud. The aerial images are processed to identify the two-dimensional bounding boxes, orientations, and category labels of all vehicle targets in the images; The two-dimensional bounding boxes, orientations, and category labels of the fan-shaped occlusion region and all vehicle targets are input into a pre-trained dual-branch deep learning network to predict whether there are occluded vehicles and the location, orientation, and category information of the occluded vehicles. Based on the category information of the occluded vehicle, a matching vehicle point cloud template is selected from a pre-built vehicle point cloud template library. Based on the position and orientation of the occluded vehicle, the vehicle point cloud template is placed at the corresponding position in the original laser point cloud data to generate a scene point cloud after vehicle point cloud completion, which is the final result of vehicle completion in the occluded area.

[0021] The process of identifying the original laser point cloud data to identify vehicles ahead includes: The original laser point cloud data is preprocessed by sequentially performing denoising, ground segmentation, and Euclidean clustering to obtain clustering results, and vehicles ahead are identified based on the clustering results.

[0022] The aerial image recognition process involves identifying the two-dimensional bounding boxes, orientations, and category labels of all vehicle targets in the image, including: The aerial image is input into a pre-trained YOLOv8 deep learning model, which outputs the target's two-dimensional bounding box, orientation, and category label.

[0023] The dual-branch deep learning network includes: an image feature extraction module, a point cloud feature extraction module, a feature fusion module, and a mapping regression module. The image feature extraction module is used to extract features from the two-dimensional bounding box, orientation, and category label of the vehicle target using ConvNeXt, thereby obtaining the semantic features of the detected target in the aerial image. ; The point cloud feature extraction module is used to extract the three-dimensional spatial structure features of the fan-shaped occlusion region using PointTransformer. ; The feature fusion module is used to integrate the semantic features of the detected targets in the aerial image based on a pre-constructed projection mapping matrix 𝑃. The point cloud space projected onto the fan-shaped occlusion region Ω, and the three-dimensional spatial structure features of the fan-shaped occlusion region. Alignment, assisted by cross-attention mechanism to generate joint features of image and point cloud The projection mapping matrix P The representation of is: ; ; in, f x , f y The focal length in the horizontal and vertical directions of the image; c x , c y Principal point of the image; K This is the camera intrinsic parameter matrix; R This is a rotation matrix used to align the point cloud coordinate system with the camera coordinate system; T This is a translation vector used to represent the relative position between the lidar and the UAV. R and T Perform coordinate transformation between point cloud data and image data; The mapping and regression module is used to perform mapping based on the joint features of the image and point cloud. Predict whether there are any obscured vehicles, as well as the location, orientation, and category of the obscured vehicles.

[0024] The training process of the dual-branch deep learning network includes: Obtain historical datasets collected synchronously by LiDAR and UAVs, and construct a training set using the historical datasets. The training set can also use datasets such as KITTI.

[0025] The feature fusion module employs an attention mechanism to achieve cross-modal information alignment, as shown below: ; in, This represents the attention matrix between the image and the point cloud. Represents the feature mapping weight matrix; The impairment function used in the dual-branch deep learning network for: ; in, Represents the point cloud coordinates predicted from image features; This represents the coordinates of the center point of a vehicle in a real point cloud; Represents the weighting coefficient of the loss term; The dual-branch deep learning network is trained using the training set, the attention mechanism, and the impairment function to obtain a trained dual-branch deep learning network.

[0026] The construction of the vehicle point cloud template library includes: Point cloud data of typical vehicles are extracted from historical LiDAR data. Euclidean clustering is used to separate the vehicle point cloud from the background based on the spatial distribution of the point cloud data of typical vehicles. The separated vehicle point clouds are then labeled with categories and stored in the vehicle point cloud template library.

[0027] The step of placing the vehicle point cloud template into the corresponding position in the original laser point cloud data based on the position and orientation of the occluded vehicle to generate a scene point cloud after vehicle point cloud completion includes: The vehicle point cloud template is spatially registered with the original laser point cloud data by an iterative nearest point algorithm. After spatial registration, a probabilistic fusion algorithm is used to eliminate noise in the overlapping area, forming a scene point cloud after the vehicle point cloud is completed.

[0028] The step of spatially registering the vehicle point cloud template with the original laser point cloud data using an iterative nearest-point algorithm includes: (1) Based on the position and orientation of the obscured vehicle, the vehicle point cloud template is initially aligned to the corresponding area of ​​the original laser point cloud data as the initial input of the iterative nearest point algorithm; (2) For each point in the vehicle point cloud template, search for the point with the closest Euclidean distance in the original laser point cloud data and establish a set of point-to-point correspondences; (3) Calculate the optimal rotation matrix by minimizing the objective function. R 1. Translation vector T 1. The expression for minimizing the objective function is: ; in, q i This represents the i-th point in the vehicle point cloud template; p i Indicates the original laser point cloud data and q i The nearest point; This represents the square of the Euclidean distance. min Indicates minimization; (4) Solve the R 1 and T 1. Apply to the vehicle point cloud template to update its spatial position; (5) Repeat steps (2) to (4) until one of the following convergence conditions is met: The difference between the transformation matrices of two consecutive iterations is less than a threshold, where the transformation matrix is ​​obtained by solving the algorithm. R 1 and T A matrix consisting of 1; The change in the value of the objective function is less than the preset tolerance; The maximum number of iterations has been reached.

[0029] The process of using a probabilistic fusion algorithm to eliminate noise in overlapping regions and form a scene point cloud after vehicle point cloud completion includes: Probabilistic fusion is performed on the overlapping areas of the registered vehicle point cloud template and the original laser point cloud data, including: Based on the point cloud density and registration error, calculate any one of the vehicle point cloud template and the original laser point cloud data. i Confidence level of point assignment ω i ; ,in, D i and E i It is the first i The point cloud density factor and registration error factor for each point (the ICP algorithm can automatically calculate the corresponding values ​​based on the point cloud data). α , β It is the adjustment coefficient; Points with confidence levels below the threshold in the overlapping region are considered outliers, while points with confidence levels above the threshold in the overlapping region are used as high-confidence vehicle point cloud templates and original laser point cloud data. Remove the outliers, retain the high-confidence vehicle point cloud template and the original laser point cloud data, and generate the scene point cloud after vehicle completion.

[0030] Example 2: This example introduces a method for completing vehicle occlusion caused by laser point clouds based on a laser-visual fusion deep learning framework, such as... Figure 1 As shown, it includes the following steps: Step 1: Synchronous Acquisition of Multi-Source Heterogeneous Data: Simultaneous acquisition of laser point cloud data and aerial images of the road scene is achieved using LiDAR and a drone. RTK-GPS is then used to align the point cloud and images in time. To enable subsequent fusion processing of the image and point cloud data, the intrinsic and extrinsic parameters K, R, T of the drone and LiDAR are used to construct a coordinate system from the point cloud coordinates. To the image coordinate system The projection mapping matrix P can be used to map spatial points to the image plane, realizing the spatial correspondence of multimodal data; ; ; in, The focal length of the image in the horizontal and vertical directions (in pixels); The principal point of the image (usually the image center); K is the camera intrinsic parameter matrix, derived from camera calibration or manufacturer parameters; This is a rotation matrix that aligns the point cloud coordinate system with the camera coordinate system. The translation vector represents the relative position between the two sensors.

[0031] Step 2, LiDAR Occlusion Area Detection: Preprocess the LiDAR point cloud data using advanced technologies, including denoising, ground segmentation, and Euclidean clustering. Based on the clustering results, identify vehicles ahead. Once a vehicle (such as a truck or bus) is identified in the point cloud through clustering, a fan-shaped occlusion area can be constructed based on the vehicle's geometric contour and the LiDAR's positional relationship to define the potentially occluded area. This area likely contains occluded vehicle targets, but these are missing from the point cloud and cannot be clustered or identified. In subsequent steps, drone aerial imagery is used to detect any missed vehicles within this area. If a target exists in the image but is missing from the point cloud, the "Occluded Vehicle Point Cloud Template Generation" module is triggered to complete the occluded vehicle point cloud. The occluded area is modeled using polar coordinates and defined as follows: ; in, Represents the radial distance from a laser radar to a point in space, expressed in meters; This indicates the horizontal scanning angle of the point, expressed in degrees or radians. Indicates the minimum and maximum radial range of the occluded area; The minimum and maximum scanning angles of the occluded area are shown.

[0032] Step 3: Vehicle Detection in Drone Aerial Images: By inputting the aerial images into a pre-trained YOLOv8 (YouOnly Look Once version 8) deep learning model, the model quickly identifies and locates all vehicle targets in the image, outputting the target's two-dimensional bounding box, orientation, and category label. In this invention, the category label mainly includes the types of motor vehicles in typical road traffic scenarios, including but not limited to cars, buses, and trucks.

[0033] Step 4: Cross-modal target matching and missing target detection within occluded areas: Construct a dual-branch deep learning network consisting of four modules: image feature extraction, point cloud feature extraction, feature fusion, and mapping-regression. The image branch uses ConvNeXt for feature extraction, the point cloud branch uses PointTransformer to extract 3D spatial structure features, and the fusion module fuses semantic features of the detected targets from the aerial image. Spatial characteristics of the fan-shaped occlusion region Ω in laser point cloud data The system learns the mapping relationship between the two, providing an "alignment" basis for feature fusion. The fusion result is then input into the mapping regression module to predict the existence of occluded vehicles and their location, orientation, and category information, providing a spatial basis for subsequent vehicle point cloud template generation. When a vehicle is detected in the aerial image but not in the corresponding point cloud, it is determined to be an occluded vehicle. Feature fusion employs an attention mechanism to achieve cross-modal information alignment. ; in, This represents the joint features of the fused image and the point cloud; This represents the attention matrix between the image and the point cloud. This represents the feature mapping weight matrix.

[0034] Supervised learning is performed using a two-branch network trained end-to-end, with the following loss function: ; in, Represents the point cloud coordinates predicted from image features; This represents the coordinates of the center point of the vehicle in the real point cloud; This represents the weighting coefficient of the loss term (generally set empirically, such as 1.0 and 0.5).

[0035] Step 5: Construction and Selection of Occluded Vehicle Point Cloud Templates: Extract point cloud data of typical vehicles from the LiDAR data, construct a vehicle point cloud template library, and use Euclidean clustering to separate the vehicle point clouds from the background based on their spatial distribution. The separated vehicle point clouds are then categorized, including but not limited to cars, buses, and trucks. Based on the occluded vehicle information predicted in Step 4, select a matching vehicle point cloud template from the template library.

[0036] Step 6: Fusion of Vehicle Point Cloud Template and Original Point Cloud Data: Using the occluded vehicle information determined in Step 4 through image-point cloud mapping, the vehicle point cloud template selected in Step 5 is placed in the corresponding position. The Iterative Closest Point (ICP) algorithm is used to spatially register the vehicle point cloud template with the original point cloud. A probabilistic fusion algorithm is then used to eliminate noise in overlapping areas, forming the scene point cloud after vehicle point cloud completion. The ICP algorithm specifically includes the following steps: (1) Initial alignment of vehicle point cloud template with original point cloud: Based on the position, orientation and category information of the occluded vehicle determined in step 4, the vehicle point cloud template selected in step 5 is initially aligned to the corresponding area in the original point cloud as the initial input of the ICP algorithm.

[0037] Closest point matching: For each point in the vehicle point cloud template Search for the point in the original point cloud with the closest Euclidean distance. Establish a set of point-to-point correspondence relationships.

[0038] (3) Solving rigid body transformation: Calculate the optimal rotation matrix by minimizing the objective function. Translation vector ; ; in, This indicates the first [item] in the vehicle point cloud template. One point; Indicates the original point cloud and The nearest point; It is a rotation matrix; It is a translation vector.

[0039] (4) Transformation Application and Iterative Update: The solution is applied... and Apply the method to the vehicle point cloud template, update its spatial position, and repeat steps (2) to (4) until one of the following convergence conditions is met: The difference between the transformation matrices of two adjacent iterations is less than a threshold. ; The change in the objective function value is less than the preset tolerance; The maximum number of iterations has been reached.

[0040] (5) Probabilistic fusion denoising: Probabilistic fusion is performed on the overlapping areas of the registered vehicle point cloud template and the original point cloud. Confidence weights are assigned to the points in the vehicle point cloud template and the original point cloud; outliers with confidence scores below the threshold are removed from the overlapping areas; and the vehicle point cloud template and the original point cloud data with high confidence scores are retained to generate the scene point cloud after vehicle completion.

[0041] (6) Output the completion result: Output the fused scene point cloud as the final result of vehicle completion in the occluded area, such as... Figure 2 As shown.

[0042] Example 3, based on the same inventive concept as Example 1, introduces a laser point cloud occlusion vehicle completion device based on a laser-visual fusion deep learning framework, comprising: The acquisition module is used to acquire raw laser point cloud data and aerial images of the road scene that have been time-aligned and collected synchronously by LiDAR and UAV. The first processing module is used to identify the original laser point cloud data, identify the vehicle in front, and construct a fan-shaped occlusion area based on the geometric contour of the vehicle and the positional relationship of the laser radar. The second processing module is used to perform recognition processing on the aerial images, and to identify the two-dimensional bounding boxes, orientations, and category labels of all vehicle targets in the images; The prediction module is used to input the two-dimensional bounding boxes, orientations, and category labels of the fan-shaped occlusion area and all vehicle targets into a pre-trained dual-branch deep learning network to predict whether there are occluded vehicles and the location, orientation, and category information of the occluded vehicles. The generation module is used to select a matching vehicle point cloud template from a pre-built vehicle point cloud template library based on the category information of the occluded vehicle; and to place the vehicle point cloud template into the corresponding position in the original laser point cloud data based on the position and orientation of the occluded vehicle, thereby generating a scene point cloud after vehicle point cloud completion as the final result of vehicle completion in the occluded area.

[0043] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0044] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0045] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0046] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0047] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for completing vehicle occlusion caused by laser point clouds based on a laser-visual fusion deep learning framework, characterized in that, include: Acquire raw laser point cloud data and aerial images of road scenes that have been time-aligned and collected simultaneously by LiDAR and UAVs; The original laser point cloud data is processed to identify vehicles ahead, and a fan-shaped occlusion area is constructed based on the geometric contour of the vehicle and the positional relationship between the laser radar and the laser point cloud. The aerial images are processed to identify the two-dimensional bounding boxes, orientations, and category labels of all vehicle targets in the images; The two-dimensional bounding boxes, orientations, and category labels of the fan-shaped occlusion region and all vehicle targets are input into a pre-trained dual-branch deep learning network to predict whether there are occluded vehicles and the location, orientation, and category information of the occluded vehicles. Based on the category information of the occluded vehicle, a matching vehicle point cloud template is selected from a pre-built vehicle point cloud template library. Based on the position and orientation of the occluded vehicle, the vehicle point cloud template is placed in the corresponding position in the original laser point cloud data to generate the scene point cloud after vehicle point cloud completion as the final result of vehicle completion in the occluded area. The step of placing the vehicle point cloud template into the corresponding position in the original laser point cloud data based on the position and orientation of the occluded vehicle to generate a scene point cloud after vehicle point cloud completion includes: The vehicle point cloud template is spatially registered with the original laser point cloud data by an iterative nearest point algorithm. After spatial registration, a probabilistic fusion algorithm is used to eliminate noise in the overlapping area, forming a scene point cloud after the vehicle point cloud is completed. The process of using a probabilistic fusion algorithm to eliminate noise in overlapping regions and form a scene point cloud after vehicle point cloud completion includes: Probabilistic fusion is performed on the overlapping areas of the registered vehicle point cloud template and the original laser point cloud data, including: Based on the point cloud density and registration error, calculate the confidence level ω for assigning any i-th point in the vehicle point cloud template and the original laser point cloud data. i ; , where D i E is the point cloud density factor of the i-th point. i is the registration error factor for the i-th point, and α and β are adjustment coefficients; Points with confidence levels below the threshold in the overlapping region are considered outliers, while points with confidence levels above the threshold in the overlapping region are used as high-confidence vehicle point cloud templates and original laser point cloud data. Remove the outliers, retain the high-confidence vehicle point cloud template and the original laser point cloud data, and generate the scene point cloud after vehicle completion.

2. The laser point cloud occlusion vehicle completion method based on a laser-visual fusion deep learning framework according to claim 1, characterized in that, The process of identifying the original laser point cloud data to identify vehicles ahead includes: The original laser point cloud data is preprocessed by sequentially performing denoising, ground segmentation, and Euclidean clustering to obtain clustering results, and vehicles ahead are identified based on the clustering results.

3. The laser point cloud occlusion vehicle completion method based on a laser-visual fusion deep learning framework according to claim 1, characterized in that, The aerial image recognition process involves identifying the two-dimensional bounding boxes, orientations, and category labels of all vehicle targets in the image, including: The aerial image is input into a pre-trained YOLOv8 deep learning model, which outputs the target's two-dimensional bounding box, orientation, and category label.

4. The laser point cloud occlusion vehicle completion method based on a laser-visual fusion deep learning framework according to claim 1, characterized in that, The dual-branch deep learning network includes: a dual-branch deep learning network comprising an image feature extraction module, a point cloud feature extraction module, a feature fusion module, and a mapping regression module; The image feature extraction module is used to extract features from the two-dimensional bounding box, orientation, and category label of the vehicle target using ConvNeXt, thereby obtaining the semantic features of the detected target in the aerial image. ; The point cloud feature extraction module is used to extract the three-dimensional spatial structure features of the fan-shaped occlusion region using PointTransformer. ; The feature fusion module is used to integrate the semantic features of the detected targets in the aerial image based on a pre-constructed projection mapping matrix 𝑃. The point cloud space projected onto the fan-shaped occlusion region Ω, and the three-dimensional spatial structure features of the fan-shaped occlusion region. Alignment, assisted by cross-attention mechanism to generate joint features of image and point cloud The projection mapping matrix P is represented as follows: ； ； Among them, f x f y c is the focal length in the horizontal and vertical directions of the image; x c y K is the principal point of the image; R is the camera intrinsic parameter matrix; T is the rotation matrix used to align the point cloud coordinate system with the camera coordinate system; and T is the translation vector used to represent the relative position between the lidar and the UAV. Coordinate transformation between point cloud data and image data is performed through R and T. The mapping and regression module is used to determine the joint features of the image and the point cloud. Predict whether there are any obscured vehicles, as well as the location, orientation, and category of the obscured vehicles.

5. The laser point cloud occlusion vehicle completion method based on a laser-visual fusion deep learning framework according to claim 4, characterized in that, The training process of the dual-branch deep learning network includes: Acquire historical datasets collected synchronously by LiDAR and UAV, and use the historical datasets to construct a training set; The feature fusion module employs an attention mechanism to achieve cross-modal information alignment, as shown below: ； Where A represents the attention matrix between the image and the point cloud; W1 and W2 represent the feature mapping weight matrices; The impairment function used in the dual-branch deep learning network for: ； in, Represents the point cloud coordinates predicted from image features; This represents the coordinates of the center point of the vehicle in the real point cloud; Represents the weighting coefficient of the loss term; Based on the training set, the attention mechanism, and the damage function The dual-branch deep learning network is trained to obtain a trained dual-branch deep learning network.

6. The laser point cloud occlusion vehicle completion method based on a laser-visual fusion deep learning framework according to claim 1, characterized in that, The construction of the vehicle point cloud template library includes: Point cloud data of typical vehicles are extracted from historical LiDAR data. Euclidean clustering is used to separate the vehicle point cloud from the background based on the spatial distribution of the point cloud data of typical vehicles. The separated vehicle point clouds are then labeled with categories and stored in the vehicle point cloud template library.

7. The laser point cloud occlusion vehicle completion method based on a laser-visual fusion deep learning framework according to claim 1, characterized in that, The step of spatially registering the vehicle point cloud template with the original laser point cloud data using an iterative nearest-point algorithm includes: (1) Based on the position and orientation of the obscured vehicle, the vehicle point cloud template is initially aligned to the corresponding area of ​​the original laser point cloud data as the initial input of the iterative nearest point algorithm; (2) For each point in the vehicle point cloud template, search for the point with the closest Euclidean distance in the original laser point cloud data and establish a set of point-to-point correspondences; (3) Calculate the optimal rotation matrix R1 and translation vector T1 by minimizing the objective function. The expression for minimizing the objective function is: ； Where, q i This indicates the first [item] in the vehicle point cloud template. One point; p i This indicates that the original laser point cloud data contains data related to q. i The nearest point; denoted by , min represents the square of the Euclidean distance; (4) Apply the solved R1 and T1 to the vehicle point cloud template to update its spatial position; (5) Repeat steps (2) to (4) until one of the following convergence conditions is met: The difference between the transformation matrices of two adjacent iterations is less than a threshold. The transformation matrix is ​​a matrix composed of R1 and T1 obtained by solving the problem; The change in the value of the objective function is less than the preset tolerance; The maximum number of iterations has been reached.

8. A laser point cloud occlusion vehicle completion device based on a laser-visual fusion deep learning framework, characterized in that, include: The acquisition module is used to acquire raw laser point cloud data and aerial images of the road scene that have been time-aligned and collected synchronously by LiDAR and UAV. The first processing module is used to identify the original laser point cloud data, identify the vehicle in front, and construct a fan-shaped occlusion area based on the geometric contour of the vehicle and the positional relationship of the laser radar. The second processing module is used to perform recognition processing on the aerial images, and to identify the two-dimensional bounding boxes, orientations, and category labels of all vehicle targets in the images; The prediction module is used to input the two-dimensional bounding boxes, orientations, and category labels of the fan-shaped occlusion area and all vehicle targets into a pre-trained dual-branch deep learning network to predict whether there are occluded vehicles and the location, orientation, and category information of the occluded vehicles. The generation module is used to select a matching vehicle point cloud template from a pre-built vehicle point cloud template library based on the category information of the occluded vehicle. Based on the position and orientation of the occluded vehicle, the vehicle point cloud template is placed in the corresponding position in the original laser point cloud data to generate the scene point cloud after vehicle point cloud completion as the final result of vehicle completion in the occluded area. The step of placing the vehicle point cloud template into the corresponding position in the original laser point cloud data based on the position and orientation of the occluded vehicle to generate a scene point cloud after vehicle point cloud completion includes: The vehicle point cloud template is spatially registered with the original laser point cloud data by an iterative nearest point algorithm. After spatial registration, a probabilistic fusion algorithm is used to eliminate noise in the overlapping area, forming a scene point cloud after the vehicle point cloud is completed. The process of using a probabilistic fusion algorithm to eliminate noise in overlapping regions and form a scene point cloud after vehicle point cloud completion includes: Probabilistic fusion is performed on the overlapping areas of the registered vehicle point cloud template and the original laser point cloud data, including: Based on the point cloud density and registration error, calculate the confidence level ω for assigning any i-th point in the vehicle point cloud template and the original laser point cloud data. i ; , where D i E is the point cloud density factor of the i-th point. i is the registration error factor for the i-th point, and α and β are adjustment coefficients; Points with confidence levels below the threshold in the overlapping region are considered outliers, while points with confidence levels above the threshold in the overlapping region are used as high-confidence vehicle point cloud templates and original laser point cloud data. Remove the outliers, retain the high-confidence vehicle point cloud template and the original laser point cloud data, and generate the scene point cloud after vehicle completion.

Images