Three-dimensional reconstruction method, device and equipment and autonomous vehicle

Abstract

The disclosure provides a three-dimensional reconstruction method, device, equipment and autonomous vehicle, relates to the technical field of computers, and particularly relates to the technical field of computer vision, point cloud processing, three-dimensional reconstruction, autonomous driving perception and the like. A specific implementation scheme is as follows: time sequence data in a vehicle driving process is acquired; based on the time sequence data, target laser points associated with a target object are determined from multiple frames of continuous laser point clouds; three-dimensional reconstruction is performed based on the target laser points, and an initial three-dimensional model is obtained; in response to failure of the initial three-dimensional model reconstruction, a target object template matched with the target object is acquired from multiple preset object templates based on a target feature descriptor of the target object; the target feature descriptor is determined based on a target voxel among multiple voxels obtained by voxelizing the target laser points along a preset sampling direction and a distance between the target voxel and a bottom center point of the target object; and the initial three-dimensional model and the target object template are fused to obtain a target three-dimensional model of the target object.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of computer vision, point cloud processing, 3D reconstruction, and autonomous driving perception. It can be applied to scenarios such as autonomous driving, and in particular to a 3D reconstruction method, device, equipment, and autonomous vehicle. Background Technology

[0002] 3D reconstruction refers to the process of converting real-world objects, scenes, or environments into 3D digital models using technologies such as computer vision, graphics, and photogrammetry. It is a fundamental supporting technology for core tasks in autonomous driving, including 3D scene perception, environment modeling, and occupancy grid prediction. It transforms the real physical scenes perceived by vehicles during operation into quantifiable and computable digital 3D structures, providing accurate spatial data support for autonomous vehicle decision-making, obstacle avoidance, and scene understanding. Compared to traditional 2D perception technologies, 3D reconstruction can completely preserve multi-dimensional information such as the spatial scale, geometric shape, and positional relationships of a scene, effectively compensating for the lack of depth and incomplete spatial representation in 2D perception. It has now become an indispensable core technology module in advanced autonomous driving perception systems.

[0003] Currently, existing technologies for 3D reconstruction of the vehicle's surrounding environment using laser point clouds face numerous unresolved issues. On one hand, under vehicle operating conditions, interference from external obstacles, changes in lighting conditions, and dynamic shifts in the observation perspective directly lead to quality defects in the acquired laser point clouds, such as sparseness, missing local data, and noise superposition. On the other hand, the reconstruction methods in related technologies rely solely on pure geometric reconstruction based on the original point cloud. For irregularly shaped vehicles such as forklifts and trailers, due to their complex shapes and low local feature recognition, reconstruction failures are easily caused by model distortion and structural incompleteness, significantly reducing the reliability and robustness of 3D reconstruction. Summary of the Invention

[0004] This disclosure provides a three-dimensional reconstruction method, apparatus, electronic device, and autonomous vehicle.

[0005] According to one aspect of this disclosure, a three-dimensional reconstruction method is provided, the method comprising: acquiring temporal data during vehicle movement; wherein the temporal data includes spatiotemporally aligned multi-frame continuous laser point clouds and multi-frame continuous images, each frame of the laser point cloud including multiple laser points; determining target laser points associated with a target object from the multi-frame continuous laser point clouds based on the temporal data; wherein the target object is located in the surrounding scene of the vehicle; performing three-dimensional reconstruction based on the target laser points associated with the target object to obtain an initial three-dimensional model of the target object; in response to the failure of the initial three-dimensional model reconstruction, obtaining a target object template matching the target object from multiple preset object templates based on the target feature descriptor of the target object; wherein the target feature descriptor is determined based on the distance between the target voxel among multiple voxels obtained by voxelization of the target laser points and the bottom center point of the target object, the target voxel being determined along a preset sampling direction; and fusing the initial three-dimensional model with the target object template to obtain a target three-dimensional model of the target object.

[0006] According to another aspect of this disclosure, a three-dimensional reconstruction apparatus is provided, the apparatus comprising: a first acquisition module, configured to acquire temporal data during vehicle movement; wherein the temporal data includes spatiotemporally aligned multi-frame continuous laser point clouds and multi-frame continuous images, each frame of the laser point cloud including multiple laser points; a first determination module, configured to determine target laser points associated with a target object from the multi-frame continuous laser point clouds based on the temporal data; wherein the target object is located in the surrounding scene of the vehicle; a reconstruction module, configured to perform three-dimensional reconstruction based on the target laser points associated with the target object to obtain an initial three-dimensional model of the target object; a second acquisition module, configured to, in response to the failure of the initial three-dimensional model reconstruction, acquire a target object template matching the target object from multiple preset object templates based on the target feature descriptor of the target object; wherein the target feature descriptor is determined based on the distance between the target voxel among multiple voxels obtained by voxelization of the target laser points and the bottom center point of the target object, the target voxel being determined along a preset sampling direction; and a first fusion module, configured to fuse the initial three-dimensional model with the target object template to obtain a target three-dimensional model of the target object.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the three-dimensional reconstruction method proposed above in this disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the three-dimensional reconstruction method proposed in this disclosure above.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the three-dimensional reconstruction method proposed above.

[0010] According to another aspect of this disclosure, an autonomous vehicle is provided, including electronic devices as described in another aspect of this disclosure.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure; Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure; Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure; Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure; Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure; Figure 6 This is a schematic diagram according to the sixth embodiment of the present disclosure; Figure 7 This is a block diagram of an electronic device used to implement the three-dimensional reconstruction method of the embodiments of this disclosure. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] This disclosure presents a three-dimensional reconstruction method, apparatus, device, and autonomous vehicle.

[0016] Figure 1This is a schematic diagram based on the first embodiment of the present disclosure. It should be noted that the three-dimensional reconstruction method of the present disclosure can be applied to a three-dimensional reconstruction device, which can be configured in an electronic device so that the electronic device can perform the three-dimensional reconstruction function.

[0017] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), mobile terminals, servers, etc. Mobile terminals can be, for example, in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, smart speakers, servers, server clusters, and other hardware devices with various operating systems, touch screens and / or displays.

[0018] The 3D reconstruction device can also be software within an electronic device, such as 3D reconstruction software. In the following embodiments, an electronic device is used as an example for illustration.

[0019] like Figure 1 As shown, the three-dimensional reconstruction method may include the following steps: Step 101: Obtain time-series data during vehicle operation.

[0020] The time-series data may include spatiotemporally aligned multi-frame continuous laser point clouds and multi-frame continuous images.

[0021] Any frame of the laser point cloud may include multiple laser points. Optionally, the attribute information of the laser points may include location information, timestamps, etc., which are not limited in this disclosure.

[0022] As an example, during vehicle operation, multiple frames of continuous laser point clouds and multiple frames of continuous images can be acquired through sensors installed on the vehicle (such as LiDAR, cameras, etc.). Then, each frame of laser point cloud and its corresponding image can be spatiotemporally aligned, matching the timestamps of the laser point cloud and the image at the same acquisition time, and unifying their spatial coordinate systems to achieve temporal synchronization and spatial matching of a single frame of laser point cloud and its corresponding image, ultimately obtaining temporally correlated and spatially corresponding temporal data.

[0023] It should be noted that the above example only uses the local spatiotemporal alignment between a single frame of laser point cloud and the corresponding image. Spatiotemporal alignment can also be global spatiotemporal alignment, that is, multiple consecutive frames of laser point cloud and multiple consecutive frames of images are unified under the global coordinate system (or world coordinate system). This disclosure does not limit this.

[0024] Step 102: Based on time-series data, determine the target laser points associated with the target object from multiple frames of continuous laser point clouds.

[0025] The target object can be located in the surrounding scene of the vehicle. For example, the target object can be a three-dimensional physical obstacle such as a motor vehicle, a non-motor vehicle, or a pedestrian in the surrounding scene of the vehicle, and can appear continuously in multiple consecutive images and multiple consecutive point clouds.

[0026] The target laser point can be a set of laser points selected from multiple consecutive laser point clouds, belonging to different time frames, but physically constituting the outer contour surface of the target object.

[0027] Step 103: Perform three-dimensional reconstruction based on the target laser points associated with the target object to obtain the initial three-dimensional model of the target object.

[0028] As an example, the target laser points associated with the target object can be voxelized to obtain a three-dimensional voxel model of the target object, which can be used as the initial three-dimensional model; or, the target laser points associated with the target object can be meshed to construct a triangular mesh surface model, which can be used as the initial three-dimensional model.

[0029] It should be noted that the above examples of three-dimensional reconstruction methods are merely illustrative, and other methods may be used in practical applications. This disclosure does not impose any restrictions on these methods.

[0030] Step 104: In response to the failure of the initial 3D model reconstruction, a target object template matching the target object is obtained from multiple preset object templates based on the target feature descriptor of the target object.

[0031] The target feature descriptor can be determined based on the distance between the target voxel and the bottom center point of the target object among multiple voxels obtained by voxelization of the target laser point; the target voxel can be determined along a preset sampling direction.

[0032] The preset sampling direction can be pre-defined and used to filter target voxels based on sampling. It should be noted that this disclosure does not limit the preset sampling direction; for example, it can be along a vertical direction, a horizontal direction, a horizontal depth direction, etc. It should also be noted that this disclosure does not limit the number of target voxels; it can be one or multiple.

[0033] Optionally, in some embodiments, the target laser points associated with the target object can be traversed, the bottom point set with the lowest vertical height can be selected, and the average coordinates can be calculated based on the position information corresponding to the bottom point set, thereby obtaining the coordinates or position information of the bottom center point.

[0034] In this embodiment of the disclosure, after obtaining the target laser points associated with the target object, the target laser points can be voxelized to obtain multiple voxels. The target voxel can be determined from the multiple voxels along a preset sampling direction, and a target feature descriptor is generated based on the distance between the target voxel and the bottom center point of the target object. For example, assuming the number of target voxels is 256, the distance between each of the 256 target voxels and the bottom center point of the target object can be calculated, and a 256-dimensional vector can be generated from these distances. This vector is then used as the target feature descriptor.

[0035] The preset object template can be a pre-built standard template, which, for example, can include point cloud templates, voxel templates, etc., corresponding to standard vehicles, pedestrians, and non-motorized vehicles.

[0036] It is understandable that initial 3D model reconstruction may fail. For example, when the target laser points are sparse, the point cloud is incomplete, there is severe occlusion, or there are too many outliers, a complete and effective target 3D model cannot be constructed, leading to initial 3D model reconstruction failure. Therefore, in this embodiment of the disclosure, a target object template matching the target object can be obtained from multiple preset object templates based on the target feature descriptor of the target object.

[0037] Step 105: Fuse the initial 3D model with the target object template to obtain the target 3D model of the target object.

[0038] As an example, the target object template is aligned to the spatial coordinate system of the initial 3D model. Then, the aligned target object template is merged with the initial 3D model to fill in the missing parts of the initial 3D model, thus obtaining the target 3D model of the target object.

[0039] As another example, when the target object template is a point cloud template, the target laser points corresponding to the initial 3D model are registered with the target object template to obtain spatially aligned target laser points and target object template; the spatially aligned target object template is used to supplement the spatially aligned target laser points to obtain a fused point cloud; finally, the fused point cloud is reconstructed again to generate the target 3D model.

[0040] As an example, the technical solutions provided in this disclosure can be applied to at least the following scenarios: 1. Data production and model training for autonomous driving OCC production lines; 2. 3D occupancy reconstruction and semantic occupancy modeling for dynamic foreground targets; 3. Environmental perception and understanding of autonomous vehicles for special target objects such as irregularly shaped vehicles, engineering vehicles, and work vehicles.

[0041] As an example, the 3D reconstruction method of this disclosure is applied to an autonomous vehicle, which integrates core perception and processing hardware such as LiDAR, onboard vision camera, and onboard computing unit. During the operation of the autonomous vehicle, the onboard LiDAR continuously outputs multiple frames of LiDAR point cloud data, and the onboard vision camera simultaneously acquires multiple frames of image data. The onboard computing unit performs spatiotemporal registration and temporal alignment on the two types of data to generate temporal data containing multiple consecutive frames of laser point clouds and multiple consecutive frames of images. Then, based on the aligned temporal data, the onboard computing unit selects target laser points corresponding to target objects in the surrounding scene of the vehicle from the multiple consecutive frames of laser point clouds, and uses the selected target laser points to perform 3D reconstruction to generate an initial 3D model of the target object. Subsequently, the onboard computing unit can verify the initial 3D model. If it is determined that the initial 3D model reconstruction fails, the onboard computing unit performs voxelization on the target laser points, selects target voxels along a preset sampling direction, and calculates the target feature descriptor of the target object by combining the distance between the target voxel and the bottom center point of the target object. Then, using the target feature descriptor as the matching basis, the unit retrieves the target object template with the highest matching degree from the locally pre-stored point cloud template library of various vehicles and obstacles, and fuses and optimizes the initial 3D model that failed to be reconstructed with the matched target object template to generate a target 3D model with complete structure and accurate shape. This solution is adaptable to all driving conditions for autonomous vehicles, including factory operations, park traffic, and urban road driving. It is particularly suitable for operating environments such as factories and logistics parks with a large number of irregularly shaped vehicles such as forklifts and trailers. The high-precision 3D model output by this solution helps autonomous vehicles accurately perceive the size, posture, and spatial position of surrounding irregularly shaped vehicles, effectively solving the recognition bias problem caused by missing point clouds and insufficient features, and improving the obstacle detection accuracy, driving safety, and rationality of traffic decisions for autonomous vehicles.

[0042] The 3D reconstruction method of this disclosure acquires temporal data during vehicle movement. The temporal data includes spatiotemporally aligned multi-frame continuous laser point clouds and multi-frame continuous images, with each frame of the laser point cloud including multiple laser points. Based on the temporal data, target laser points associated with a target object are determined from the multi-frame continuous laser point clouds. The target object is located in the surrounding scene of the vehicle. 3D reconstruction is performed based on the target laser points associated with the target object to obtain an initial 3D model of the target object. In response to the failure of the initial 3D model reconstruction, a target object template matching the target object is obtained from multiple preset object templates based on the target object's target feature descriptor. The target feature descriptor is determined based on the distance between the target voxel and the bottom center point of the target object among multiple voxels obtained by voxelizing the target laser points, and the target voxel is determined along a preset sampling direction. The initial 3D model and the target object template are fused to obtain the target 3D model of the target object. Therefore, by utilizing spatiotemporally aligned multi-frame continuous laser point clouds and multi-frame continuous images acquired during vehicle operation, point cloud data of the target object can be fully accumulated from temporal data, effectively overcoming the information deficiency caused by sparse, occluded, and structurally missing single-frame point clouds, and improving the completeness and accuracy of target laser point extraction. Simultaneously, when the initial 3D model reconstruction fails, by voxelizing the target laser points, selecting target voxels based on preset sampling directions, and constructing corresponding target feature descriptors based on the distance between the voxel and the bottom center point of the target object, the spatial structure and shape features of the target object can be accurately depicted. For example, it can effectively depict the differentiated key structural features of irregularly shaped target objects such as pump truck booms, mixer truck tanks, forklift forks, and multi-section trailer bodies, achieving efficient and robust matching with preset object templates. Then, the reconstructed... The fusion and complementation of failed initial 3D models with standard templates can preserve the true posture and detailed features of real-world target objects, while using templates to fill in missing structures and correct morphological deviations, thereby improving the integrity, contour accuracy, and effectiveness of 3D models. This helps to enhance the fault tolerance and operational stability of vehicle-mounted 3D reconstruction in complex road conditions in autonomous driving scenarios. It can effectively solve the problems of contour distortion, structural incompleteness, and morphological deviation that easily occur in the reconstruction of irregular target objects in related technologies, improve the integrity, geometric accuracy, and morphological restoration of irregular vehicle 3D models, enhance the fault tolerance and robustness of irregular target reconstruction in complex driving scenarios, and improve the geometric accuracy and morphological restoration effect of irregular vehicle models. It can provide reliable 3D data support for intelligent driving to accurately perceive irregular obstacles, calculate vehicle size, judge behavior, and make driving decisions.

[0043] like Figure 2 As shown, this disclosure also proposes a three-dimensional reconstruction method. Figure 2 The diagram is based on the second embodiment of this disclosure. Based on any embodiment of this disclosure, the three-dimensional reconstruction method may further include the following steps: Step 201: Voxelize the target laser points associated with the target object to obtain multiple voxels.

[0044] Step 202: Determine the bottom center point of the target object based on the position information of the target laser point.

[0045] As an example, one can iterate through the target laser points associated with the target object, filter out the bottom point set with the lowest vertical height, and calculate the average coordinates based on the position information corresponding to the bottom point set, thereby obtaining the coordinates or position information of the bottom center point.

[0046] Step 203: Construct multiple sampling rays in a hemispherical space with the bottom center point as the origin and a radius of a set distance to determine the preset sampling direction.

[0047] The distance can be preset, such as 1 meter, 1.5 meters, etc., and this disclosure does not limit it.

[0048] In this embodiment of the disclosure, multiple sampling rays can be constructed in a hemispherical space with the bottom center point as the origin and a radius of a set distance, and the ray direction corresponding to any sampling ray can be determined as a preset sampling direction.

[0049] As an example, assuming the target object is a vehicle, we can construct 20 layers of uniformly distributed sampling rays, totaling 256 rays, within a unit hemisphere space, with the bottom center point of the vehicle as the origin. Here, the i-th ray r... i By direction vector L i =( , , ), where θ i Let be the pitch angle corresponding to the i-th ray. Let be the azimuth angle corresponding to the i-th ray, where i ∈ [1, 256]. Furthermore, the ray direction corresponding to any of the above sampling rays can be determined as the preset sampling direction. It should be noted that the above example of the number of constructed sampling rays is merely exemplary; in practical applications, the number of sampling rays can be other than those specified in this disclosure.

[0050] Step 204: For any sampling ray, starting from the origin, collect the density of non-empty voxels and the corresponding center-to-center distance along the sampling ray with a set step size.

[0051] Density can be used to indicate the number of target laser points in a corresponding voxel.

[0052] Among them, the center distance can be used to indicate the distance between the corresponding voxel and the origin. For example, the distance between the center point of a voxel and the origin can be determined as the center distance of the voxel.

[0053] The step size can be preset and used to indicate the fixed sampling interval along the camera defense line corresponding to the sampling ray. This disclosure does not limit its value.

[0054] Among them, a non-empty voxel refers to a solid voxel that includes at least one target laser point.

[0055] In this embodiment of the present disclosure, for any sampling ray, the origin corresponding to the bottom center point can be used as the sampling starting point, and the sampling space can be traversed step by step along the ray direction corresponding to the sampling ray according to a preset step size, so as to filter out the non-empty voxels along the way, and collect the density of each non-empty voxel and the corresponding center-to-center distance.

[0056] Step 205: The voxel with the highest density among the non-empty voxels collected along the sampling ray is taken as the target voxel, and the distance from the center of the target voxel to the sphere is taken as the feature distance corresponding to the sampling ray.

[0057] As an example, suppose the non-empty voxels collected along the sampling ray r include voxel 1, voxel 2, voxel 3 and voxel 4, among which voxel 3 has the highest density. Then voxel 3 can be taken as the target voxel, and the distance to the center of the sphere corresponding to voxel 3 can be taken as the feature distance corresponding to the sampling ray.

[0058] Step 206: Fuse the feature distances corresponding to multiple sampled rays to obtain the target feature descriptor.

[0059] As an example, suppose there are n sampled rays, and the feature distance corresponding to the i-th sampled ray is di. Then, by fusing the feature distances corresponding to the multiple sampled rays, the target feature descriptor is obtained as [d1, d2, ..., d]. i , ..., d n ], where i∈[1,n], and n is a non-zero natural number; that is, the target feature descriptor can include values ​​of n dimensions.

[0060] The three-dimensional reconstruction method of this disclosure involves voxelizing the target laser points associated with the target object to obtain multiple voxels; determining the bottom center point of the target object based on the position information of the target laser points; constructing multiple sampling rays in a hemispherical space with the bottom center point as the origin and a radius of a set distance to determine a preset sampling direction; for any sampling ray, collecting the density of non-empty voxels and the corresponding center-of-sphere distance along the sampling ray with the origin as the starting point and a set step size; wherein, the density is used to indicate the number of laser points in the corresponding voxel, and the center-of-sphere distance is used to indicate the distance between the corresponding voxel and the origin; taking the voxel with the highest density among the non-empty voxels collected along the sampling ray as the target voxel, and taking the center-of-sphere distance corresponding to the target voxel as the feature distance corresponding to the sampling ray; and fusing the feature distances corresponding to multiple sampling rays to obtain a target feature descriptor. Therefore, by relying on voxelization processing combined with hemispherical spatial sampling rays to complete feature extraction, a sampling space is constructed with the bottom center point of the target object and the density and distance information of voxels are collected at a fixed step size. The voxel with the highest density is selected as the target voxel and the feature distance is extracted. Finally, multiple sets of feature distances are fused to form a target feature descriptor, which can accurately represent the spatial distribution, shape contour and three-dimensional structure of the target object from multiple angles and levels. It has high feature discrimination and strong anti-interference ability, and can effectively adapt to the structural feature expression needs of various irregular targets, improve the accuracy and stability of subsequent template matching, and provide a highly reliable feature basis for the three-dimensional reconstruction, model matching and fusion optimization of target objects in complex scenes.

[0061] like Figure 3 As shown, this disclosure also proposes a three-dimensional reconstruction method. Figure 3 The diagram is based on a third embodiment of the present disclosure. Based on any embodiment of the present disclosure, the three-dimensional reconstruction method may further include the following steps: Step 301: For any laser point cloud, voxelize the target laser points associated with the target object in the laser point cloud to obtain multiple voxels of the target object under the laser point cloud.

[0062] In this embodiment of the disclosure, the target laser points associated with the target object can be summarized as the laser points associated with the target object in multiple consecutive laser point clouds. Therefore, for any frame of laser point cloud, the target laser points associated with the target object in that frame of laser point cloud can be voxelized to obtain multiple voxels of the target object in that frame of laser point cloud.

[0063] Step 302: Based on the position information of the target laser points associated with the target object in the laser point cloud, determine the bottom center point of the target object under the laser point cloud.

[0064] It should be noted that the implementation process of step 302 is similar to that of step 202, and will not be described in detail here.

[0065] Step 303: Based on multiple voxels of the target object in the laser point cloud, determine the sub-feature descriptor of the target object in the laser point cloud.

[0066] The sub-feature descriptor can be determined based on the distance between the target voxel determined along a preset sampling direction among multiple voxels of the target object under the laser point cloud and the bottom center point of the target object under the laser point cloud.

[0067] It should be noted that the explanation of the preset sampling direction and target voxel in step 104 also applies to this embodiment, and will not be repeated here.

[0068] As one possible implementation, multiple sampling rays are constructed in a hemispherical space with the bottom center point of the target object under the laser point cloud as the origin and a radius of a set distance to determine the preset sampling direction. For any sampling ray, starting from the origin, the density of non-empty voxels and the corresponding center-to-center distance are collected along the sampling ray with a set step size. Here, density is used to indicate the number of target laser points in the corresponding voxel, and center-to-center distance is used to indicate the distance between the corresponding voxel and the origin. The voxel with the highest density among the non-empty voxels collected along the sampling ray is taken as the target voxel, and the center-to-center distance of the target voxel is taken as the feature distance corresponding to the sampling ray. The feature distances corresponding to multiple sampling rays are fused to obtain the sub-feature descriptor of the target object under the laser point cloud. It should be noted that the implementation principle of this process is similar to the corresponding processing process in the aforementioned embodiment, and will not be repeated here.

[0069] Step 304: Fuse the sub-feature descriptors corresponding to the target object under multiple consecutive laser point clouds to obtain the target feature descriptor.

[0070] As one possible implementation, when any sub-feature descriptor includes values ​​of multiple dimensions, multiple keyframe laser point clouds are determined from multiple consecutive laser point clouds; for any dimension, the maximum value of the sub-feature descriptor corresponding to the target object in the multiple keyframe laser point clouds is determined as the target value corresponding to that dimension; based on the target values ​​corresponding to multiple dimensions, a target feature descriptor is generated.

[0071] As an example, multiple consecutive laser point clouds are filtered to remove invalid frames that are overexposed, severely occluded, or have missing targets, retaining frames with complete outlines and excellent point cloud quality as keyframe laser point clouds. Assuming there are k keyframe laser point clouds, and any sub-feature descriptor includes n dimensions, then the sub-feature descriptor d of the target object in the i-th keyframe laser point cloud is... i for: d i =[d i1 di2 , ..., d ij , ..., d in ];(1) Here, represents the value of the sub-feature descriptor of the target object in the i-th keyframe laser point cloud in the j-th dimension, i∈[1,k], j∈[1,n]; then for the j-th dimension, the maximum value can be taken from the values ​​of the sub-feature descriptors of the target object in the j-th dimension corresponding to the k keyframe laser point clouds, that is, d. 1j d 2j ...d kj The maximum value among these k values ​​is taken as the target value corresponding to the j-th dimension; finally, the target values ​​corresponding to the n dimensions are used to generate the target feature descriptor.

[0072] Therefore, by selecting keyframes and taking the maximum value according to the dimension to complete feature fusion, the typical structural features of irregular target objects such as pump truck boom, mixer truck tank, forklift fork arm, and trailer multi-section body can be fully preserved, the feature distortion caused by occlusion, viewpoint deviation and local point cloud missing can be weakened, the structural differences of different irregular vehicles can be highlighted to the maximum extent, the stability and recognizability of feature descriptors can be improved, and the overall effect of template matching and 3D reconstruction under complex working conditions can be improved.

[0073] The 3D reconstruction method of this disclosure involves voxelizing the target laser points associated with the target object in any laser point cloud to obtain multiple voxels of the target object in the laser point cloud; determining the bottom center point of the target object in the laser point cloud based on the position information of the target laser points associated with the target object in the laser point cloud; determining a sub-feature descriptor of the target object in the laser point cloud based on the multiple voxels of the target object in the laser point cloud; wherein the sub-feature descriptor is determined based on the distance between the target voxel determined along a preset sampling direction and the bottom center point of the target object in the laser point cloud; and fusing the sub-feature descriptors corresponding to the target object in multiple consecutive frames of laser point clouds to obtain a target feature descriptor. Therefore, by first generating sub-feature descriptors based on the voxel distribution and bottom center point distance of a single-frame laser point cloud, and then fusing the sub-feature descriptors corresponding to multiple consecutive frames of laser point clouds to obtain the target feature descriptor, the feature descriptor can be obtained. This fully integrates point cloud features from different perspectives and at different times, effectively avoiding the problem of insufficient feature representation caused by single-frame point cloud perspective, limited perspective, and local occlusion. It can completely restore the differentiated structural features of irregular target objects such as pump truck booms, mixer truck tanks, forklift forks, and multi-section trailer bodies, enhance the anti-interference ability and discriminability of the target feature descriptor, improve the accuracy and stability of template matching, and help improve the 3D reconstruction effect of irregular target objects.

[0074] like Figure 4As shown, this disclosure also proposes a three-dimensional reconstruction method. Figure 4 As shown in the schematic diagram of the fourth embodiment of this disclosure, the three-dimensional reconstruction method may include the following steps: Step 401: Obtain time-series data during vehicle operation.

[0075] Step 402: Based on time-series data, determine the target laser points associated with the target object from multiple frames of continuous laser point clouds.

[0076] Step 403: Perform three-dimensional reconstruction based on the target laser points associated with the target object to obtain the initial three-dimensional model of the target object.

[0077] It should be noted that the execution process of steps 401 to 403 can refer to the execution process of any embodiment of this disclosure, and will not be repeated here.

[0078] Step 404: In response to the failure of the initial 3D model reconstruction, for any preset object template, determine the descriptor similarity between the preset object template and the target object based on the template feature descriptor and the target feature descriptor corresponding to the preset object template.

[0079] It should be noted that the explanations of the initial 3D model reconstruction failure, the preset object model, and the target feature descriptor in step 104 also apply to this embodiment, and will not be repeated here.

[0080] In this embodiment of the disclosure, any preset object template may have a corresponding template feature descriptor. As an example, the method for obtaining the template feature descriptor corresponding to the preset object template may be similar to the method for obtaining the target feature descriptor, and will not be described in detail here.

[0081] Among them, descriptor similarity can be used to measure the similarity between the template feature descriptor corresponding to the preset object template and the target feature descriptor.

[0082] To obtain descriptor similarity, when both the target feature descriptor and the template feature descriptor include values ​​in multiple dimensions, one possible approach is to use the following steps 4041 to 4044: Step 4041: For any dimension, determine the weight corresponding to the dimension based on the values ​​of the template feature descriptor and the target feature descriptor corresponding to the preset object template in the dimension.

[0083] As an example, for any dimension, when the value of the template feature descriptor corresponding to the preset object template in that dimension is not 0, and the value of the target feature descriptor in that dimension is not 0, then the weight corresponding to that dimension is determined to be a first set value (e.g., 1); when the value of the template feature descriptor corresponding to the preset object template in that dimension is 0 or the value of the target feature descriptor in that dimension is 0, then the weight corresponding to that dimension is determined to be a second set value (e.g., 0), wherein the first set value is greater than the second set value.

[0084] Step 4042: Based on the weights corresponding to the dimensions and the difference between the values ​​of the template feature descriptors and the target feature descriptors corresponding to the dimensions, determine the first coefficient corresponding to the dimensions.

[0085] As an example, suppose both the target feature descriptor and the template feature descriptor include values ​​in n dimensions. For the i-th dimension, i∈[1,n], the weight corresponding to the i-th dimension is m. i The template feature descriptor takes the value a in dimension . i The target feature descriptor takes the value b in the corresponding dimension. i Then the difference between the value of the template feature descriptor in dimension 1 and the value of the target feature descriptor in dimension 2 is (a) i -b i Then, the first coefficient s corresponding to the i-th dimension can be determined according to the following formula. i : (2) Step 4043: Fuse the first coefficients corresponding to multiple dimensions to obtain the second coefficient.

[0086] Using the example above, let's assume the first coefficient s corresponds to the i-th dimension. i As shown in formula (2), the first coefficients corresponding to the n dimensions can be fused according to the following formula to obtain the second coefficient S: (3) Step 4044: Determine the descriptor similarity based on the second coefficient.

[0087] Using the example above, assuming the second coefficient is S, the descriptor similarity C can be determined using the following formula: C = 1 / S; (4) Therefore, based on the values ​​of the target feature descriptor and the template feature descriptor in each dimension, the weights of the corresponding dimensions are calculated, and the corresponding coefficients are obtained by combining the differences in the values ​​of each dimension. Then, the descriptor similarity is solved by fusion operation. The weights can be allocated differently according to the importance of different dimensions in representing the structure of the target object. This helps to accurately capture the key structural differences of irregular target objects such as pump truck booms, mixer tanks, and forklift forks, weaken the interference of secondary dimensions, and take into account the influence differences of features in each dimension. It can accurately quantify the degree of matching between the target object and the preset object template, effectively improve the rationality and accuracy of the descriptor similarity, and thus improve the accuracy and stability of template matching.

[0088] Step 405: Based on the descriptor similarity of each preset object template, determine the target object template from multiple preset object templates.

[0089] As an example, the preset object template with the highest descriptor similarity among multiple preset object templates can be identified as the target object template.

[0090] Understandably, the selected target object template needs to be similar in size to the target object. Therefore, as a possible implementation, based on the initial 3D model, a first compact bounding box corresponding to the target object is constructed, and the first volume size of the first compact bounding box is determined; for any preset object template, the second volume size of the second compact bounding box corresponding to the preset object template is determined; in response to the difference between the first volume size and the second volume size being less than a first set threshold, the preset object template is used as a candidate object template; based on the descriptor similarity corresponding to at least one candidate object template, the target object template is determined from at least one candidate object template.

[0091] The first compact box can be the minimum bounding box for generating the target object based on the initial 3D model, and the first compact box can have corresponding length, width, and height.

[0092] It should be noted that a second compact box corresponding to any preset object template can be pre-constructed, and the second compact box can be the minimum bounding box of the first preset object template, and the second compact box can have corresponding length, width and height.

[0093] The first set threshold can be preset, and this disclosure does not restrict its value.

[0094] As an example, based on the initial 3D model, a first compact bounding box corresponding to the target object is constructed, and the first volume size of the first compact bounding box is determined based on its length, width, and height. For any preset object template, the second volume size of the second compact bounding box corresponding to the preset object template is determined based on its length, width, and height. When the difference between the first volume size and the second volume size is less than a first set threshold, it indicates that the sizes of the first compact bounding box and the second compact bounding box are similar, and the preset object template can be used as a candidate object template. Finally, the candidate object template with the highest descriptor similarity among the candidate object templates obtained above can be determined as the target object template.

[0095] Therefore, the volume of the compact box is used to initially screen the preset object templates, filtering out preset object templates with excessive size differences. Then, the descriptor similarity is combined to complete the final matching, forming a two-layer verification logic. This can quickly filter out irrelevant templates with mismatched size and reduce computational overhead, and accurately match templates corresponding to irregularly shaped target objects such as pump trucks, mixer trucks, forklifts, and trailers based on structural features. This effectively avoids matching errors caused by misjudgment of overall size and improves the reliability of template selection and overall computational efficiency in complex scenarios.

[0096] It is understandable that there may be situations where the object type of the candidate object template does not match (or is different from) the object type of the target object. As another possible implementation, if the target object has a corresponding object type and any preset object template has a corresponding object type, after determining the candidate object templates, the candidate object templates whose object type matches (or is the same as) the target object can be determined as reference object templates; then, the reference object template with the highest descriptor similarity among the aforementioned reference object templates is determined as the target object template.

[0097] The object type can include, but is not limited to, pedestrians, vehicles, etc., and can be set according to the actual situation. This disclosure does not impose any restrictions on this.

[0098] Therefore, based on the candidate templates obtained from the initial volume screening, a second screening is performed by combining the object type corresponding to the target object to lock in the reference object templates that match the type. Then, the template with the highest descriptor similarity is selected as the final target object template. Through multi-level joint verification of volume, type, and feature similarity, the matching range is narrowed down layer by layer, reducing the risk of mismatch between cross-type templates and further improving the accuracy and reliability of template selection.

[0099] Step 406: Fuse the initial 3D model with the target object template to obtain the target 3D model of the target object.

[0100] It should be noted that the execution process of step 406 can refer to the execution process of any embodiment of this disclosure, and will not be repeated here.

[0101] The 3D reconstruction method of this disclosure determines the descriptor similarity between a preset object template and a target object based on the template feature descriptor and the target feature descriptor corresponding to the preset object template for any given preset object template; and determines the target object template from multiple preset object templates based on the descriptor similarity corresponding to each preset object template. Therefore, by relying on feature descriptor quantification to calculate similarity and selecting the best preset object template, the structural differences of different irregularly shaped target objects can be accurately distinguished, and object templates suitable for irregularly shaped target objects such as pump trucks, mixer trucks, forklifts, and multi-section trailers can be quickly matched, avoiding template mismatch problems, improving the accuracy and reliability of template selection, and contributing to improved subsequent model fusion effects and 3D reconstruction quality.

[0102] like Figure 5 As shown, this disclosure also proposes a three-dimensional reconstruction method. Figure 5 The schematic diagram is based on the fifth embodiment of this disclosure. The three-dimensional reconstruction method may include the following steps: Step 501: Obtain time-series data during vehicle operation.

[0103] It should be noted that the execution process of step 501 can refer to the execution process of any embodiment of this disclosure, and will not be repeated here.

[0104] Step 502: For any frame of laser point cloud, generate point cloud semantic information and target detection box corresponding to the laser point cloud based on the laser point cloud and the corresponding image.

[0105] The target detection bounding box can be used to define the target object and its corresponding laser point. It should be noted that the target detection bounding box can be a 3D bounding box.

[0106] Optionally, in some embodiments, the target detection box may have corresponding attribute information.

[0107] The attribute information of the target detection box may include, but is not limited to: the object category to which the target object bound by the target detection box belongs, the size information of the target detection box, the confidence level, the position information of the target detection box (such as the length, width, height of the target detection box, the three-dimensional coordinates of the center point of the target detection box, etc.), etc., and this disclosure does not limit it.

[0108] The object categories can include, but are not limited to, vehicles, pedestrians, non-motorized vehicles, etc., and can be set as needed.

[0109] Among them, the image corresponding to the laser point cloud can be an image that is spatiotemporally aligned with the laser point cloud.

[0110] Among them, point cloud semantic information can be used to indicate the object category to which any laser point in the corresponding frame of laser point cloud belongs.

[0111] As an example, for any frame of laser point cloud, the laser point cloud and the corresponding image can be input into the BEV (Bird's Eye View) detection model and the point cloud semantic segmentation model, respectively, and the 3D target detection box output by the BEV detection model and the point cloud semantic information output by the point cloud semantic segmentation model can be obtained.

[0112] Step 503: Based on point cloud semantic information and / or target detection boxes, extract candidate laser points associated with the target object from the laser point cloud.

[0113] To extract candidate laser points associated with a target object from a laser point cloud, one possible approach is to use the following steps 5031 to 5033: Step 5031: Use the laser point cloud as the laser point cloud of the current frame.

[0114] Step 5032: Determine the motion state of the target object based on the position information of the target detection box corresponding to the target object in multiple frames of continuous laser point cloud.

[0115] The motion state can be static or dynamic.

[0116] As one possible implementation, based on the position information of multiple consecutive laser point clouds, a laser scene flow model is used to determine the inter-frame displacement information of any laser point in any frame of laser point cloud; for any frame of laser point cloud, based on the position information of the target object's target detection box under that frame of laser point cloud, the seventh laser point located within the target detection box of that frame of laser point cloud is determined; furthermore, based on the inter-frame displacement information of the seventh laser point located within the target detection box of the target object under the corresponding frame of laser point cloud, the motion state of the target object is determined.

[0117] The inter-frame displacement information may include, but is not limited to, three-dimensional displacement components (e.g., represented by (Δx, Δy, Δz)) and their corresponding confidence levels, etc.

[0118] As an example, for any frame of laser point cloud, a laser scene flow model (such as the Flow4D model) can be used to infer the three-dimensional laser scene flow information of the laser points in that frame of laser point cloud, that is, the three-dimensional displacement components of any laser point in that frame of laser point cloud from that frame of laser point cloud to the next frame of laser point cloud, and the above information is determined as the inter-frame displacement information of the corresponding laser point in that frame of laser point cloud; for any frame of laser point cloud, based on the position information of the target detection box of the target object under that frame of laser point cloud, multiple laser points located within the target detection box of the target object under that frame of laser point cloud are selected from that frame of laser point cloud, and are denoted as the seventh laser point in this disclosure; then, based on the inter-frame displacement information of any seventh laser point located within the target detection box of the target object under that frame of laser point cloud, assuming that the inter-frame displacement information is (Δx, Δy, Δz), the inter-frame displacement distance d of the corresponding seventh laser point can be determined according to the following formula: (5) Then, the average inter-frame displacement distance of multiple seventh laser points located within the target detection box of the target object in the laser point cloud of that frame can be calculated to obtain the average inter-frame displacement distance of the target object in the laser point cloud of that frame; the average inter-frame displacement distance of the target object in each frame of the laser point cloud is summed to obtain the first target displacement distance; when the first target displacement distance is greater than the first set distance threshold, the motion state of the target object is determined to be dynamic, otherwise, when the first target displacement distance is not greater than the first set distance threshold, the motion state of the target object is determined to be static.

[0119] As another possible implementation, the second target displacement distance of the target object is determined based on the position information of the target detection box corresponding to the target object in the multi-frame continuous laser point cloud. When the second target displacement distance is greater than the second set distance threshold, the motion state of the target object is determined to be dynamic; otherwise, when the second target displacement distance is not greater than the second set distance threshold, the motion state of the target object is determined to be static.

[0120] As an example, for any two consecutive laser point clouds, based on the position information of the target detection box corresponding to the target object in each of the two consecutive laser point clouds, the moving distance of the target object between the two consecutive laser point clouds is determined; the moving distances of the target object between adjacent laser point clouds in multiple consecutive laser point clouds are summed to obtain the second target displacement distance; when the second target displacement distance is greater than the second set distance threshold, the motion state of the target object is determined to be dynamic; otherwise, when the second target displacement distance is not greater than the second set distance threshold, the motion state of the target object is determined to be static.

[0121] Step 5033: Based on the motion state, point cloud semantic information and / or target detection boxes, extract candidate laser points associated with the target object from the laser point cloud of the current frame.

[0122] As one possible implementation, when the motion state is static, candidate laser points can be extracted from the current frame's laser point cloud based on the point cloud semantic information and target detection bounding box. Therefore, for stationary target objects, candidate laser points can be extracted directly using the current frame's point cloud semantic information and target detection bounding box. This adapts to the data distribution characteristics of static target objects, reduces redundant computation caused by multi-frame data fusion, and further improves the overall algorithm's operating efficiency while accurately preserving the point cloud corresponding to the key structures of irregularly shaped vehicles.

[0123] As an example, the laser points other than the first laser point in the laser point cloud of the current frame are identified as the second laser point; wherein the first laser point is located within the target detection box of the target object in the laser point cloud of the current frame; and candidate laser points are determined based on the laser points in the second laser point whose point cloud semantic information is consistent with the point cloud semantic information corresponding to the first laser point, as well as the first laser point.

[0124] For example, a laser point located within the target detection box of the target object in the current frame laser point cloud is identified as the first laser point, and the laser points other than the first laser point among the multiple laser points in the current frame laser point cloud are identified as the second laser points; when the point cloud semantic information includes the object category, the laser point in the second laser point whose object category is consistent with the object category corresponding to the first laser point is identified as the eighth laser point, and both the eighth laser point and the first laser point are identified as candidate laser points associated with the target object in the current frame laser point cloud.

[0125] For example, the spatial distance between the eighth laser point and the target object in the target detection box of the laser point cloud in the current frame can also be determined, and the laser points in the eighth laser point whose spatial distance is less than the third set distance threshold, as well as the first laser point, are all determined as candidate laser points associated with the target object in the laser point cloud of the current frame.

[0126] Therefore, by using the first laser point within the target detection box as a reference, second laser points with the same semantics are selected and together they form candidate laser points. This can completely collect the laser points of the extended structure and peripheral area of ​​irregularly shaped target objects such as pump truck booms, mixer truck tanks, and forklift forks, preventing the loss of special structure points, improving the comprehensiveness and completeness of candidate laser point extraction, and providing complete data support for subsequent feature calculation and 3D reconstruction.

[0127] As another possible implementation, when any laser point has a corresponding velocity vector, and the motion state is dynamic, multiple laser points in the current frame laser point cloud are clustered based on the velocity vectors corresponding to multiple laser points in the current frame laser point cloud to obtain at least one target cluster; and candidate laser points are extracted from the current frame laser point cloud based on the target detection box of the target object in the current frame laser point cloud and at least one target cluster.

[0128] Among them, the velocity vector can be used to indicate the magnitude and direction of the spatial displacement of the corresponding laser point per unit time. It is a three-dimensional vector that includes both magnitude and direction, and can also be called a motion vector.

[0129] To obtain the velocity vector, as an example, for any laser point, the inter-frame displacement information of that laser point can be obtained, and the velocity vector of that laser point can be determined based on the inter-frame displacement information and the time interval between two adjacent frames. For example, assuming the inter-frame displacement information of the laser point is (Δx, Δy, Δz), and the time interval between two adjacent frames is Δt, the velocity vector f of the laser point can be determined according to the following formula: (6) Among them, v x v y v z This represents the velocity component of the corresponding laser point in the corresponding direction in three-dimensional space.

[0130] To achieve clustering of multiple laser points in the current frame's laser point cloud, one possible approach is to traverse multiple laser points in the current frame's laser point cloud. For any traversed laser point, it can be designated as the third laser point. Given that the third laser point is determined to be the core point, an initial cluster is constructed based on the third laser point and its spatiotemporal nearest neighbor set. This initial cluster is then processed at least once. During any of these processing steps, a fourth laser point in the current initial cluster is traversed. If the spatiotemporal nearest neighbor set of the fourth laser point has a difference with the current initial cluster, the set of its spatiotemporal nearest neighbors is merged into the current initial cluster, resulting in the next initial cluster. The fourth laser point is the core point and is a laser point that has not been traversed. If the fourth laser point does not exist in the current initial cluster, or if the spatiotemporal nearest neighbor set of the fourth laser point has no difference with the current initial cluster, the current initial cluster is determined as the target cluster.

[0131] Among them, the core point can be determined based on the number of laser points in the spatiotemporal nearest neighbor set of the corresponding laser point.

[0132] Among them, the spatial distance between the laser point in the spatiotemporal nearest neighbor set and the third laser point is less than the second set threshold, and the difference in motion vector between them and the third laser point is less than the third set threshold.

[0133] The motion vector difference can be determined based on the velocity vectors corresponding to the two laser points.

[0134] As an example, for the i-th laser point and the j-th laser point in the current frame's laser point cloud, the following formula can be used to determine that the i-th laser point and the j-th laser point are spatiotemporally nearest neighbors: (7) Where, neighbor(i,j) indicates that the i-th laser point and the j-th laser point are spatiotemporally nearest neighbors; p i p represents the position information of the i-th laser point. j f represents the position information of the j-th laser point; i f represents the velocity vector corresponding to the i-th laser point; j This represents the velocity vector corresponding to the j-th laser point; This represents the spatial distance between the i-th laser point and the j-th laser point; ε represents the difference in motion vectors between the i-th laser point and the j-th laser point; s Indicates the second set threshold; ε f This represents the third set threshold. Therefore, the spatiotemporal nearest neighbor set of any laser point refers to the set of laser points that are spatiotemporally nearest to that laser point. If the spatial distance between the laser points in the spatiotemporal nearest neighbor set of the third laser point and the third laser point is less than the second set threshold, and the difference in motion vector between them and the third laser point is less than the third set threshold, then the laser points in the spatiotemporal nearest neighbor set of the third laser point are spatiotemporally nearest to the third laser point.

[0135] Furthermore, for example, when the number of laser points in the spatiotemporal nearest neighbor set of a laser point is greater than or equal to a second preset threshold, the laser point is determined to be a core point.

[0136] As an example, suppose the current frame's laser point cloud includes laser points a, b, c, d, e, and f. Traverse all laser points in the current frame's laser point cloud, with laser points a, b, and e as core points. Assume that laser point a is encountered for the first time and is designated as the third laser point. When laser point a is determined to be a core point, assume its spatiotemporal nearest neighbor set includes laser points b and d. Based on laser point a and its spatiotemporal nearest neighbors b and d, construct an initial cluster including laser points a, b, and d. Then, perform at least one processing step on this initial cluster. During the first processing step, traverse the initial cluster containing laser point b that is a core point but has not yet been traversed. If the set of spatiotemporal nearest neighbors of laser point b has a difference with the initial cluster of this processing, the set of spatiotemporal nearest neighbors of laser point b is merged into the initial cluster of this processing to obtain the initial cluster of the next processing. For example, if the set of spatiotemporal nearest neighbors of laser point b includes laser point a, laser point e, and laser point f, then the set of spatiotemporal nearest neighbors of laser point b is merged into the initial cluster of this processing, and the initial cluster of the next processing includes laser points a, b, d, e, and f. In the second processing, laser point e, which is a core point in the initial cluster of this processing and has not been traversed, is traversed. If the set of spatiotemporal nearest neighbors of laser point e has no difference with the initial cluster of this processing, the initial cluster of this processing is determined as the target cluster.

[0137] Therefore, dynamic laser point clustering is achieved by determining core points and combining iterative expansion of spatiotemporal nearest neighbor sets. Relying on the spatiotemporal correlation characteristics of laser points, effective nearest neighbors are continuously absorbed, and complete target clusters are gradually generated iteratively. This method can accurately aggregate the global motion laser points of dynamic irregular target objects. It can effectively adapt to the characteristics of irregular target structures such as pump trucks, mixer trucks, forklifts, and multi-section trailers, which are scattered, have large point cloud spans, and have varied motion postures. It avoids the problems of structural fragmentation, missing point cloud clustering, and mis-clustering of noisy points that are prone to occur in fixed threshold clustering in related technologies. It can completely preserve the motion point cloud corresponding to the differentiated key structures of irregular target objects such as booms, tanks, forks, and multi-section vehicles, accurately distinguish the point cloud of dynamic target objects from background noise and static interference point clouds, and improve the integrity, continuity, and accuracy of target clustering in dynamic scenes. This provides reliable clustering results to support the subsequent extraction of candidate laser points, feature construction, and high-precision 3D reconstruction of dynamic target objects.

[0138] Optionally, in some embodiments, a deletion process is performed on target clusters in which the number of laser points is less than a first preset threshold; and / or, for any target cluster, a rigid body motion model is constructed based on the position information of the laser points in the target cluster; based on the rigid body motion model, the motion residuals of the laser points in the target cluster are determined; and laser points in the target cluster whose motion residuals are greater than a preset residual threshold are removed from the target cluster.

[0139] The first set quantity threshold can be less than the second set quantity threshold.

[0140] To construct the cylinder motion model, as an example, for any target cluster, based on the position information of the laser points in the target cluster under the laser point cloud in the current frame, and the inter-frame displacement information of the laser points in the target cluster, the predicted position information of the laser points in the next frame is determined. Based on the position information of the laser points in the target cluster under the laser point cloud in the current frame and the predicted position information of the next frame, the optimal rotation matrix R and translation vector T of the laser points in the target cluster are solved using the least squares method or ICP (Iterative Closest Point algorithm). This allows the construction of a rigid body motion model. P t+1 =RP t +T; (8) Among them, P t P represents the position information of laser points within the target cluster in the current frame's laser point cloud; t This provides the predicted position information for the corresponding laser point in the next frame.

[0141] Furthermore, based on the position information of the laser points in the target cluster in the next frame of the laser point cloud, and the Euclidean distance between the predicted position information of the laser points in the target cluster in the next frame, the corresponding motion residual is determined; and laser points in the target cluster whose motion residual is greater than a preset residual threshold are removed.

[0142] Therefore, by filtering out weak and invalid clusters through a point count threshold and calculating the motion residuals of laser points using a rigid body motion model, discrete noise points with abnormal residuals are accurately eliminated, thus achieving optimized purification of target clusters. This effectively filters out scattered noise, drift noise, and abnormal motion interference points in dynamic scenes, while retaining effective laser points that conform to the overall rigid body motion law of the target. It is suitable for the point cloud distribution characteristics and motion features of irregularly shaped dynamic target objects such as pump trucks, mixer trucks, forklifts, and multi-section trailers, avoiding target structure distortion and cluster shift problems caused by abnormal point interference, improving the purity and effectiveness of dynamic target clusters, and helping to improve the accuracy and stability of subsequent candidate laser point extraction, feature descriptor construction, and 3D reconstruction.

[0143] Before clustering multiple laser points in the current frame laser point cloud, optionally, in some embodiments, at least one of the following processes can be performed on the current frame laser point cloud: 1. Use a plane fitting method to remove ground points from the laser point cloud in the current frame; 2. Assuming the velocity vector corresponding to any laser point is f, calculate the motion amplitude in the laser point cloud of the current frame. Less than the set amplitude threshold τ v Laser point deletion; 3. Downsample the laser point cloud of the current frame.

[0144] It should be noted that laser points with motion amplitude less than the set amplitude threshold can be considered as static points, while laser points with motion amplitude not less than the set amplitude threshold can be considered as dynamic points.

[0145] Optionally, in some embodiments, for any target cluster, laser points in the target cluster whose motion residuals are greater than a preset residual threshold are identified as first noise points; the first noise points in the laser point cloud of the current frame are traversed; for any traversed first noise point, if the first noise point is determined to be a core point, a first cluster is constructed based on the first noise point and its spatiotemporal nearest neighbor set; the first cluster is processed at least once, wherein, in any processing step, the fifth laser point in the first cluster being processed is traversed, and if the spatiotemporal nearest neighbor set of the fifth laser point has a difference set with the first cluster being processed, the spatiotemporal nearest neighbor set of the fifth laser point is merged into the first cluster being processed, resulting in the first cluster being processed next; wherein the fifth laser point is a core point and is a first noise point that has not been traversed; if the fifth laser point does not exist in the first cluster being processed, the first cluster being processed is identified as the target cluster.

[0146] As an example, for any target cluster, laser points in the target cluster whose motion residuals are greater than a preset residual threshold are identified as rigid outliers (referred to as first noise points in this disclosure). These rigid outliers are re-clustered. If these rigid outliers can be clustered into new stable clusters, they are considered as non-rigid moving parts (such as pedestrian arms, vehicle doors, and other hinged structures), and the resulting stable clusters are used as the target clusters under the laser point cloud of the current frame.

[0147] Therefore, instead of directly eliminating the first noise point with excessive motion residuals, a secondary core point determination and iterative clustering and recombination are performed on this type of noise point. Discrete noise points with spatiotemporal correlation characteristics are re-aggregated into effective target clusters. This avoids the problem of easily deleting the edges of irregular targets and the effective point cloud of extended structures that is easily caused by a single elimination method. It can accurately recover suspected noise points caused by motion deviations in irregular structures such as pump truck booms, mixer tanks, forklift forks, and multi-section trailer bodies. It preserves the complete structural point cloud of dynamic irregular target objects to the maximum extent. It not only effectively filters out irregular discrete interference noise, but also fully restores the real point distribution of the target object, further improving the integrity and accuracy of target clustering in dynamic scenes. This provides more complete and pure point cloud data for subsequent laser point extraction, feature matching, and high-precision 3D reconstruction.

[0148] It is understood that there may be a second noise point in the current frame's laser point cloud, and the second noise point is a laser point that does not belong to any of the target clusters in at least one target cluster. Therefore, optionally, in some embodiments, for any target cluster, based on the position information of the laser points in the target cluster, the spatial centroid of the target cluster is determined; the velocity vectors corresponding to the laser points in the target cluster are averaged to determine the target motion parameters of the target cluster; for any second noise point in the laser point cloud, in response to the spatial distance between the second noise point and the spatial centroid of the target cluster being less than a fourth preset threshold, and the difference between the velocity vector corresponding to the second noise point and the target motion parameters of the target cluster being less than a third preset threshold, the target cluster is determined to be a second cluster associated with the second noise point; the cluster with the smallest spatial distance to the second noise point in the second cluster is determined as the third cluster; and the second noise point is added to the third cluster.

[0149] Among them, the spatial centroid refers to the average three-dimensional coordinates of all laser points in the corresponding target cluster, which can also be called the geometric center or geometric centroid.

[0150] The fourth set threshold can be greater than the second set threshold.

[0151] As an example, for the j-th target cluster, based on the position information of the laser points in the target cluster, the three-dimensional spatial coordinates of the spatial centroid of the target cluster are determined as follows: The average velocity vectors corresponding to the laser points in the target cluster are averaged to determine the average velocity vector of the target cluster (denoted as the target motion parameter in this disclosure). For the i-th second noise point in the laser point cloud, the position information of the i-th second noise point is p. i The corresponding velocity vector is f i The second threshold is set as ε s The third threshold is set as ε.f If the i-th second noise point satisfies the following formula, the j-th target cluster can be determined as the second cluster associated with the i-th second noise point; (9) (10) in, This indicates the fourth threshold value, k∈[1.5,2]; Furthermore, the cluster with the smallest spatial distance to the i-th second noise point in the second cluster associated with the i-th second noise point can be identified as the third cluster, and the i-th second noise point can be added to the third cluster.

[0152] Therefore, by accurately determining the cluster to which the second noise point belongs through the dual dimensions of the spatial centroid of the target cluster and the target motion parameters, and by selecting and matching clusters based on the dual thresholds of spatial distance and motion vector difference, the noise point is added to the optimally associated third cluster. This achieves the accurate recovery and merging of outlier effective point clouds, effectively solving the problem of effective point cloud dispersion and misjudgment as noise due to the large structural span and local motion differences of irregular dynamic targets such as pump truck booms, mixer tanks, forklift forks, and multi-section trailer bodies. It fully takes into account the consistency of target spatial position and motion state, greatly improves the overall point cloud structure of irregular target objects, and further enhances the integrity, continuity and accuracy of dynamic target clustering, providing a more realistic and complete point cloud data foundation for subsequent high-precision feature extraction, template matching and 3D reconstruction.

[0153] To extract candidate laser points from the current frame laser point cloud based on the target detection bounding box of the target object and at least one target cluster, in one possible implementation of this disclosure, a target laser point set is constructed based on a first laser point located within the target detection bounding box of the target object in the current frame laser point cloud; for any target cluster in the current frame laser point cloud, it is determined whether there is an intersection between the target cluster and the target laser point set; when there is an intersection between the target cluster and the target laser point set, candidate laser points are determined based on the laser points in the target cluster and the target laser point set, for example, all laser points in the target cluster and the target laser point set are determined as candidate laser points associated with the target object. Therefore, a precise target laser point set is first constructed based on the first laser point within the target detection box. Then, effective clusters are selected by determining the intersection relationship between the target cluster and the target laser point set, i.e., target clusters with intersection. The target cluster and the target laser point set are then fused to determine candidate laser points. This achieves bidirectional linkage verification between the semantic localization of the target detection box and the motion clustering results. It can accurately lock the effective motion point cloud belonging to the target object, effectively avoiding the problems of cross-target clustering mixing and background motion noise points being mistakenly included in dynamic scenes. It can completely collect the full-domain effective point cloud of irregular dynamic target objects such as pump truck booms, mixer truck tanks, forklift forks, and trailer multi-section bodies, improving the targeting and accuracy of candidate laser point extraction for dynamic target objects, and ensuring the effectiveness and integrity of the point cloud data of irregular target objects from the source.

[0154] Understandably, for target objects in a dynamic state, point cloud clustering is performed using the velocity vector inherent in the laser points, aggregating target clusters with consistent motion attributes. This is then combined with target detection boxes to achieve dual constraints of spatial and motion features, accurately extracting candidate laser points in dynamic scenes. This can precisely distinguish dynamic irregular target objects from static backgrounds and irrelevant moving interference objects, efficiently collecting effective point clouds of irregular dynamic target objects such as pump trucks, mixer trucks, forklifts, and trailers, avoiding the problems of mis-extraction and omission of point clouds that are prone to occur in dynamic scenes, and improving the accuracy and completeness of candidate laser point extraction for dynamic target objects.

[0155] In this embodiment, the laser point cloud is used as the current frame laser point cloud; the motion state of the target object is determined based on the position information of the target detection boxes corresponding to the target object in multiple consecutive laser point clouds; based on the motion state, point cloud semantic information and / or target detection boxes, candidate laser points associated with the target object are extracted from the current frame laser point cloud. Thus, by using the position information of the target detection boxes corresponding to multiple consecutive laser point clouds to accurately determine the motion state of the target object, and then adaptively combining point cloud semantic information and target detection boxes to extract candidate laser points in the current frame according to different motion states, a differentiated point cloud extraction logic separating static and dynamic states is realized. This adapts to various working conditions where the target object is stationary or dynamically moving, effectively solving the problem of missed or false detections of point clouds caused by the inability of a single extraction algorithm to adapt to the switching of target motion states. This comprehensively improves the extraction accuracy and completeness of candidate laser points for irregularly shaped target objects such as pump trucks, mixer trucks, forklifts, and trailers under different motion states.

[0156] Step 504: The candidate laser points associated with the target object in the multi-frame continuous laser point cloud are determined as the target laser points.

[0157] As one possible implementation, for any frame of a multi-frame continuous laser point cloud, the inter-frame motion parameters of the target object are determined based on the velocity vectors corresponding to the candidate laser points associated with the target object in the laser point cloud; wherein, the inter-frame motion parameters may include rotation matrices and translation vectors; based on the inter-frame motion parameters, motion compensation and temporal registration are performed on the candidate laser points associated with the target object in the multi-frame continuous laser point cloud to obtain aligned candidate laser points; the aligned candidate laser points are determined as the target laser points associated with the target object.

[0158] As an example, for the set of candidate laser points P in the current frame's laser point cloud that are associated with a target object (such as a vehicle), where the i-th candidate laser point P in the set... i (x i ,y i ,z i All of them carry the inter-frame displacement vector (denoted as inter-frame displacement information in this disclosure) Δp calculated by lidarflow. i Then: Construct the matching point set for the next frame: Based on the precise point-to-point mapping relationship, calculate the coordinates of the matching point in the laser point cloud of the next frame that matches the i-th candidate laser point mentioned above as Q. i : Q i = P i + Δp i (11) This yields a complete set of matching points Q. Then, using the least squares method, based on the coordinates of the laser points in the candidate laser point set and the coordinates of the matching points in the matching point set, the initial inter-frame motion parameters, i.e., the initial rotation matrix R and the initial translation vector T, are solved. Next, the GICP (Generalized Iterative Closest Point) algorithm is used to perform temporal registration iterative optimization on the candidate point clouds in the current and next frame laser point clouds, correcting the initial error of the least squares solution and obtaining the optimized inter-frame motion parameters. Since a single frame scan of the lidar has an exposure time, the acquisition timestamp t of each candidate laser point within the frame is... i Unlike other objects, the motion of dynamic targets causes point clouds to stretch and deform. Therefore, based on the optimized inter-frame motion parameters, point-by-point temporal alignment compensation is performed on all candidate points to uniformly align them to the reference timestamp t of the laser point cloud in the current frame. mid (This is the intermediate exposure time), and the specific execution process is as follows: 1. Calculate the inter-frame time span ΔT (= t) next - t mid The optimized rotation matrix R is then mapped to a Lie algebra rotation vector φ = ln(R). v ; where t next This serves as a reference timestamp for the next frame of the laser point cloud. 2. Traverse the candidate laser points associated with the target object in the current frame's laser point cloud, and determine the candidate laser points based on their acquisition timestamp t. i Reference timestamp t of the current frame laser point cloud mid The difference between them, the calculation time interpolation ratio s i (= (t) i -t mid () / ΔT); by interpolation, the candidate laser point at the acquisition timestamp t is obtained. i The corresponding relative rotation matrix R i (= exp(s) i ·φ) + ), relative translation vector T i (=s) i ·T); 3. Eliminate point cloud distortion caused by target object motion through inverse coordinate system transformation. Candidate laser points acquired at different times within the frame are uniformly corrected to the same reference time coordinate system according to the following formula to obtain aligned candidate points: P comp_i = R i T (P i - T i (12) 4. Summarize all compensated points to obtain a temporally aligned, motion-distortion-free candidate laser point set.

[0159] 5. The aligned candidate laser points are identified as the target laser points associated with the target object.

[0160] Therefore, by solving the inter-frame motion parameters, which include rotation and translation matrices, based on the velocity vectors of candidate laser points frame by frame, and relying on these parameters to perform motion compensation and temporal registration on candidate laser points in multiple frames, the precise alignment of multi-frame temporal point clouds is achieved and the final target laser point is generated. This effectively solves the problems of misalignment, temporal offset, and structural overlap distortion of multi-frame point clouds caused by attitude rotation and position translation during the dynamic movement of dynamic irregular target objects. It can accurately correct the temporal point cloud deviation of irregular structures such as pump truck booms, mixer truck tanks, forklift forks, and trailer multi-section bodies, fully integrate the effective detailed information of multi-frame point clouds, make up for the defects of sparse and structurally missing single-frame point clouds, and improve the temporal consistency and structural integrity of target laser points.

[0161] Step 505: Perform three-dimensional reconstruction based on the target laser points associated with the target object to obtain the initial three-dimensional model of the target object.

[0162] It should be noted that the execution process of step 505 can refer to the execution process of any embodiment of this disclosure, and will not be described in detail here.

[0163] In one possible implementation of this disclosure, when there are multiple target objects, for any one target object, the target object can be determined as a first object, and the other objects among the multiple target objects other than the first object can be determined as second objects; based on the position information of the target detection box corresponding to the first object in the multi-frame continuous laser point cloud and the position information of the target detection box corresponding to the second object in the multi-frame continuous laser point cloud, the target occlusion rate corresponding to the first object is determined; in response to the target occlusion rate being greater than a set occlusion rate threshold, it is determined that the initial 3D model reconstruction of the first object has failed.

[0164] For example, when the location information includes the 3D coordinates and size parameters of the corresponding target detection box, for any frame of a multi-frame continuous laser point cloud, the target detection boxes of the first object and the second object corresponding to that frame of the laser point cloud are projected onto a preset target plane (such as the XOY plane or the image plane, etc.), and the rectangular projection areas of the target detection boxes of the two objects on the preset target plane are obtained accordingly; the center coordinates, length, width and rotation angle of each rectangular projection area are extracted, and then the two rectangular projection areas are aligned to the same coordinate system according to the rotation angle (for example, by rotating the two rectangular projection areas to the axis alignment direction using a rotation matrix), and the overlapping rectangle of them in the axis alignment state is calculated. The width and height are used to obtain the projected overlap area of ​​the first and second objects; then, the first total projected area of ​​the rectangular projection region of the first object is determined; based on the ratio of the projected overlap area of ​​the first and second objects to the first total projected area, the single-frame occlusion rate of the first object in that frame of laser point cloud is determined; the single-frame occlusion rates of the first object in multiple frames of laser point cloud are weighted and averaged to obtain the average occlusion rate, which is then determined as the target occlusion rate; finally, when the target occlusion rate is greater than a set occlusion rate threshold, it indicates that most of the surface area of ​​the first object is continuously or densely occluded by other target objects (the second object) during the observation of multiple frames of continuous laser point cloud. This directly means that the currently acquired laser point cloud data cannot completely and continuously cover the shape structure of the first object, and there is a significant lack of information. Therefore, it is difficult to recover the true and complete geometry based on these occluded point clouds (e.g., holes, collapses, or false structures will appear), and the initial 3D model reconstruction of the first object is determined to be a failure.

[0165] For example, when there are multiple second objects, and the position information includes the 3D coordinates and size parameters of the corresponding target detection boxes, for any frame of a multi-frame continuous laser point cloud, the target detection boxes of the first and second objects corresponding to that frame of the laser point cloud are projected onto a preset target plane (such as the XOY plane or the image plane, etc.), and the rectangular projection areas of the target detection boxes of the two objects on the preset target plane are obtained accordingly. The center coordinates, length, width, and rotation angle of each rectangular projection area are extracted, and then the two rectangular projection areas are aligned to the same coordinate system according to the rotation angle (for example, by rotating the two rectangular projection areas to the axis alignment direction using a rotation matrix), and their coordinates on the axis are calculated. The width and height of the overlapping rectangles in the aligned state are used to obtain the projected overlap area between the first and second objects; then, the first total projected area of ​​the rectangular projection region of the first object is determined; the projected overlap areas of multiple second objects with the first object are summed to obtain the total projected overlap area; based on the ratio of the total projected overlap area to the first total projected area, the single-frame occlusion rate of the first object corresponding to the laser point cloud in that frame is determined; the single-frame occlusion rates of the first object corresponding to multiple frames of laser point clouds are weighted and averaged to obtain the average occlusion rate, and this average occlusion rate is determined as the target occlusion rate; finally, when the target occlusion rate is greater than the set occlusion rate threshold, the initial 3D model reconstruction of the first object is determined to have failed.

[0166] Therefore, for complex scenarios with multiple coexisting targets, the target occlusion rate is accurately calculated by using the position information of each target detection box, and the failure of the initial 3D model reconstruction is determined based on the occlusion rate threshold. It can accurately identify the mutual occlusion status of irregularly shaped targets such as pump trucks, mixer trucks, forklifts, and trailers in scenarios with multiple vehicles crossing and passing at close range. It effectively avoids problems such as model incompleteness and structural distortion caused by the reconstruction algorithm's inability to actively perceive occlusion and blind reconstruction. It can accurately trigger the subsequent template fusion and completion logic, ensuring the effectiveness and integrity of the 3D reconstruction of target objects under occlusion conditions, and improving the environmental adaptability and robustness of the 3D reconstruction algorithm in complex multi-target interaction scenarios.

[0167] In another possible implementation of the embodiments of this disclosure, a third compact frame of the target object is constructed based on the initial three-dimensional model; when the third compact frame satisfies the target conditions, it is determined that the reconstruction of the initial three-dimensional model of the target object has failed.

[0168] The target conditions may include at least one or more of the following combinations: 1. The difference between the length of the third compact bounding box and the target length of the target detection box is greater than the set length threshold; 2. The difference between the width of the third compact bounding box and the target width of the target detection box is greater than the set width threshold; 3. The difference between the height of the third compact box and the target height of the target detection box is greater than the set height threshold.

[0169] The target length of the target detection box can, for example, be the average or median length of the target detection box of the target object in multiple consecutive frames of laser point cloud, and this disclosure does not limit it.

[0170] The target width of the target detection box can, for example, be the average or median width of the target detection box of the target object under multiple consecutive frames of laser point cloud, and this disclosure does not limit it.

[0171] The target height of the target detection box can, for example, be the average or median height of the target detection box of the target object under multiple consecutive frames of laser point cloud, and this disclosure does not limit it.

[0172] The third compact box can be the smallest bounding box for generating the target object based on the initial 3D model, and the third compact box can have corresponding length, width, and height. It should be noted that the third compact box can be the same as or different from the first compact box in this disclosure, and this disclosure does not impose any restrictions on this.

[0173] It should be noted that when the difference between the length of the third compact bounding box and the target length of the target detection box is greater than a set length threshold, it indicates that the length of the third compact bounding box of the initial 3D model is much smaller than the target length of the target detection box; when the difference between the width of the third compact bounding box and the target width of the target detection box is greater than a set width threshold, it indicates that the width of the third compact bounding box of the initial 3D model is much smaller than the target width of the target detection box; when the difference between the height of the third compact bounding box and the target height of the target detection box is greater than a set height threshold, it indicates that the height of the third compact bounding box of the initial 3D model is much smaller than the target height of the target detection box. These situations indicate that the size of the third compact bounding box is much smaller than the size of the target detection box, suggesting that the reconstructed initial 3D model suffers from severe "under-envelope" or "shrinkage," resulting in a serious lack of model integrity. Therefore, it can be determined that the initial 3D model reconstruction of the target object has failed.

[0174] Therefore, by constructing a third compact bounding box corresponding to the initial 3D model, the difference between the length, width, and height dimensions and the corresponding dimensions of the target detection box is compared. Based on the preset thresholds of each dimension, it is determined whether the initial 3D model reconstruction has failed. This achieves quantitative verification of reconstruction quality from the perspective of the overall contour size of the model. It can accurately identify reconstruction anomalies such as model size distortion, contour loss, and scale deviation caused by point cloud missing, limited viewpoint, and local occlusion of irregularly shaped target objects such as pump trucks, mixer trucks, forklifts, and trailers. It makes up for the limitations of judging reconstruction failure solely by occlusion rate, enriches the judgment dimensions of reconstruction failure, improves the comprehensiveness and accuracy of reconstruction anomaly identification, and can accurately trigger subsequent template fusion and completion strategies. It effectively improves the contour accuracy and structural integrity of 3D reconstruction of irregularly shaped target objects, and further enhances the reliability and adaptability of 3D reconstruction algorithms under complex road conditions.

[0175] In another possible implementation of this disclosure, the sixth laser point belonging to the target part among the target laser points associated with the target object is projected onto the target coordinate plane to obtain the coordinates of the sixth laser point in the target coordinate plane; based on the coordinates of the sixth laser point in the target coordinate plane, the projected area of ​​the sixth laser point in the target coordinate plane is determined; for any frame of laser point cloud, the area of ​​the target detection box of the target object under the laser point cloud in the target coordinate plane is determined; based on the ratio between the projected area and the first area of ​​the target object's target detection box in the target coordinate plane under each frame of laser point cloud, a target ratio is determined; when the target ratio is less than a fifth preset threshold, it is determined that the initial three-dimensional model reconstruction of the target object has failed.

[0176] The target part can be pre-defined, such as the front of the vehicle.

[0177] The target coordinate plane can be a pre-defined plane, such as the YOZ plane.

[0178] To obtain the target ratio, for example, the target ratio can be obtained by weighted summing the ratio between the projected area and the first area of ​​the target detection box in the target coordinate plane under each frame of laser point cloud.

[0179] It should be noted that when the target ratio is less than the fifth set threshold, it indicates that the area covered by the effective laser point cloud belonging to the target location on the target coordinate plane is significantly smaller than the expected area that the target location should occupy. This directly means that even if a target detection box exists, the point cloud data actually falling on the key location is extremely sparse or only concentrated in a local area, far from sufficient to support a complete and continuous 3D surface reconstruction of the target location. Therefore, the initial 3D model reconstructed based on these point clouds will have large areas of missing parts, holes, or severe distortions in the target location, failing to meet the requirements of geometric accuracy and completeness, thus indicating that the initial 3D model reconstruction of the target object has failed.

[0180] Therefore, by calculating the projected area of ​​the sixth laser point of the target part through planar projection, and combining the planar area of ​​the target detection box in each frame to obtain the target ratio, the initial 3D model reconstruction failure is accurately determined based on the ratio threshold. The reconstruction anomaly judgment criterion is added from the dimension of the effective part point cloud ratio of the target object. Together with the occlusion rate and compact box size verification mentioned above, a multi-dimensional complementary judgment system is formed. It can effectively identify the reconstruction failure problem caused by the sparse point cloud of key parts of irregular target objects and the low proportion of effective structural points. It avoids the detection blind spots of single size and occlusion judgment, improves the comprehensiveness and accuracy of reconstruction failure scene recognition, and ensures that various structural defects and information deficiencies can be effectively detected. It accurately activates the subsequent model template fusion and completion mechanism, and ensures the integrity and geometric accuracy of the 3D reconstruction of the target object in complex driving scenarios from multiple dimensions, further improving the fault tolerance and robustness of 3D reconstruction.

[0181] Step 506: In response to the failure of the initial 3D model reconstruction, a target object template matching the target object is obtained from multiple preset object templates based on the target feature descriptor of the target object.

[0182] Step 507: Merge the initial 3D model with the target object template to obtain the target 3D model of the target object.

[0183] It should be noted that the execution process of steps 506 to 507 can refer to the execution process of any embodiment of this disclosure, and will not be described in detail here.

[0184] In one possible implementation of this disclosure, when the initial 3D model is successfully reconstructed, a new preset object template can be generated based on the initial 3D model; the new preset object template is then saved to the point cloud template library.

[0185] For example, when the initial 3D model is successfully reconstructed, the initial 3D model can be registered with the corresponding category of preset object model in the point cloud template library to align the two to a unified coordinate system; then, the aligned initial 3D model is used as a candidate template; then, the similarity between the candidate template and the existing preset object template in the point cloud template library is compared. If the similarity is lower than the set entry threshold, the candidate template is entered into the library as a new preset object template.

[0186] As an example, the following process can also be performed on preset object templates in the point cloud template library: 1. Complete, standardize, and deduplicate existing vehicle point clouds: For existing vehicle templates in the point cloud template library, firstly, symmetry completion (using the approximate symmetry of the vehicle to fill missing parts), hole filling algorithms, or generative model-based completion methods are used to geometrically complete the incomplete areas in the templates caused by missing original data. Then, normalization operations are performed on all templates, including translating the templates to the origin of the coordinate system, rotating their principal directions to align with the coordinate axes through principal component analysis, and scaling the templates to a standard size range. Finally, by calculating the pairwise distance between templates, templates with a distance less than the deduplication threshold are merged into a single template (retaining the one with the highest reconstruction quality or performing a weighted average), thus achieving template deduplication.

[0187] 2. Establish a hierarchical template index based on vehicle type, size range, and structural features: First, a three-layer index structure is constructed: the first layer can coarsely classify templates by vehicle type (e.g., sedan, SUV, truck, bus, etc.); the second layer can create subcategories within each category based on size ranges (e.g., dividing vehicle length, width, and height into several intervals); the third layer can cluster within each size subcategory based on structural features (e.g., hatchback / sedan, presence of roof rack, A-pillar angle, and other geometric feature descriptors). During querying, for example, the category can be matched first, then the size range can be filtered, and finally, a precise comparison can be performed on a small number of structurally similar templates.

[0188] 3. Generate retrieval information such as descriptors, dimensions, and bounding boxes for each template: For each vehicle template in the point cloud template library, extract and store the following retrieval metadata: ① Template feature descriptors, used for similarity matching; ② Three-dimensional dimensions: calculate the actual length, width, and height of the template before or after normalization; ③ Directed bounding box: Records the center point coordinates, three axis direction vectors, and half length of each axis of the template's smallest directed bounding box, used for fast spatial exclusion filtering; ④ Quality metadata, including template source (reconstruction batch number), version number, update time, and confidence score, to support continuous evaluation and elimination of templates.

[0189] It should be noted that the above example of processing the point cloud template library is merely exemplary, and other methods may be used in practical applications.

[0190] Therefore, after the initial 3D model is successfully reconstructed, a new preset object template is adaptively generated based on the accurate and complete initial 3D model, and archived and stored in the point cloud template library. This realizes the dynamic iteration and autonomous expansion of the templates in the point cloud template library, and can continuously accumulate template data of irregular objects that are adapted to real scene conditions. This effectively makes up for the shortcomings of related technologies that use fixed template libraries with incomplete object coverage and poor scene adaptability. It can continuously enrich the template resources of various irregular target objects such as pump trucks, mixer trucks, forklifts, and trailers, so that subsequent 3D reconstruction and template matching can adapt to more object shapes and on-site postures, continuously improve the accuracy and universality of template matching, and enable 3D reconstruction to have scene self-learning and self-optimization capabilities. This further enhances the adaptive capability and long-term stability of 3D reconstruction of irregular target objects in complex dynamic traffic scenes.

[0191] The 3D reconstruction method of this disclosure generates point cloud semantic information and target detection boxes corresponding to any given frame of laser point cloud, based on the laser point cloud and the corresponding image. The target detection boxes are used to define the target object and its corresponding laser point. Based on the point cloud semantic information and / or the target detection boxes, candidate laser points associated with the target object are extracted from the laser point cloud. The candidate laser points associated with the target object in multiple consecutive frames of laser point cloud are determined as the target laser point. Thus, by fusing laser point cloud and image data to generate point cloud semantic information and target detection boxes, and using this dual information to filter candidate laser points, and then integrating the candidate laser points corresponding to multiple frames of laser point cloud to obtain the final target laser point, this method can accurately distinguish various target laser points within a scene, eliminate background noise and invalid point clouds, accurately locate irregularly shaped target objects such as pump trucks, mixer trucks, forklifts, and trailers, and ensure that laser points corresponding to key structures such as booms, tanks, forklifts, and multi-section vehicle bodies are not lost. This improves data quality from the source, providing a basis for subsequent feature representation and 3D reconstruction of the target object.

[0192] To clearly illustrate the three-dimensional reconstruction method disclosed herein, a detailed explanation is provided below with examples.

[0193] As an example, a 3D reconstruction method is applied to a high-precision vehicle reconstruction framework in autonomous driving scenarios. This framework employs a three-pronged approach—static reconstruction, dynamic reconstruction, and template matching optimization—to achieve high-quality geometric restoration of foreground vehicles, especially irregularly shaped vehicles, and outputs vehicle occupancy results suitable for OCC tasks. The core processing modules of this framework include: I. Static Cumulative Reconstruction Module; The static cumulative reconstruction module can be used to perform the following steps: 1.1 Based on the BEV detection box (referred to as the target detection box in this disclosure) and the point cloud semantic results (referred to as the point cloud semantic information in this disclosure), candidate point clouds (referred to as candidate laser points in this disclosure) of the target vehicle (referred to as the target object in this disclosure) are extracted; 1.2 Associate candidate point clouds of the same target vehicle in adjacent time series; 1.3 Fusion of multi-frame candidate point clouds under semantic consistency and spatial constraints; 1.4 Generate a dense static vehicle point cloud representation (referred to as the initial 3D model in this disclosure); the static vehicle point cloud representation can be used for subsequent point cloud template library supplementation, template matching, or direct OCC modeling.

[0194] II. Dynamic Temporal Reconstruction Module; The dynamic time-series reconstruction module is used to perform the following steps: 2.1 Obtain 3D lidar flow information (referred to as inter-frame displacement information in this disclosure) of point cloud for each frame based on Flow4D model inference (the spatial displacement information of each lidar point (referred to as laser point in this disclosure) from the current frame to the next frame). 2.2 Spatiotemporal clustering is performed on lidarFlow to group dynamic point clouds with strong motion consistency and close spatial distance into one class (referred to as the target cluster in this disclosure). 2.3 Using lidarFlow spatiotemporal clustering results and BEV detection boxes, a sparse point cloud of a single frame of a complete target vehicle is obtained (referred to as candidate laser points in this disclosure); 2.4 Using lidarFlow to estimate the motion relationship data of the same vehicle between different time frames (referred to as inter-frame motion parameters in this disclosure). 2.5 Perform temporal registration and motion compensation on the sparse point cloud of dynamic target vehicles; 2.6 The aligned multi-frame sparse point clouds are fused to generate a high-density point cloud of the dynamic target vehicle (referred to as the target laser point in this disclosure). 2.7 Based on high-density point clouds, dynamic reconstruction results are obtained (referred to as the initial 3D model in this disclosure); the dynamic reconstruction results can be used for subsequent point cloud template library supplementation, template matching or direct OCC modeling.

[0195] III. Dynamic and Static Reconstruction Quality Assessment Module; The dynamic and static reconstruction quality assessment module can be used to perform the following steps: Reconstruction is considered a failure and template matching may be triggered if any of the following conditions occur: (Static vehicle point cloud representation and / or dynamic reconstruction results) 1. High occlusion rate: Calculate the mutual occlusion rate between obstacles in the scene, calculate the average occlusion rate of the target vehicle throughout the entire track, and exclude obstacles below the threshold from reconstruction; specifically, iterate through the BEV detection boxes of all target objects in the scene, and calculate the occlusion rate of each BEV detection box based on the vehicle lidar system; calculate the average occlusion rate of the target object throughout the entire trajectory lifecycle (multiple frames); if the average occlusion rate is greater than the threshold (referred to as the set occlusion rate threshold in this disclosure) (e.g., 0.6~0.8, adjustable), it is considered that the observable point cloud is insufficient to support reliable reconstruction, and template matching is triggered for the target object.

[0196] 2. The length of the reconstructed point cloud's compact bounding box (referred to as the third compact bounding box in this disclosure) is much smaller than that of the BEV detection box; 3. The width of the compact bounding box in the reconstructed point cloud is much smaller than that of the BEV detection box; 4. The height of the compact bounding box in the reconstructed point cloud is much smaller than that of the BEV detection box; 5. The integrity of the front of the vehicle is too low: The point cloud of the front of the vehicle (referred to as the target part in this disclosure) is cropped out and then projected onto the YOZ plane. The ratio of the projected area of ​​the front point cloud to the area of ​​the corresponding surface of the detection box (referred to as the target ratio in this disclosure) is calculated. If the area ratio is less than the area threshold (referred to as the fifth set threshold in this disclosure), the reconstruction fails.

[0197] IV. Efficient Template Library Expansion Module; The template library high-efficiency expansion module is used to perform the following steps: 1. Develop new templates based on historical high-quality reconstruction results; 2. Complete, standardize, and deduplicate existing vehicle point clouds using templates; 3. Establish a hierarchical template index based on vehicle type, size range, and structural features; 4. Generate retrieval information such as descriptors, dimensions, and bounding boxes for each template.

[0198] It should be noted that by expanding the template library of point cloud templates, the ability to express long-tailed, irregularly shaped vehicles such as forklifts, pump trucks, cement mixer trucks, and trailers can be significantly enhanced.

[0199] V. Template matching module based on point cloud descriptors; This invention introduces a template matching method based on point cloud descriptors to jointly characterize the local and global geometry of candidate vehicles.

[0200] The matching process includes: 1. Extract point cloud descriptors (referred to as sub-feature descriptors in this disclosure) from the single-frame point cloud of the target vehicle to describe its shape; wherein, the point cloud descriptors are constructed as follows: Using the center of the vehicle's bottom as the origin, N=265 uniformly distributed sampling rays with L=20 layers are constructed within a unit hemispherical space. Voxels are queried step-by-step along each ray direction with a fixed step size, recording the density and outer contour distance (denoted as sphere center distance in this disclosure) of each non-empty voxel. The outer contour distance of the non-empty voxel with the highest density is selected as the descriptor for that ray direction; if no non-empty voxel exists, the descriptor for the corresponding ray direction is set to 0. The resulting N-dimensional feature vector encodes the vehicle's three-dimensional outer contour into a compact spatial topological descriptor, i.e., a point cloud descriptor.

[0201] 2. Recall candidate templates with similar size or category from the point cloud template library (referred to as candidate object templates in this disclosure); 3. The point cloud descriptors of multiple frames are fused in temporal order, and the similarity between the fused point cloud descriptor (referred to as the target feature descriptor in this disclosure) and the template descriptor of the candidate template (referred to as the template feature descriptor in this disclosure) is calculated and sorted. 4. Select the optimal template (e.g., the one with the highest similarity) and complete the alignment and completion; 5. The matched template is fused with the current observation point cloud to generate the final vehicle structure representation (referred to as the target 3D model in this disclosure).

[0202] VI. Vehicle OCC Generation Module.

[0203] It should be noted that after obtaining the static reconstruction result, dynamic reconstruction result, and optimal template matching result, the vehicle OCC generation module can fuse the three to output a vehicle occupancy representation for OCC; the output result can cover: Rigid body vehicle target; Non-rigid or nearly irregular vehicle targets with complex structures; The outer contours and main structural areas of various irregularly shaped vehicles.

[0204] It should be noted that during the fusion process, the weights of each module can be adaptively adjusted based on the target type, point cloud quality, motion state, and template matching confidence to obtain a more robust final reconstruction result.

[0205] In summary, the high-precision vehicle reconstruction framework in autonomous driving scenarios has at least the following advantages: 1. Significantly improves the accuracy of reconstructing irregularly shaped vehicles: For irregularly shaped target objects such as forklifts, pump trucks, cement mixers, and trailers, it can restore OCC expressions that are more consistent with the real structure and reduce shape distortion.

[0206] 2. Enhanced adaptability to complex dynamic scenes: By introducing dynamic temporal reconstruction and LiDAR Flow, the structural recovery of moving targets is more stable.

[0207] 3. Improve the overall quality of the OCC production line: By working together with static reconstruction, dynamic reconstruction and template optimization, improve the quality of foreground target occupancy modeling and improve the training effect of downstream models.

[0208] 4. Improve performance in long-tail scenarios: The efficient expansion mechanism of the template library can continuously accumulate data on new models and new scenarios, forming a data closed loop and improving mass production adaptation capabilities.

[0209] 5. It has good engineering feasibility: it can directly serve the production of autonomous driving data, model training and performance optimization.

[0210] To achieve the above embodiments, this disclosure also provides a three-dimensional reconstruction apparatus. For example... Figure 6 As shown, Figure 6 This is a schematic diagram according to the fifth embodiment of the present disclosure. The three-dimensional reconstruction apparatus 600 may include: a first acquisition module 610, a first determination module 620, a reconstruction module 630, a second acquisition module 640, and a first fusion module 650.

[0211] The first acquisition module 610 is used to acquire time-series data during the vehicle's driving process; wherein the time-series data includes spatiotemporally aligned multi-frame continuous laser point clouds and multi-frame continuous images, and any frame of laser point cloud includes multiple laser points.

[0212] The first determining module 620 is used to determine the target laser point associated with the target object from multiple frames of continuous laser point cloud based on time-series data; wherein the target object is located in the surrounding scene of the vehicle.

[0213] The reconstruction module 630 is used to perform three-dimensional reconstruction based on the target laser points associated with the target object to obtain the initial three-dimensional model of the target object.

[0214] The second acquisition module 640 is used to, in response to the failure of the initial 3D model reconstruction, acquire a target object template that matches the target object from multiple preset object templates based on the target feature descriptor of the target object; wherein, the target feature descriptor is determined based on the distance between the target voxel and the bottom center point of the target object among multiple voxels obtained by target laser point voxelization, and the target voxel is determined along a preset sampling direction.

[0215] The first fusion module 650 is used to fuse the initial 3D model with the target object template to obtain the target 3D model of the target object.

[0216] In one possible implementation of this disclosure, the three-dimensional reconstruction apparatus 600 may further include: The first processing module is used to voxelize the target laser points associated with the target object to obtain multiple voxels.

[0217] The second determining module is used to determine the bottom center point of the target object based on the position information of the target laser point.

[0218] The first construction module is used to construct multiple sampling rays in a hemispherical space with the bottom center point as the origin and a radius of a set distance to determine the preset sampling direction.

[0219] The acquisition module is used to acquire the density of non-empty voxels and the corresponding center-to-center distance along the sampling ray with a set step size, starting from the origin, for any sampling ray; where density is used to indicate the number of target laser points in the corresponding voxel, and center-to-center distance is used to indicate the distance between the corresponding voxel and the origin.

[0220] The second processing module is used to take the voxel with the highest density among the non-empty voxels collected along the sampling ray as the target voxel, and take the distance from the center of the target voxel as the feature distance corresponding to the sampling ray. The second fusion module is used to fuse the feature distances corresponding to multiple sampling rays to obtain the target feature descriptor.

[0221] In one possible implementation of this disclosure, the three-dimensional reconstruction apparatus 600 may further include: The third processing module is used to voxelize the target laser points associated with the target object in any laser point cloud, so as to obtain multiple voxels of the target object in the laser point cloud.

[0222] The third determination module is used to determine the bottom center point of the target object under the laser point cloud based on the position information of the target laser point associated with the target object in the laser point cloud.

[0223] The fourth determination module is used to determine the sub-feature descriptor of the target object under the laser point cloud based on multiple voxels of the target object under the laser point cloud; wherein, the sub-feature descriptor is determined based on the distance between the target voxel determined along the preset sampling direction and the bottom center point of the target object under the laser point cloud.

[0224] The third fusion module is used to fuse the sub-feature descriptors corresponding to the target object under multiple frames of continuous laser point clouds to obtain the target feature descriptor.

[0225] In one possible implementation of this disclosure, any sub-feature descriptor includes values ​​for multiple dimensions; the third fusion module is configured to: determine multiple keyframe laser point clouds from multiple consecutive laser point clouds; for any dimension, determine the maximum value of the sub-feature descriptor corresponding to the target object in the multiple keyframe laser point clouds as the target value corresponding to the dimension; and generate a target feature descriptor based on the target values ​​corresponding to multiple dimensions.

[0226] In one possible implementation of this disclosure, any preset object template has a corresponding template feature descriptor; the second acquisition module 640 is configured to: for any preset object template, determine the descriptor similarity between the preset object template and the target object based on the template feature descriptor corresponding to the preset object template and the target feature descriptor; and determine the target object template from multiple preset object templates based on the descriptor similarity corresponding to each preset object template.

[0227] In one possible implementation of this disclosure, both the target feature descriptor and the template feature descriptor include values ​​for multiple dimensions; the second acquisition module 640 is configured to: for any dimension, determine the weight corresponding to the dimension based on the values ​​of the template feature descriptor and the target feature descriptor corresponding to the preset object template in the dimension; determine the first coefficient corresponding to the dimension based on the weight corresponding to the dimension and the difference between the values ​​of the template feature descriptor and the target feature descriptor in the dimension; fuse the first coefficients corresponding to multiple dimensions to obtain a second coefficient; and determine the descriptor similarity based on the second coefficient.

[0228] In one possible implementation of this disclosure, the three-dimensional reconstruction apparatus 600 may further include: The second building module is used to construct the first compact bounding box corresponding to the target object based on the initial 3D model, and to determine the first volume size of the first compact bounding box.

[0229] The fifth determining module is used to determine the second volume size of the second compact frame corresponding to any preset object template.

[0230] The fourth processing module is used to select a preset object template as a candidate object template in response to the difference between the first volume size and the second volume size being less than a first set threshold.

[0231] The sixth determining module is used to determine the target object template from at least one candidate object template based on the descriptor similarity corresponding to at least one candidate object template.

[0232] In one possible implementation of this disclosure, the first determining module 620 is configured to: for any frame of laser point cloud, generate point cloud semantic information and a target detection box corresponding to the laser point cloud based on the laser point cloud and the corresponding image; wherein the target detection box is used to define the target object and the corresponding laser point; extract candidate laser points associated with the target object from the laser point cloud based on the point cloud semantic information and / or the target detection box; and determine the candidate laser points associated with the target object in multiple consecutive frames of laser point cloud as the target laser points.

[0233] In one possible implementation of this disclosure, the target detection box has corresponding position information; the first determining module 620 is configured to: take the laser point cloud as the current frame laser point cloud; determine the motion state of the target object based on the position information of the target detection boxes corresponding to the target object in multiple consecutive frames of laser point clouds; and extract candidate laser points associated with the target object from the current frame laser point cloud based on the motion state, point cloud semantic information and / or target detection boxes.

[0234] In one possible implementation of this disclosure, the first determining module 620 is configured to: in response to a static motion state, extract candidate laser points from the current frame laser point cloud based on the point cloud semantic information and target detection box corresponding to the current frame laser point cloud.

[0235] In one possible implementation of this embodiment, the first determining module 620 is configured to: determine the laser points other than the first laser point among a plurality of laser points in the current frame laser point cloud as second laser points; wherein the first laser point is located within the target detection box of the target object in the current frame laser point cloud; and determine candidate laser points based on the laser points in the second laser points whose point cloud semantic information is consistent with the point cloud semantic information corresponding to the first laser point, and the first laser point.

[0236] In one possible implementation of this disclosure, any laser point has a corresponding velocity vector; the first determining module 620 is configured to: in response to the motion state being dynamic, cluster multiple laser points in the current frame laser point cloud based on the velocity vectors corresponding to multiple laser points in the current frame laser point cloud to obtain at least one target cluster; and extract candidate laser points from the current frame laser point cloud based on the target detection box of the target object in the current frame laser point cloud and at least one target cluster.

[0237] In one possible implementation of this disclosure, the first determining module 620 is configured to: traverse multiple laser points in the laser point cloud of the current frame; for any traversed third laser point, if the third laser point is determined to be a core point, construct an initial cluster based on the third laser point and its spatiotemporal nearest neighbor set; wherein, the core point is determined based on the number of laser points in the spatiotemporal nearest neighbor set of the corresponding laser point, the spatial distance between the laser points in the spatiotemporal nearest neighbor set and the third laser point is less than a second preset threshold, and the motion vector difference between them is less than a third preset threshold, the motion vector difference being determined based on the velocity vectors corresponding to the two laser points. The process involves processing the initial cluster at least once. In any given processing step, the fourth laser point in the initial cluster is traversed. If the set of the fourth laser point's spatiotemporal nearest neighbors has a difference with the initial cluster, the set of the fourth laser point's spatiotemporal nearest neighbors is merged into the initial cluster to obtain the initial cluster for the next processing step. The fourth laser point is a core point and is a laser point that has not been traversed. If the fourth laser point does not exist in the initial cluster or the set of the fourth laser point's spatiotemporal nearest neighbors has no difference with the initial cluster, the initial cluster is determined as the target cluster.

[0238] In one possible implementation of this disclosure, the three-dimensional reconstruction apparatus 600 may further include: The fifth processing module is used to perform deletion processing on target clusters in at least one target cluster where the number of laser points is less than a first set threshold; and / or, for any target cluster, construct a rigid body motion model based on the position information of the laser points in the target cluster; determine the motion residual of the laser points in the target cluster based on the rigid body motion model; and remove laser points in the target cluster whose motion residual is greater than a preset residual threshold from the target cluster.

[0239] In one possible implementation of this embodiment, the 3D reconstruction device 600 may further include: a sixth processing module, configured to: for any target cluster, identify laser points in the target cluster whose motion residuals are greater than a preset residual threshold as first noise points; traverse the first noise points in the laser point cloud of the current frame; for any traversed first noise point, if the first noise point is determined to be a core point, construct a first cluster based on the first noise point and its spatiotemporal nearest neighbor set; perform at least one processing step on the first cluster, wherein, in any processing step, traverse the fifth laser point in the first cluster being processed, and if the spatiotemporal nearest neighbor set of the fifth laser point has a difference set with the first cluster being processed, merge the spatiotemporal nearest neighbor set of the fifth laser point into the first cluster being processed to obtain the first cluster being processed next; wherein the fifth laser point is a core point and is a first noise point that has not been traversed; if the fifth laser point does not exist in the first cluster being processed, determine the first cluster being processed as the target cluster.

[0240] In one possible implementation of this embodiment, the current frame laser point cloud includes a second noise point, which is a laser point that does not belong to any target cluster in at least one target cluster. The three-dimensional reconstruction device 600 may further include a seventh processing module, configured to: determine the spatial centroid of any target cluster based on the position information of the laser points in the target cluster; perform an average calculation on the velocity vectors corresponding to the laser points in the target cluster to determine the target motion parameters of the target cluster; for any second noise point in the laser point cloud, in response to the spatial distance between the second noise point and the spatial centroid of the target cluster being less than a fourth preset threshold, and the difference between the velocity vector corresponding to the second noise point and the target motion parameters of the target cluster being less than a third preset threshold, determine the target cluster as a second cluster associated with the second noise point; wherein the fourth preset threshold is greater than the second preset threshold; determine the cluster with the smallest spatial distance to the second noise point in the second cluster as a third cluster; and add the second noise point to the third cluster.

[0241] In one possible implementation of this disclosure, the first determining module 620 is configured to: construct a target laser point set based on a first laser point located within a target detection box of the target object in the current frame laser point cloud; determine whether there is an intersection between the target cluster and the target laser point set for any target cluster; and, in response to the existence of an intersection between the target cluster and the target laser point set, determine candidate laser points based on the laser points in the target cluster and the target laser point set.

[0242] In one possible implementation of this disclosure, the first determining module 620 is configured to: for any frame of a multi-frame continuous laser point cloud, determine the inter-frame motion parameters of the target object based on the velocity vectors corresponding to candidate laser points associated with the target object in the laser point cloud; wherein the inter-frame motion parameters include a rotation matrix and a translation vector; based on the inter-frame motion parameters, perform motion compensation and temporal registration on the candidate laser points associated with the target object in the multi-frame continuous laser point cloud to obtain aligned candidate laser points; and determine the aligned candidate laser points as the target laser points associated with the target object.

[0243] In one possible implementation of this disclosure, there are multiple target objects; the 3D reconstruction device 600 may further include: The seventh determination module is used to determine, for any given target object, the target object as the first object, and the other objects among multiple target objects besides the first object as the second object.

[0244] The eighth determining module is used to determine the target occlusion rate corresponding to the first object based on the position information of the target detection box corresponding to the first object in the multi-frame continuous laser point cloud and the position information of the target detection box corresponding to the second object in the multi-frame continuous laser point cloud.

[0245] The ninth determination module is used to determine that the initial 3D model reconstruction of the first object has failed in response to the target occlusion rate being greater than a set occlusion rate threshold.

[0246] In one possible implementation of this disclosure, the three-dimensional reconstruction apparatus 600 may further include: The third building module is used to construct a third compact frame of the target object based on the initial 3D model.

[0247] The tenth determination module is used to determine that the initial 3D model reconstruction of the target object has failed in response to the third compact box satisfying the target conditions; wherein the target conditions include at least one or more of the following combinations: the difference between the length of the third compact box and the target length of the target detection box is greater than a set length threshold; the difference between the width of the third compact box and the target width of the target detection box is greater than a set width threshold; the difference between the height of the third compact box and the target height of the target detection box is greater than a set height threshold.

[0248] In one possible implementation of this disclosure, the three-dimensional reconstruction apparatus 600 may further include: The projection module is used to project the sixth laser point, which belongs to the target part, from the target laser points associated with the target object onto the target coordinate plane, so as to obtain the coordinates of the sixth laser point on the target coordinate plane.

[0249] The eleventh determining module is used to determine the projected area of ​​the sixth laser point on the target coordinate plane based on the coordinates of the sixth laser point on the target coordinate plane.

[0250] The twelfth determination module is used to determine the area of ​​the target object's bounding box in the target coordinate plane under the laser point cloud for any given frame of laser point cloud.

[0251] The thirteenth determination module is used to determine the target ratio based on the ratio between the projected area and the first area of ​​the target detection box of the target object in each frame of laser point cloud on the target coordinate plane.

[0252] The fourteenth determination module is used to determine that the initial 3D model reconstruction of the target object has failed in response to the target ratio being less than the fifth set threshold.

[0253] In one possible implementation of this disclosure, multiple preset object templates are stored in a point cloud template library; the 3D reconstruction device 600 may further include: The generation module is used to generate a new preset object template based on the initial 3D model in response to the successful reconstruction of the initial 3D model.

[0254] The save module is used to save new preset object templates to the point cloud template library.

[0255] The 3D reconstruction apparatus of this disclosure acquires temporal data during vehicle movement; wherein the temporal data includes spatiotemporally aligned multi-frame continuous laser point clouds and multi-frame continuous images, and each frame of the laser point cloud includes multiple laser points; based on the temporal data, target laser points associated with a target object are determined from the multi-frame continuous laser point clouds; wherein the target object is located in the surrounding scene of the vehicle; 3D reconstruction is performed based on the target laser points associated with the target object to obtain an initial 3D model of the target object; in response to the failure of the initial 3D model reconstruction, a target object template matching the target object is obtained from multiple preset object templates based on the target feature descriptor of the target object; wherein the target feature descriptor is determined based on the distance between the target voxel and the bottom center point of the target object among multiple voxels obtained by voxelization of the target laser points, and the target voxel is determined along a preset sampling direction; the initial 3D model and the target object template are fused to obtain a target 3D model of the target object. Therefore, by utilizing spatiotemporally aligned multi-frame continuous laser point clouds and multi-frame continuous images acquired during vehicle operation, point cloud data of the target object can be fully accumulated from temporal data, effectively overcoming the information deficiency caused by sparse, occluded, and structurally missing single-frame point clouds, and improving the completeness and accuracy of target laser point extraction. Simultaneously, when the initial 3D model reconstruction fails, by voxelizing the target laser points, selecting target voxels based on preset sampling directions, and constructing corresponding target feature descriptors based on the distance between the voxel and the bottom center point of the target object, the spatial structure and shape features of the target object can be accurately depicted. For example, it can effectively depict the differentiated key structural features of irregularly shaped target objects such as pump truck booms, mixer truck tanks, forklift forks, and multi-section trailer bodies, achieving efficient and robust matching with preset object templates. Then, the reconstructed... The fusion and complementation of failed initial 3D models with standard templates can preserve the true posture and detailed features of real-world target objects, while using templates to fill in missing structures and correct morphological deviations, thereby improving the integrity, contour accuracy, and effectiveness of 3D models. This helps to enhance the fault tolerance and operational stability of vehicle-mounted 3D reconstruction in complex road conditions in autonomous driving scenarios. It can effectively solve the problems of contour distortion, structural incompleteness, and morphological deviation that easily occur in the reconstruction of irregular target objects in related technologies, improve the integrity, geometric accuracy, and morphological restoration of irregular vehicle 3D models, enhance the fault tolerance and robustness of irregular target reconstruction in complex driving scenarios, and improve the geometric accuracy and morphological restoration effect of irregular vehicle models. It can provide reliable 3D data support for intelligent driving to accurately perceive irregular obstacles, calculate vehicle size, judge behavior, and make driving decisions.

[0256] The collection, storage, use, processing, transmission, provision, and disclosure of users' personal information are all conducted with the consent of the users and comply with relevant laws and regulations, and do not violate public order and good morals.

[0257] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, a computer program product, and an autonomous vehicle. The autonomous vehicle may include the electronic device provided in this disclosure.

[0258] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0259] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0260] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0261] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the three-dimensional reconstruction method. For example, in some embodiments, the three-dimensional reconstruction method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the three-dimensional reconstruction method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the three-dimensional reconstruction method by any other suitable means (e.g., by means of firmware).

[0262] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0263] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0264] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0265] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0266] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0267] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0268] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0269] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A three-dimensional reconstruction method, the method comprising: Acquire time-series data during vehicle operation; wherein, the time-series data includes spatiotemporally aligned multi-frame continuous laser point clouds and multi-frame continuous images, and any frame of the laser point cloud includes multiple laser points; Based on the time-series data, target laser points associated with the target object are determined from the multi-frame continuous laser point cloud; wherein, the target object is located in the surrounding scene of the vehicle. Based on the target laser points associated with the target object, a three-dimensional reconstruction is performed to obtain the initial three-dimensional model of the target object; In response to the failure of the initial 3D model reconstruction, a target object template matching the target object is obtained from multiple preset object templates based on the target feature descriptor of the target object; wherein, the target feature descriptor is determined based on the distance between the target voxel and the bottom center point of the target object among multiple voxels obtained by voxelization of the target laser point, and the target voxel is determined along a preset sampling direction; The initial 3D model is fused with the target object template to obtain the target 3D model of the target object.

2. The method according to claim 1, wherein, The method for determining the target feature descriptor includes the following steps: The target laser points associated with the target object are voxelized to obtain the plurality of voxels; Based on the position information of the target laser point, the bottom center point of the target object is determined; Multiple sampling rays are constructed in a hemispherical space with the bottom center point as the origin and a radius of a set distance to determine the preset sampling direction; For any of the sampling rays, starting from the origin, the density of non-empty voxels and the corresponding center-to-center distance are collected along the sampling ray with a set step size; wherein, the density is used to indicate the number of target laser points in the corresponding voxel, and the center-to-center distance is used to indicate the distance between the corresponding voxel and the origin. The voxel with the highest density among the non-empty voxels collected along the sampling ray is taken as the target voxel, and the distance from the center of the target voxel is taken as the feature distance corresponding to the sampling ray. The feature distances corresponding to the multiple sampled rays are fused to obtain the target feature descriptor.

3. The method according to claim 1, wherein, The method for determining the target feature descriptor includes the following steps: For any of the laser point clouds, the target laser points associated with the target object in the laser point cloud are voxelized to obtain multiple voxels of the target object under the laser point cloud; Based on the position information of the target laser point associated with the target object in the laser point cloud, the bottom center point of the target object under the laser point cloud is determined; Based on multiple voxels of the target object under the laser point cloud, a sub-feature descriptor of the target object under the laser point cloud is determined; wherein, the sub-feature descriptor is determined based on the distance between the target voxel determined along the preset sampling direction and the bottom center point of the target object under the laser point cloud; The target feature descriptor is obtained by fusing the sub-feature descriptors corresponding to the target object under the multi-frame continuous laser point cloud.

4. The method according to claim 3, wherein, Each of the sub-feature descriptors includes values ​​for multiple dimensions; The step of fusing the sub-feature descriptors corresponding to the target object under the multi-frame continuous laser point cloud to obtain the target feature descriptor includes: Multiple keyframe laser point clouds are determined from the multi-frame continuous laser point clouds; For any of the dimensions, the maximum value of the sub-feature descriptor corresponding to the target object under the multiple keyframe laser point clouds in the dimension is determined as the target value corresponding to the dimension; Based on the target values ​​corresponding to the multiple dimensions, the target feature descriptor is generated.

5. The method according to claim 1, wherein, Each of the preset object templates has a corresponding template feature descriptor; the step of obtaining a target object template matching the target object from multiple preset object templates based on the target feature descriptor of the target object includes: For any of the preset object templates, the descriptor similarity between the preset object template and the target object is determined based on the template feature descriptor corresponding to the preset object template and the target feature descriptor. The target object template is determined from the plurality of preset object templates based on the descriptor similarity corresponding to each preset object template.

6. The method according to claim 5, wherein, Both the target feature descriptor and the template feature descriptor include values ​​for multiple dimensions; The step of determining the descriptor similarity between the preset object template and the target object based on the template feature descriptor corresponding to the preset object template and the target feature descriptor includes: For any of the dimensions, the weight corresponding to the dimension is determined based on the values ​​of the template feature descriptor corresponding to the preset object template and the target feature descriptor corresponding to the dimension, respectively. Based on the weights corresponding to the dimensions, and the difference between the values ​​of the template feature descriptors and the target feature descriptors corresponding to the dimensions, the first coefficient corresponding to the dimensions is determined. The first coefficients corresponding to the multiple dimensions are fused to obtain the second coefficient; The descriptor similarity is determined based on the second coefficient.

7. The method according to claim 5, wherein, Before determining the target object template from the plurality of preset object templates based on the descriptor similarity corresponding to each preset object template, the method further includes: Based on the initial 3D model, a first compact frame corresponding to the target object is constructed, and the first volume size of the first compact frame is determined; For any of the preset object templates, determine the second volume size of the second compact frame corresponding to the preset object template; In response to the fact that the difference between the first volume size and the second volume size is less than a first preset threshold, the preset object template is used as a candidate object template; The target object template is determined from at least one candidate object template based on the descriptor similarity corresponding to at least one of the candidate object templates.

8. The method according to claim 1, wherein, The step of determining the target laser point associated with the target object from the multi-frame continuous laser point cloud based on the time-series data includes: For any frame of the laser point cloud, based on the laser point cloud and the corresponding image, point cloud semantic information and a target detection box corresponding to the laser point cloud are generated; wherein, the target detection box is used to define the target object and the corresponding laser point. Based on the point cloud semantic information and / or the target detection box, candidate laser points associated with the target object are extracted from the laser point cloud; Candidate laser points associated with the target object in the multi-frame continuous laser point cloud are identified as the target laser points.

9. The method according to claim 8, wherein, The target detection box has corresponding location information; the step of extracting candidate laser points associated with the target object from the laser point cloud based on the point cloud semantic information and / or the target detection box includes: Use the laser point cloud as the current frame laser point cloud; Based on the position information of the target detection box corresponding to the target object in the multi-frame continuous laser point cloud, the motion state of the target object is determined; Based on the motion state, the point cloud semantic information, and / or the target detection box, candidate laser points associated with the target object are extracted from the current frame laser point cloud.

10. The method according to claim 9, wherein, The step of extracting candidate laser points associated with the target object from the current frame laser point cloud based on the motion state, the point cloud semantic information, and / or the target detection box includes: In response to the motion state being static, the candidate laser points are extracted from the current frame laser point cloud based on the point cloud semantic information and target detection box corresponding to the current frame laser point cloud.

11. The method according to claim 10, wherein, In response to the motion state being static, the process of extracting the candidate laser points from the current frame laser point cloud based on the point cloud semantic information and target detection box corresponding to the current frame laser point cloud includes: The laser point other than the first laser point in the multiple laser points in the current frame laser point cloud is identified as the second laser point; wherein, the first laser point is located within the target detection box of the target object in the current frame laser point cloud; Based on the laser points in the second laser point whose point cloud semantic information is consistent with the point cloud semantic information corresponding to the first laser point, and the first laser point, the candidate laser point is determined.

12. The method according to claim 9, wherein, Each of the laser points has a corresponding velocity vector; the step of extracting candidate laser points associated with the target object from the current frame laser point cloud based on the motion state, the point cloud semantic information, and / or the target detection box includes: In response to the motion state being dynamic, based on the velocity vectors corresponding to multiple laser points in the current frame laser point cloud, the multiple laser points in the current frame laser point cloud are clustered to obtain at least one target cluster. Based on the target detection bounding box of the target object in the current frame laser point cloud and the at least one target cluster, the candidate laser points are extracted from the current frame laser point cloud.

13. The method according to claim 12, wherein, The step of clustering multiple laser points in the current frame laser point cloud based on the velocity vectors corresponding to these laser points to obtain at least one target cluster includes: Traverse multiple laser points in the current frame's laser point cloud; For any third laser point encountered during the traversal, if the third laser point is determined to be a core point, an initial cluster is constructed based on the third laser point and its spatiotemporal nearest neighbor set. The core point is determined based on the number of laser points within the spatiotemporal nearest neighbor set of the corresponding laser point. The spatial distance between the laser points in the spatiotemporal nearest neighbor set and the third laser point is less than a second preset threshold, and the difference in motion vectors between them is less than a third preset threshold. This difference in motion vectors is determined based on the velocity vectors corresponding to the two laser points. The process of processing the initial cluster at least once includes, in any one processing step, traversing the fourth laser point in the initial cluster of the current processing, and if there is a difference between the spatiotemporal nearest neighbor set of the fourth laser point and the initial cluster of the current processing, merging the spatiotemporal nearest neighbor set of the fourth laser point into the initial cluster of the current processing to obtain the initial cluster of the next processing step; wherein the fourth laser point is a core point and is a laser point that has not been traversed. If the fourth laser point is not present in the initial cluster of the current processing, or if the set of spatiotemporal nearest neighbors of the fourth laser point does not have a difference set with the initial cluster of the current processing, the initial cluster of the current processing is determined as the target cluster.

14. The method according to claim 12, wherein, The method further includes: Deletion processing is performed on target clusters in at least one target cluster where the number of laser points is less than a first preset threshold; and / or, For any of the target clusters, a rigid body motion model is constructed based on the position information of the laser points in the target cluster; based on the rigid body motion model, the motion residuals of the laser points in the target cluster are determined; and laser points in the target cluster whose motion residuals are greater than a preset residual threshold are removed from the target cluster.

15. The method according to claim 14, wherein, The method further includes: For any of the target clusters, laser points in the target clusters whose motion residuals are greater than a preset residual threshold are identified as first noise points; Traverse the first noise point in the laser point cloud of the current frame; For any of the first noise points encountered during the traversal, if the first noise point is determined to be a core point, a first cluster is constructed based on the first noise point and the set of its spatiotemporal nearest neighbors. The process of processing the first cluster at least once includes, in any one processing step, traversing the fifth laser point in the first cluster being processed, and if there is a difference between the spatiotemporal nearest neighbor set of the fifth laser point and the first cluster being processed, merging the spatiotemporal nearest neighbor set of the fifth laser point into the first cluster being processed, to obtain the first cluster being processed next; wherein the fifth laser point is a core point and is a first noise point that has not been traversed. If the fifth laser point is not present in the first cluster of the current processing, the first cluster of the current processing is determined as the target cluster.

16. The method according to claim 12, wherein, The current frame laser point cloud includes a second noise point, which is a laser point that does not belong to any target cluster in the at least one target cluster; the method further includes: For any of the target clusters, the spatial centroid of the target cluster is determined based on the position information of the laser points in the target cluster; The velocity vectors corresponding to the laser points in the target cluster are averaged to determine the target motion parameters of the target cluster. For any second noise point in the laser point cloud, in response to the spatial distance between the second noise point and the spatial centroid of the target cluster being less than a fourth preset threshold, and the difference between the velocity vector corresponding to the second noise point and the motion vector of the target cluster being less than the third preset threshold, the target cluster is determined to be a second cluster associated with the second noise point; wherein, the fourth preset threshold is greater than the second preset threshold; The cluster with the smallest spatial distance to the second noise point in the second cluster is determined as the third cluster; The second noise point is added to the third cluster.

17. The method according to claim 12, wherein, The step of extracting the candidate laser points from the current frame laser point cloud based on the target detection bounding box of the target object in the current frame laser point cloud and the at least one target cluster includes: Based on the first laser point located within the target detection box of the target object in the current frame laser point cloud, construct a target laser point set; For any of the target clusters, determine whether there is an intersection between the target cluster and the target laser point set; In response to the intersection between the target cluster and the target laser point set, the candidate laser point is determined based on the laser points in the target cluster and the target laser point set.

18. The method according to claim 12, wherein, The step of determining the candidate laser points associated with the target object in the multi-frame continuous laser point cloud as the target laser point includes: For any frame of the multi-frame continuous laser point cloud, based on the velocity vectors corresponding to the candidate laser points associated with the target object in the laser point cloud, the inter-frame motion parameters of the target object are determined; wherein, the inter-frame motion parameters include rotation matrices and translation vectors; Based on the inter-frame motion parameters, motion compensation and temporal registration are performed on the candidate laser points associated with the target object in the multi-frame continuous laser point cloud to obtain aligned candidate laser points. The aligned candidate laser points are determined as target laser points associated with the target object.

19. The method according to claim 8, wherein, The target objects are multiple; After performing three-dimensional reconstruction based on the target laser points associated with the target object to obtain an initial three-dimensional model of the target object, the method further includes: For any one of the target objects, the target object is determined as a first object, and the other objects among the multiple target objects other than the first object are determined as second objects; Based on the position information of the target detection box corresponding to the first object in the multi-frame continuous laser point cloud and the position information of the target detection box corresponding to the second object in the multi-frame continuous laser point cloud, the target occlusion rate corresponding to the first object is determined. In response to the target occlusion rate being greater than a set occlusion rate threshold, it is determined that the initial 3D model reconstruction of the first object has failed.

20. The method according to claim 8, wherein, After performing three-dimensional reconstruction based on the target laser points associated with the target object to obtain an initial three-dimensional model of the target object, the method further includes: Based on the initial 3D model, construct the third compact bounding box of the target object; In response to the third compact box satisfying the target condition, it is determined that the initial 3D model reconstruction of the target object has failed. The target conditions include at least one or more of the following combinations: The difference between the length of the third compact frame and the target length of the target detection frame is greater than a set length threshold; The difference between the width of the third compact frame and the target width of the target detection frame is greater than a set width threshold; The difference between the height of the third compact frame and the target height of the target detection frame is greater than a set height threshold.

21. The method according to claim 8, wherein, After performing three-dimensional reconstruction based on the target laser points associated with the target object to obtain an initial three-dimensional model of the target object, the method further includes: Project the sixth laser point belonging to the target part among the target laser points associated with the target object onto the target coordinate plane to obtain the coordinates of the sixth laser point in the target coordinate plane; Based on the coordinates of the sixth laser point in the target coordinate plane, determine the projected area of ​​the sixth laser point in the target coordinate plane; For any frame of the laser point cloud, determine the area of ​​the target object's target detection box under the laser point cloud in the target coordinate plane; The target ratio is determined based on the ratio between the projected area and the first area of ​​the target detection box of the target object under the laser point cloud in each frame on the target coordinate plane. In response to the target ratio being less than a fifth preset threshold, it is determined that the initial 3D model reconstruction of the target object has failed.

22. The method according to any one of claims 1-21, wherein, The plurality of preset object templates are stored in a point cloud template library; the method further includes: In response to the successful reconstruction of the initial 3D model, a new preset object template is generated based on the initial 3D model; Save the new preset object template to the point cloud template library.

23. A three-dimensional reconstruction apparatus, the apparatus comprising: The first acquisition module is used to acquire time-series data during vehicle operation; wherein, the time-series data includes spatiotemporally aligned multi-frame continuous laser point clouds and multi-frame continuous images, and any frame of the laser point cloud includes multiple laser points; The first determining module is used to determine the target laser point associated with the target object from the multi-frame continuous laser point cloud based on the time-series data; wherein the target object is located in the surrounding scene of the vehicle. The reconstruction module is used to perform three-dimensional reconstruction based on the target laser points associated with the target object to obtain an initial three-dimensional model of the target object. The second acquisition module is used to, in response to the failure of the initial 3D model reconstruction, acquire a target object template matching the target object from multiple preset object templates based on the target feature descriptor of the target object; wherein, the target feature descriptor is determined based on the distance between the target voxel and the bottom center point of the target object among multiple voxels obtained by voxelization of the target laser point, and the target voxel is determined along a preset sampling direction; The first fusion module is used to fuse the initial 3D model with the target object template to obtain the target 3D model of the target object.

24. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 22.

25. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 22.

26. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 22.

27. An autonomous vehicle, including the electronic equipment as claimed in claim 24.

Images