Special vehicle three-dimensional reconstruction method and device, autonomous vehicle and electronic equipment

Abstract

The special vehicle three-dimensional reconstruction method, device, autonomous vehicle and electronic equipment provided by the present disclosure relate to the technical fields of autonomous driving, maps, perception, large models and the like. The specific implementation scheme is: obtaining state estimation information of a special vehicle in a to-be-processed scene; obtaining a scene flow point cloud clustering result of the to-be-processed scene; according to the state estimation information of the special vehicle and the scene flow point cloud clustering result, completing point clouds of the special vehicle; taking the state estimation information of the special vehicle as an initial value, and obtaining an optimized pose and a reconstructed point cloud of the special vehicle through multi-frame point cloud registration and factor graph global optimization. The present disclosure can improve the quality of the special vehicle reconstruction result.

Description

Technical Field

[0001] This disclosure relates to the field of data processing, particularly to the fields of autonomous driving, mapping, perception, and large-scale models, and especially to methods, devices, autonomous vehicles, and electronic devices for 3D reconstruction of special vehicles. Background Technology

[0002] 3D occupancy grid (OCC) maps are a core data representation form for describing the spatial occupancy status of the surrounding environment. They are widely used in intelligent vehicles, smart cockpits, artificial intelligence / AI, or scenarios combining vehicles and AI, in areas such as the production of ground truth (for perception training data) and environmental modeling for planning and decision-making modules. The OCC production process requires the separate 3D reconstruction of foreground objects (such as vehicles and pedestrians) and background objects (such as roads and buildings) in the scene, which are then fused to generate a complete occupancy grid map.

[0003] In the foreground reconstruction stage, mainstream solutions typically employ template matching: a library of standard vehicle 3D CAD (Computer-Aided Design) models is pre-built, BEV (Bird's-Eye-View) state estimation results are obtained through object detection, and then the closest template model is matched and placed at the corresponding pose to generate an occupancy grid. This method works well for conventional standard vehicle models. However, real-world road scenarios involve special vehicles (such as cargo trucks carrying irregularly shaped goods, various engineering vehicles, extra-long / extra-wide trailers, agricultural vehicles, and modified vehicles), and currently, there is a lack of effective methods for reconstructing dynamic special vehicles. Summary of the Invention

[0004] This disclosure provides a method, apparatus, autonomous vehicle, and electronic equipment for three-dimensional reconstruction of special vehicles, which can solve the problem of the lack of effective means for reconstructing dynamic special vehicles in related technologies.

[0005] According to a first aspect of this disclosure, a method for 3D reconstruction of a special vehicle is provided, comprising: acquiring state estimation information of a special vehicle in a scene to be processed; acquiring scene flow point cloud clustering results of the scene to be processed; completing the point cloud of the special vehicle based on the state estimation information of the special vehicle and the scene flow point cloud clustering results; and obtaining the optimized pose and reconstructed point cloud of the special vehicle by using the state estimation information of the special vehicle as the initial value and through multi-frame point cloud registration and factor graph global optimization.

[0006] According to a second aspect of this disclosure, a special vehicle 3D reconstruction device is provided, comprising: a first acquisition module for acquiring state estimation information of a special vehicle in a scene to be processed; a second acquisition module for acquiring scene flow point cloud clustering results of the scene to be processed; a point cloud completion module for completing the point cloud of the special vehicle based on the state estimation information of the special vehicle and the scene flow point cloud clustering results; and a reconstruction module for obtaining the optimized pose and reconstructed point cloud of the special vehicle by using the state estimation information of the special vehicle as the initial value and through multi-frame point cloud registration and factor graph global optimization.

[0007] According to a third aspect of this disclosure, an autonomous vehicle is provided, comprising: a domain controller for performing the method described in the first aspect above.

[0008] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect above.

[0009] According to a fifth aspect of this disclosure, a storage medium is provided that stores instructions, wherein when the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method described in the first aspect above.

[0010] According to a sixth aspect of this disclosure, a program product is provided, comprising at least one of a program and instructions, wherein the program and instructions are executed by an electronic device to implement the steps of the method described in the first aspect.

[0011] According to the technical solution of this disclosure, the point cloud of a special vehicle is completed by using the state estimation information of the special vehicle and the clustering results of the scene flow point cloud, so that the point cloud of protruding components can be effectively included in the reconstruction process. Therefore, this disclosure directly reconstructs the true 3D morphology of a special vehicle from observation data through multi-source data fusion and data-driven methods, completely eliminating the dependence on a limited template library. No matter how special the shape of the special vehicle is (such as oversized cargo, construction machinery, or irregularly shaped modifications), it can be reconstructed through data-driven methods, improving the reconstruction quality of special vehicles. It can be applied to intelligent vehicles, intelligent cockpits, artificial intelligence / AI, or scenarios combining vehicles and artificial intelligence, and greatly expands the coverage of special vehicles in the automated production process of OCC truth data.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0014] Figure 1 A flowchart of a special vehicle three-dimensional reconstruction method provided in this embodiment of the disclosure; Figure 2 A flowchart of a special vehicle three-dimensional reconstruction method provided in this embodiment of the disclosure; Figure 3 A flowchart of a special vehicle three-dimensional reconstruction method provided in this disclosure embodiment; Figure 4 A block diagram of a special vehicle three-dimensional reconstruction device provided in an embodiment of this disclosure; Figure 5 A block diagram of a special vehicle three-dimensional reconstruction device provided in an embodiment of this disclosure; Figure 6 A block diagram of a special vehicle three-dimensional reconstruction device provided in an embodiment of this disclosure; Figure 7 Example diagram of an autonomous vehicle provided in this disclosure embodiment; Figure 8 This is a block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0015] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0016] The examples in this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with relevant laws, regulations, and provisions. In the examples of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario, and the scope of the solution is not limited in this regard.

[0017] In this manual and the sample solutions, any processing of personal information will be conducted only under legal grounds (such as obtaining the consent of the data subject or being necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0018] This disclosure is not exhaustive, but merely illustrative of some embodiments, and is not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined, for example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0019] In each of the disclosed embodiments, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of the embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0020] The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure.

[0021] In some embodiments, terms such as “multiple,” “at least two,” and “two or more” can be used interchangeably.

[0022] In some embodiments, “including A,” “containing A,” “for indicating A,” and “carrying A” can be interpreted as directly carrying A or indirectly indicating A.

[0023] In some embodiments, the terms “in response to…”, “in response to determining…”, “in the case of…”, “when…”, “if…”, “if…”, etc., can be used interchangeably.

[0024] In some embodiments, the apparatus and device may be interpreted as physical or virtual, and their names are not limited to those described in the embodiments. In some cases, they may also be understood as "equipment", "device", "circuit", "node", "function", "unit", "section", "system", "network", "chip", "chip system", "entity", "body", etc.

[0025] 3D occupancy grid (OCC) maps are a core data representation form for describing the spatial occupancy status of the surrounding environment. They are widely used in intelligent vehicles, smart cockpits, artificial intelligence / AI, or scenarios combining vehicles and AI, in areas such as the production of ground truth (for perception training data) and environmental modeling for planning and decision-making modules. The OCC production process requires the separate 3D reconstruction of foreground objects (such as vehicles and pedestrians) and background objects (such as roads and buildings) in the scene, which are then fused to generate a complete occupancy grid map.

[0026] In the foreground reconstruction stage, mainstream solutions in related technologies typically employ a template library matching approach: a 3D CAD (Computer Aided Design) model library of standard vehicle models is pre-established, BEV (Bird's-Eye-View) state estimation results are obtained through object detection, and then the closest template model is matched and placed at the corresponding pose to generate an occupancy grid. This method works well for conventional standard vehicle models (such as sedans).

[0027] However, this solution faces three core problems in real-world road scenarios: Problem 1: The template library is limited and cannot cover the diverse forms of special vehicles (such as cargo trucks carrying irregularly shaped goods, various engineering vehicles, extra-long / extra-wide trailers, agricultural vehicles, modified vehicles, etc.). In real-world road scenarios, there are numerous special vehicles with varying shapes, whose appearances differ significantly from standard templates. The limited template library simply cannot exhaustively represent all possible irregular shapes, leading to severely distorted template matching results. This is especially true for non-rigid large vehicles (such as excavator booms that can swing or trailers that can bend), where the shape of the same vehicle changes at different times, and static templates are completely unable to represent this dynamic deformation.

[0028] Question 2: Incomplete detection of protruding parts on special vehicles leads to missing parts in dynamic vehicle reconstruction. Special vehicles often have protruding parts that extend beyond the standard vehicle body contour, such as crane booms, excavator buckets, oversized / overweight cargo stacked on trucks, and equipment mounted on the roof. These protruding parts are often truncated or missed during the 3D target detection stage. The bounding boxes output by the detector are usually predicted based on the main body area of ​​the vehicle, and protruding parts easily exceed the bounding box range. Furthermore, for moving special vehicles, the point cloud from a single-frame LiDAR scan is sparse, with particularly insufficient coverage of protruding parts, resulting in obvious missing parts in the reconstruction results.

[0029] Issue 3: Template library matching (or template matching method) relies too heavily on BEV state estimation results. The reconstruction quality of the template library matching scheme is highly dependent on the accuracy of BEV state estimation. Once the state estimation is inaccurate, the placement and orientation of the template will be significantly wrong. For special vehicles, due to their irregular shape and uneven point cloud distribution, the accuracy of BEV state estimation itself is significantly lower than that of standard models (especially orientation and size estimation). This error is further amplified in the template matching process, resulting in OCC annotation quality for special vehicles being far lower than that for standard vehicles.

[0030] Based on this, this disclosure provides a method for 3D reconstruction of special vehicles, and in particular, aims to provide a dynamic high-precision 3D reconstruction method for special vehicles that does not rely on template libraries or BEV state estimation. It directly reconstructs the true 3D shape of special vehicles from observation data through multi-source data fusion and data-driven approach, which can solve the above three core problems. It can be applied to intelligent vehicles, intelligent cockpits, artificial intelligence / AI, or the combination of vehicles and artificial intelligence, as well as the automated production process of OCC ground truth data.

[0031] The following description, with reference to the accompanying drawings, outlines a method, apparatus, autonomous vehicle, and electronic device for three-dimensional reconstruction of a special vehicle according to embodiments of this disclosure.

[0032] It should be noted that the execution entity of the special vehicle 3D reconstruction method in this embodiment can be a special vehicle 3D reconstruction device, which can be implemented by software and / or hardware, and can be configured in a vehicle (such as an autonomous vehicle) or an electronic device. When the device is configured in an electronic device, the electronic device can include, but is not limited to, a terminal, a server, etc.

[0033] Figure 1 A flowchart illustrating a specific vehicle 3D reconstruction method provided in this embodiment of the disclosure. Figure 1 As shown, the special vehicle three-dimensional reconstruction method disclosed herein may include, but is not limited to, the following steps.

[0034] In step 101, state estimation information of special vehicles in the scene to be processed is obtained.

[0035] In some embodiments, the scenario to be processed can be understood as the road environment scenario in which a special vehicle is located.

[0036] In some embodiments, the special vehicle may refer to vehicles of various shapes and sizes, such as including but not limited to one or more of the following: cargo trucks carrying irregular goods (the shape and stacking of the goods vary greatly), various engineering vehicles (such as excavators, cranes, pump trucks, mixer trucks, etc.), extra-long / extra-wide trailers, agricultural vehicles, modified vehicles, etc.

[0037] In some possible embodiments, special vehicles in the scene to be processed can be detected based on a multi-frame point cloud sequence of the scene to be processed, so as to obtain state estimation information of special vehicles in the scene to be processed.

[0038] In some embodiments, the multi-frame point cloud sequence of the scene to be processed can be a multi-frame point cloud sequence obtained by a sensor (such as, but not limited to, a LiDAR unit) acquiring the scene to be processed. The multi-frame point cloud sequence can be a set of three-dimensional spatial data that are continuously acquired and arranged at different time points, recording a 3D scene that is continuously moving or changing.

[0039] Given a multi-frame point cloud sequence of the scene to be processed, special vehicles in the scene can be detected based on this sequence to obtain state estimation information for the special vehicles. Optionally, this state estimation information can be BEV state estimation information (or BEV state estimation result).

[0040] In one possible implementation, the method of detecting special vehicles in the scene to be processed based on a multi-frame point cloud sequence to obtain state estimation information of special vehicles in the scene to be processed may include: aligning the multi-frame point cloud sequence to obtain an aligned multi-frame point cloud sequence; calculating the scene flow vector of each point between adjacent frames based on the aligned multi-frame point cloud sequence; performing motion / static background separation based on the scene flow vector to obtain a subset of point clouds containing moving objects; performing non-rigid target consistency clustering on the moving point clouds in the subset of point clouds to obtain candidate special vehicle point cloud clusters; establishing cross-frame target association and trajectory maintenance based on the candidate special vehicle point cloud clusters; projecting the point cloud cluster onto the BEV space for each special vehicle; estimating the geometric state parameters of the special vehicle in the BEV space, estimating the motion state parameters of the special vehicle, thereby obtaining the state estimation information of the special vehicle. The state estimation information of the special vehicle may include, but is not limited to, one or more of the following: the special vehicle's position (e.g., center coordinates), orientation, size (e.g., length, width), velocity vector, etc.

[0041] For example, the point clouds in a multi-frame point cloud sequence can be transformed to the same world coordinate system to eliminate coordinate offsets caused by sensor motion, resulting in a spatiotemporally aligned set of point cloud frames. For two consecutive aligned point cloud frames, the three-dimensional pose vector (scene flow information or scene flow vector) of each point between frames is calculated using nearest neighbor search or a deep learning scene flow network as the instantaneous motion feature of that point. A dynamic threshold is set based on the magnitude of the scene flow vector, and points with a magnitude greater than the threshold are recorded as moving points. These moving points are then separated from the static background to obtain a subset of point clouds containing only moving objects. On this subset of moving points, density clustering or graph neural network clustering methods can be used to group spatially adjacent points with similar motion patterns (similar scene flow vector directions and magnitudes) into the same special vehicle cluster, excluding discrete noise points, thus obtaining candidate special vehicle point cloud clusters. For each candidate point cloud cluster, calculate its matching cost with the existing target trajectory in the previous frame. The matching cost may include the Euclidean distance between the center points, the point cloud overlap rate, and the shape descriptor similarity. Data association is then performed, and historical trajectory IDs (identifiers) are assigned to each target in the current frame. For each specific vehicle, the height information (e.g., Z-axis) in its 3D point cloud coordinates is ignored or compressed using statistics of the height dimension (e.g., maximum value, average value), resulting in a bird's-eye view 2D point set for that specific vehicle. Contour extraction is performed on the 2D point set in the BEV space, and geometric state parameters can be obtained using at least one of the following methods: Fit the smallest circumscribed convex polygon as the target BEV occupied region; The centroid of the point set is calculated as the BEV center position of the target; Calculate the 2D bounding box (rotated rectangle) along the main direction to obtain its length, width, and orientation angle.

[0042] Based on the sequence of BEV center positions of each target in multiple consecutive frames, Kalman filtering or sliding window least squares method can be used to estimate the instantaneous velocity vector (v_x, v_y) and acceleration of the current frame, which serve as motion state information. The identifier ID, BEV center coordinates (x, y), orientation angle, length, width, velocity vector (v_x, v_y), and corresponding timestamp of each special vehicle are combined into structured data as the final state estimation information, thus obtaining the state estimation information of the special vehicle in each frame.

[0043] In step 102, the scene flow point cloud clustering results of the scene to be processed are obtained.

[0044] In some embodiments, scene flow estimation can be performed based on a multi-frame point cloud sequence of the scene to be processed to obtain scene flow information for each point; point cloud clustering can be performed based on the scene flow information for each point to obtain scene flow point cloud clustering results; wherein, the scene flow point cloud clustering results may include several clusters, and each cluster may correspond to an independent moving object instance.

[0045] For example, for two consecutive frames of point clouds, nearest neighbor search or a deep learning scene flow network can be used to calculate the 3D pose vector (scene flow information or scene flow vector) of each point between frames. The 3D pose vector (scene flow information) of each point is then used to cluster the point cloud using a clustering method. For example, the similarity between the 3D pose vectors of each point can be calculated, and point clouds with similarity greater than or equal to a preset threshold can be clustered to obtain the scene flow point cloud clustering result.

[0046] In step 103, the point cloud of the special vehicle is completed based on the state estimation information of the special vehicle and the scene flow point cloud clustering results.

[0047] In the embodiments of this disclosure, the point cloud integrity of the protruding parts of the special vehicle can be extracted based on the state estimation information of the special vehicle and the scene flow point cloud clustering results, and the point cloud of the protruding parts can be incorporated into the point cloud of the special vehicle to expand the point cloud of the special vehicle, thereby improving the integrity of the point cloud of the special vehicle.

[0048] In some embodiments, based on the state estimation information of a special vehicle and the scene flow point cloud clustering results, the point clouds of protruding components with consistent motion but truncated by detection boxes can be associated with the corresponding special vehicle to complete the point cloud of that special vehicle, ensuring that the protruding components participate in the reconstruction. Scene flow point cloud clustering fusion can solve the reconstruction gap problem caused by the truncation of protruding component detection. Common protruding components such as crane booms, oversized cargo, and excavator buckets can be effectively included in the reconstruction process.

[0049] In step 104, using the state estimation information of the special vehicle as the initial value, the optimized pose and reconstructed point cloud of the special vehicle are obtained through multi-frame point cloud registration and factor map global optimization.

[0050] It should be noted that after fusing the point cloud of the protruding component to the point cloud of the corresponding special vehicle, the state estimation information of the special vehicle after fusing the point cloud of the protruding component can be used as the initial value. Through multi-frame point cloud registration and global optimization of the factor map, the optimized pose and reconstructed point cloud of the special vehicle can be obtained. In other words, participating in the reconstruction with the point cloud of the protruding component can improve the quality of the reconstruction results of the special vehicle.

[0051] In some embodiments, the state estimation information of the special vehicle can be used as the initial value to perform iterative closest point (ICP) registration on point clouds of multiple consecutive frames to generate inter-frame relative motion constraints; a factor map can be constructed based on the inter-frame relative motion constraints, which can cover all constraints of multiple obstacles and multiple frames; all constraints are globally jointly optimized under the framework of the factor map to obtain the optimized pose of the special vehicle; based on the optimized pose of the special vehicle, the original point cloud of the special vehicle is fused and reconstructed to obtain the reconstructed point cloud of the special vehicle.

[0052] For example, the state estimation information of a special vehicle can be used as the initial value. More accurate inter-frame relative motion constraints can be obtained through registration between point clouds of multiple frames. Then, the factor graph framework can be used to perform global joint optimization of all constraints of multiple obstacles and multiple frames to obtain accurate frame-by-frame pose and high-quality reconstructed point clouds.

[0053] The aforementioned "global joint optimization of all constraints of multiple obstacles and multiple frames using the factor graph framework" can refer to putting all constraints such as observation, motion, and relative relationship of all obstacles in all time frames into a factor graph in a scenario where multiple moving or stationary obstacles exist simultaneously. This allows for the simultaneous solution of the optimal state (such as pose and velocity) of each obstacle at each time moment, thereby avoiding the neglect of cross constraints and accumulated errors when optimizing each obstacle or frame individually. Optionally, all constraints for the multi-obstacle, multi-frame configuration may include, but are not limited to, one or more of the following: inter-frame odometry constraints for a single obstacle (for each obstacle, the relative pose change between adjacent frames can be constrained by point cloud registration or visual odometry, which is a temporal smoothing constraint for each obstacle itself); global prior constraints for a single obstacle (e.g., fixed pose in the first frame (first frame prior), or absolute pose prior given by GPS / inertial sensors); spatial relative constraints between obstacles (fixed or dynamic relative pose relationships exist between different obstacles in the same frame); closed-loop constraints across frames and obstacles (the same obstacle returns to the same position at different times, and the relative observations between different obstacles at different times); sensor observation constraints (factors that directly correlate the original point cloud or image features with the obstacle state, such as the observation model of each obstacle in each frame (e.g., reprojection error, point cloud matching residual)); motion model constraints (if it is assumed that the obstacle conforms to a uniform or constant turning rate motion model, velocity / acceleration consistency constraints between adjacent frames can be added).

[0054] It is worth noting that the aforementioned "global joint optimization" can integrate all the constraints (such as temporal constraints of each obstacle itself, spatial constraints between obstacles, global closed loops, etc.) into a unified factor graph, where the state variables are the poses (and possible motion parameters) of all obstacles at all times. The benefits of this approach are as follows: the pose correction of an obstacle can be passed to other obstacles through common-view constraints, achieving information sharing; conflicts between the trajectories of different obstacles (such as penetration, discontinuities in relative distance, etc.) can be avoided, improving consistency; when tracking of an obstacle is lost, its trajectory can be constrained by the constraints of other obstacles, thereby improving robustness. The factor graph optimization framework in this disclosure downgrades the state estimation (BEV state estimation) from "final dependency" to "initial value," correcting the position, orientation, and size deviations in the state estimation through ICP registration + global optimization, with particularly significant effects on the orientation correction of special vehicles.

[0055] In the above embodiments, the point cloud of the special vehicle is completed based on the state estimation information of the special vehicle and the clustering results of the scene flow point cloud, so that the point cloud of the protruding parts can be effectively included in the reconstruction process. Therefore, this disclosure directly reconstructs the true 3D form of the special vehicle from the observation data through multi-source data fusion and data-driven methods, completely eliminating the dependence on a limited template library. No matter how special the form of the special vehicle is (such as oversized cargo, construction machinery, or irregularly shaped modifications), it can be reconstructed through a data-driven approach, improving the reconstruction quality of special vehicles. It can be applied to intelligent vehicles, intelligent cockpits, artificial intelligence / AI, or scenarios combining vehicles and artificial intelligence, and greatly expands the coverage of special vehicles in the automated production process of OCC truth data.

[0056] To further improve the reconstruction quality of special vehicles, a multi-dimensional admission mechanism can be introduced to ensure that only reconstruction results that meet the integrity requirements are included in the OCC annotation. Optionally, Figure 2 A flowchart illustrating a specific vehicle 3D reconstruction method provided in this embodiment of the disclosure. Figure 2 As shown, the special vehicle three-dimensional reconstruction method disclosed herein may include, but is not limited to, the following steps.

[0057] In step 201, the state estimation information of special vehicles in the scene to be processed is obtained.

[0058] Optionally, step 201 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0059] In step 202, the scene flow point cloud clustering results of the scene to be processed are obtained.

[0060] Optionally, step 202 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0061] In step 203, the point cloud of the special vehicle is completed based on the state estimation information of the special vehicle and the scene flow point cloud clustering results.

[0062] Optionally, step 203 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0063] In step 204, using the state estimation information of the special vehicle as the initial value, the optimized pose and reconstructed point cloud of the special vehicle are obtained through multi-frame point cloud registration and factor map global optimization.

[0064] Optionally, step 204 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0065] In step 205, the integrity of the reconstructed point cloud of the special vehicle is evaluated based on multi-dimensional evaluation indicators to obtain integrity evaluation information of the special vehicle reconstruction.

[0066] In some embodiments, the multidimensional evaluation metrics may cover at least two of the following dimensions: occlusion; vehicle length; vehicle height; front end integrity; and horizontal scan angle.

[0067] For example, for the reconstructed point cloud of a special vehicle, integrity assessment can be performed from dimensions such as occlusion, vehicle length, vehicle height, front integrity, and horizontal scanning angle. After obtaining the assessment information for each dimension, mathematical operations can be performed on these dimensions, and the result can be used as the integrity assessment information for the reconstructed special vehicle. These mathematical operations may include, but are not limited to, any of the following: weighting; weighted averaging; or taking the mean. For instance, taking the assessment information as an assessment score, after obtaining the assessment score for each dimension, the assessment scores for these dimensions can be weighted and averaged, and the resulting weighted average score can be used as the integrity assessment information for the reconstructed special vehicle.

[0068] In step 206, if the integrity assessment information meets the preset conditions, the reconstructed point cloud of the special vehicle is voxelized to obtain the three-dimensional occupancy grid of the special vehicle.

[0069] In the embodiments of this disclosure, it can be determined whether the integrity assessment information of the reconstructed special vehicle meets preset conditions. For example, taking the integrity assessment information as an integrity assessment score, it can be determined whether the integrity assessment score of the reconstructed special vehicle is greater than or equal to a preset threshold. If the integrity assessment score is greater than or equal to the preset threshold, it can be considered that the integrity assessment information of the reconstructed special vehicle meets the preset conditions; if the integrity assessment score is less than the preset threshold, it can be considered that the integrity assessment information of the reconstructed special vehicle does not meet the preset conditions.

[0070] If the integrity assessment information of the special vehicle reconstruction meets the preset conditions, the reconstructed point cloud of the special vehicle can be voxelized to obtain the three-dimensional occupancy grid (OCC) of the special vehicle.

[0071] In the above embodiments, the reconstruction quality of special vehicles can be further improved by introducing a multi-dimensional admission mechanism. This ensures that only reconstruction results that meet the integrity requirements are included in the OCC annotation, avoids the pollution of OCC data by low-quality reconstruction, and improves the overall reliability of OCC production.

[0072] To obtain overall OCC data for the scene, the reconstructed point cloud of the specific vehicle can be fused with the reconstruction results of other targets and the background in the scene. Optionally, Figure 3 A flowchart illustrating a specific vehicle 3D reconstruction method provided in this embodiment of the disclosure. Figure 3 As shown, the special vehicle three-dimensional reconstruction method disclosed herein may include, but is not limited to, the following steps.

[0073] In step 301, state estimation information of special vehicles in the scene to be processed is obtained.

[0074] Optionally, step 301 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0075] In step 302, the scene flow point cloud clustering results of the scene to be processed are obtained.

[0076] Optionally, step 302 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0077] In step 303, the point cloud of the special vehicle is completed based on the state estimation information of the special vehicle and the clustering results of the scene flow point cloud.

[0078] Optionally, step 303 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0079] In step 304, using the state estimation information of the special vehicle as the initial value, the optimized pose and reconstructed point cloud of the special vehicle are obtained through multi-frame point cloud registration and factor map global optimization.

[0080] Optionally, step 304 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0081] In step 305, the reconstructed point cloud of the special vehicle is fused with the template matching results of the standard vehicle in the scene to be processed and the background reconstruction results to obtain the scene-level 3D occupancy raster of the scene to be processed.

[0082] Optionally, before fusing the reconstructed point cloud of the special vehicle with the template matching results of the standard vehicle in the scene to be processed and the background reconstruction results, the reconstructed point cloud of the special vehicle can be evaluated for completeness based on multi-dimensional evaluation indicators to obtain the completeness evaluation information of the special vehicle reconstruction. If the completeness evaluation information meets the preset conditions, the reconstructed point cloud of the special vehicle can be fused with the template matching results of the standard vehicle in the scene to be processed and the background reconstruction results to obtain the scene-level three-dimensional occupancy raster of the scene to be processed.

[0083] In the above embodiments, the reconstructed point cloud of special vehicles can be fused with the template matching results of standard vehicles in the scene to be processed and the background reconstruction results to generate a scene-level 3D occupancy raster of the scene to be processed, which can obtain the overall OCC data of the scene, thereby improving the overall reliability of OCC production.

[0084] Figure 4 A block diagram of a special vehicle three-dimensional reconstruction apparatus provided in an embodiment of this disclosure. Figure 4 As shown, the special vehicle 3D reconstruction device may include a first acquisition module 401, a second acquisition module 402, a point cloud completion module 403, and a reconstruction module 404.

[0085] The first acquisition module 401 can be used to acquire state estimation information of special vehicles in the scene to be processed.

[0086] The second acquisition module 402 can be used to acquire the scene flow point cloud clustering results of the scene to be processed.

[0087] The point cloud completion module 403 can be used to complete the point cloud of a special vehicle based on the state estimation information of the special vehicle and the clustering results of the scene flow point cloud.

[0088] The reconstruction module 404 can be used to obtain the optimized pose and reconstructed point cloud of the special vehicle by taking the state estimation information of the special vehicle as the initial value and performing multi-frame point cloud registration and factor map global optimization.

[0089] In some embodiments, the second acquisition module 402 is used to: perform scene flow estimation based on a multi-frame point cloud sequence of the scene to be processed, and obtain scene flow information for each point; perform point cloud clustering based on the scene flow information for each point, and obtain scene flow point cloud clustering results; wherein the scene flow point cloud clustering results include several clusters, and each cluster corresponds to an independent moving object instance.

[0090] In some embodiments, the point cloud completion module 403 is used to: based on the state estimation information of the special vehicle, and using the scene flow point cloud clustering results, associate the point clouds of protruding parts with consistent motion but truncated by the detection box with the corresponding special vehicle, so as to complete the point cloud of the special vehicle.

[0091] In some embodiments, the reconstruction module 404 is configured to: use the state estimation information of the special vehicle as the initial value, perform iterative nearest-point registration on the point cloud of multiple consecutive frames to generate inter-frame relative motion constraints; construct a factor graph based on the inter-frame relative motion constraints, the factor graph covering all constraints of multiple obstacles and multiple frames; perform global joint optimization on all constraints under the factor graph framework to obtain the optimized pose of the special vehicle; and fuse and reconstruct the original point cloud of the special vehicle according to the optimized pose of the special vehicle to obtain the reconstructed point cloud of the special vehicle.

[0092] In some possible embodiments, such as Figure 5 As shown, the special vehicle 3D reconstruction device may further include: an evaluation module 505 and a voxelization processing module 506. The evaluation module 505 is used to: perform an integrity assessment on the reconstructed point cloud of the special vehicle based on multi-dimensional evaluation indicators, obtaining integrity assessment information for the reconstructed special vehicle. The voxelization processing module 506 is used to: perform voxelization processing on the reconstructed point cloud of the special vehicle when the integrity assessment information meets preset conditions, obtaining a 3D occupancy grid of the special vehicle. The multi-dimensional evaluation indicators cover at least two of the following dimensions: occlusion; vehicle length; vehicle height; front integrity; and horizontal scanning angle. Figure 5 501-504 and Figure 4 401-404 in the series have the same function and structure.

[0093] In some possible embodiments, such as Figure 6 As shown, the special vehicle 3D reconstruction device may further include a fusion module 607. The fusion module 607 is used to fuse the reconstructed point cloud of the special vehicle with the template matching results of standard vehicles in the scene to be processed and the background reconstruction results to obtain a scene-level 3D occupancy raster of the scene to be processed. Figure 6 601-606 and Figure 5 501-506 in the series have the same function and structure.

[0094] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0095] Figure 7 An example diagram of an autonomous vehicle provided in an embodiment of this disclosure. (See diagram for reference.) Figure 7 As shown, the autonomous vehicle may include, but is not limited to, domain controller 701. The domain controller 701 can be used to execute the specific vehicle 3D reconstruction method provided in this disclosure.

[0096] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0097] like Figure 8 The diagram shown is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0098] like Figure 8 As shown, the electronic device includes one or more processors 801, a memory 802, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take the 801 processor as an example.

[0099] The memory 802 is the non-transitory computer-readable storage medium provided in this disclosure. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the special vehicle 3D reconstruction method provided in this disclosure. The non-transitory computer-readable storage medium of this disclosure stores computer instructions for causing a computer to perform the special vehicle 3D reconstruction method provided in this disclosure.

[0100] Memory 802, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the special vehicle 3D reconstruction method in this disclosure embodiment (e.g., appendix). Figure 4 The first acquisition module 401, the second acquisition module 402, the point cloud completion module 403, and the reconstruction module 404 are shown. The processor 801 executes various functional applications and data processing of the server by running non-transient software programs, instructions, and modules stored in the memory 802, thereby realizing the special vehicle three-dimensional reconstruction method in the above method embodiment.

[0101] The memory 802 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 802 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 802 may optionally include memory remotely located relative to the processor 801, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0102] The electronic device may also include an input device 803 and an output device 804. The processor 801, memory 802, input device 803, and output device 804 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0103] Input device 803 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as touch screens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, joysticks, etc. Output device 804 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The display device may include, but is not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touch screen.

[0104] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0105] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0107] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0108] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0109] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0110] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A special method for three-dimensional reconstruction of vehicles, comprising: Obtain state estimation information for special vehicles in the scene to be processed; Obtain the scene flow point cloud clustering results of the scene to be processed; Based on the state estimation information of the special vehicle and the clustering results of the scene flow point cloud, the point cloud of the special vehicle is completed; Using the state estimation information of the special vehicle as the initial value, the optimized pose and reconstructed point cloud of the special vehicle are obtained through multi-frame point cloud registration and factor map global optimization.

2. The method according to claim 1, wherein, The step of obtaining the scene flow point cloud clustering result of the scene to be processed includes: Scene flow estimation is performed based on the multi-frame point cloud sequence of the scene to be processed to obtain scene flow information for each point; Point cloud clustering is performed based on the scene flow information of each point to obtain the scene flow point cloud clustering result; wherein, the scene flow point cloud clustering result includes several clusters, and each cluster corresponds to an independent moving object instance.

3. The method according to claim 1, wherein, The step of completing the point cloud of the special vehicle based on the state estimation information of the special vehicle and the scene flow point cloud clustering result includes: Based on the state estimation information of the special vehicle, the point cloud clustering results of the scene flow are used to associate the point cloud of protruding parts that move in the same direction but are truncated by the detection box with the corresponding special vehicle, so as to complete the point cloud of the special vehicle.

4. The method according to claim 1, wherein, The process of obtaining the optimized pose and reconstructed point cloud of the special vehicle by using the state estimation information of the special vehicle as the initial value and through multi-frame point cloud registration and factor map global optimization includes: Using the state estimation information of the special vehicle as the initial value, iterative nearest point registration is performed on multiple consecutive frame point clouds to generate inter-frame relative motion constraints. A factor graph is constructed based on the inter-frame relative motion constraints, and the factor graph covers all constraints for multiple obstacles and multiple frames. Under the aforementioned factor graph framework, all constraints are globally and jointly optimized to obtain the optimized pose of the special vehicle. Based on the optimized pose of the special vehicle, the original point cloud of the special vehicle is fused and reconstructed to obtain the reconstructed point cloud of the special vehicle.

5. The method according to claim 1, further comprising: The integrity of the reconstructed point cloud of the special vehicle is evaluated based on multi-dimensional evaluation indicators to obtain integrity evaluation information of the reconstructed special vehicle. If the integrity assessment information meets the preset conditions, the reconstructed point cloud of the special vehicle is voxelized to obtain the three-dimensional occupancy grid of the special vehicle.

6. The method according to claim 5, wherein, The multi-dimensional evaluation indicators cover at least two of the following dimensions: Obstruction; Vehicle length; Vehicle height; Integrity of the front of the vehicle; Horizontal scanning angle.

7. The method according to any one of claims 1-6, further comprising: The reconstructed point cloud of the special vehicle is fused with the template matching results of the standard vehicle in the scene to be processed and the background reconstruction results to obtain the scene-level 3D occupancy raster of the scene to be processed.

8. A special vehicle three-dimensional reconstruction device, comprising: The first acquisition module is used to acquire state estimation information of special vehicles in the scene to be processed; The second acquisition module is used to acquire the scene flow point cloud clustering results of the scene to be processed; The point cloud completion module is used to complete the point cloud of the special vehicle based on the state estimation information of the special vehicle and the scene flow point cloud clustering results. The reconstruction module is used to obtain the optimized pose and reconstructed point cloud of the special vehicle by taking the state estimation information of the special vehicle as the initial value and performing multi-frame point cloud registration and factor map global optimization.

9. An autonomous vehicle, comprising: A domain controller for performing the method of any one of claims 1-7.

10. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

11. A storage medium storing instructions, wherein, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method of any one of claims 1-7.

12. A program product comprising at least one of a program and instructions, wherein, At least one of the program or instructions is executed by an electronic device to implement the steps of the method according to any one of claims 1-7.

Images