3D Reconstruction and Real-Scene Restoration Method Based on Deep Fusion of Images and Point Clouds

By using deep fusion technology of point cloud and image, obstacles and ground points are labeled, and plane fitting and triangulation algorithms are used to solve the problem of insufficient accuracy of existing 3D scene reconstruction technology, and more refined 3D reconstruction and realistic scene restoration are achieved.

CN117274453BActive Publication Date: 2026-11-17WUHAN KOTEI INFORMATICS
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311063423.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-08-22
Publication Date
2026-11-17
Estimated Expiration
2043-08-22

AI Technical Summary

Technical Problem

Existing 3D scene reconstruction technology lacks sufficient accuracy, making it difficult to achieve higher-precision scene reconstruction.

Method used

By acquiring point cloud data and image information, marking obstacles and ground points, and using plane fitting and triangulation algorithms, combined with the matching relationship between the camera viewpoint and the deflection angle of the point cloud scanning device, obstacle and ground mapping and restoration are performed to achieve deep fusion of point cloud and image.

Benefits of technology

It enables more refined scene understanding and analysis, improves the accuracy and realism of 3D reconstruction, and has strong reference value, especially in point cloud semantic segmentation and lane line annotation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117274453B_ABST
    Figure CN117274453B_ABST
Patent Text Reader

Abstract

This invention provides a method for 3D reconstruction and realistic scene restoration based on deep fusion of images and point clouds, comprising: acquiring point cloud data and image information from all perspectives; labeling the point cloud data from all perspectives to obtain a set of labeled objects O containing all obstacles; calculating a set of ground points S and a fitted ground plane P in the point cloud scene; acquiring a set of points S1 for the current perspective in the point cloud data and a set of labeled objects O1 for the current perspective in the labeled object set O based on the matching relationship between the camera viewpoint and the deflection angle of the point cloud scanning device; performing mapping and restoration on the labeled objects based on the set of labeled objects O1; acquiring a set of ground points S2 projected from the set of points S1 for the current perspective onto the fitted ground plane P; performing mapping and restoration on the ground based on the set of ground points S2; and traversing the point cloud data and image information from all perspectives to complete the mapping and restoration of all perspectives. This invention enables more refined scene understanding and analysis, resulting in more realistic 3D reconstruction effects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of scene reconstruction technology, and more specifically, to a method for 3D reconstruction and real-world scene reconstruction based on deep fusion of images and point clouds. Background Technology

[0002] Scene recreation generally refers to the rendering and redrawing of a 3D scene using unique materials, lighting, and other auxiliary materials. It is commonly used in fields such as architecture, geology, and engineering. Scene recreation can provide people with a good direction and approach, bridging imagination and reality, and allowing them to use all their senses to understand new things, thereby achieving the goal of displaying the expected effect.

[0003] However, current 3D scene reconstruction technology mainly relies on training 2D images to reconstruct 3D scenes, and there is still considerable room for improvement in scene reconstruction accuracy. Therefore, it is necessary to study 3D scene reconstruction schemes with higher accuracy. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing a method, system, and electronic device for 3D reconstruction and real scene restoration based on deep fusion of images and point clouds. It realistically restores obstacles and the ground in the scene from the point cloud data, thereby achieving more refined scene understanding and analysis and obtaining more realistic 3D reconstruction results.

[0005] According to a first aspect of the present invention, a method for 3D reconstruction and real-world scene restoration based on deep fusion of images and point clouds is provided, comprising:

[0006] Acquire point cloud data and image information from all perspectives, annotate the point cloud data from all perspectives to obtain an annotated object set O containing all obstacles, and calculate the ground point set S and the fitted ground plane P in the point cloud scene.

[0007] Based on the matching relationship between the camera viewpoint and the deflection angle of the point cloud scanning device, obtain the point set S1 under the current viewpoint in the point cloud data and the annotation object set O1 under the current viewpoint in the annotation object set O, and perform texture mapping and restoration on the annotation objects based on the annotation object set O1;

[0008] Obtain the set of points S1 under the current viewpoint and project it onto the set of ground points S2 that fits the ground plane P. Then, perform texture mapping and restoration on the ground based on the set of ground points S2.

[0009] Iterate through the point cloud data and image information from all viewpoints to complete the texturing and restoration of all viewpoints.

[0010] Based on the above technical solution, the present invention can also be improved as follows.

[0011] Optionally, the step of acquiring point cloud data and image information from all viewpoints, and labeling the point cloud data from all viewpoints to obtain a set of labeled objects O containing all obstacles, includes:

[0012] Acquire image information from multiple cameras around the vehicle and point cloud data from multiple radar scans;

[0013] AI pre-annotation or manual annotation is performed based on the current point cloud scene to obtain a set of annotated objects, denoted as O. Each annotated object in the set of annotated objects O contains a center point C and a set of points contained in the current annotated object.

[0014] Optionally, the set of ground points S and the fitted ground plane P in the calculated point cloud scene include:

[0015] Based on the current point cloud scene, the ground point set S and the ground fitting plane P are obtained according to the plane fitting algorithm.

[0016] Optionally, the step of obtaining the point set S1 under the current viewpoint in the point cloud data and the annotation object set O1 under the current viewpoint in the annotation object set O, based on the matching relationship between the camera viewpoint and the deflection angle of the point cloud scanning device, includes:

[0017] Obtain the points of the point cloud data under the current camera viewpoint, filter out the point set S1 under the current viewpoint from the ground point set S based on the points, and obtain the annotation object set O1 under the current viewpoint from the annotation object set O based on the inverse trigonometric function.

[0018] Optionally, the step of mapping and restoring the labeled objects according to the labeled object set O1 includes:

[0019] The convex hull algorithm is used to obtain the set of all outer points G1 in the set of labeled objects O1. The set of outer points G1 is triangulated to obtain the set of triangular meshes T1. The two-dimensional coordinates of each triangle point in the set of triangular meshes T1 to the camera image of that viewpoint are calculated. The two-dimensional coordinates of each triangle point are transformed into UV texture mapping coordinates to obtain the UV array set UV1. The texture mapping and restoration of the labeled objects are completed based on the set of triangular meshes T1 and the UV array set UV1.

[0020] Optionally, the step of calculating the two-dimensional coordinates of each triangle point in the triangular mesh array set T1 to the camera image at that viewpoint, and transforming the two-dimensional coordinates of each triangle point to UV texture map coordinates to obtain the UV array set UV1, includes:

[0021] The triangular mesh array of each labeled object in the triangular mesh array set T1 is transformed from the three-dimensional coordinate system to the two-dimensional coordinate system in the camera image of that viewpoint according to the point cloud image fusion annotation method. Then, the coordinates of each triangle point in the triangular mesh array in the two-dimensional coordinate system are transformed to the UV texture map coordinate system to obtain the UV array set UV1.

[0022] Optionally, the step of obtaining the ground point set S2 projected from the current viewpoint point set S1 onto the fitted ground plane P, and performing texture mapping and restoration on the ground based on the ground point set S2, includes:

[0023] Obtain the ground point set S2 that is projected onto the fitted ground plane P from the current viewpoint set S1. Triangulate the ground point set S2 to obtain the triangular mesh set T2. Calculate the two-dimensional coordinates of each triangle point in the triangular mesh set T2 to the camera image of the current viewpoint. Transform the two-dimensional coordinates of each triangle point to the UV texture mapping coordinates to obtain the UV coordinate set UV2. Complete the mapping and restoration of the ground based on the triangular mesh set T2 and the UV coordinate set UV2.

[0024] Optionally, the calculation of the two-dimensional coordinates of each triangle point in the triangular mesh set T2 to the camera image at that viewpoint, and the transformation of the two-dimensional coordinates of each triangle point to the UV texture map coordinates to obtain the UV coordinate set UV2, includes:

[0025] Each triangular mesh array in the triangular mesh set T2 is transformed from the three-dimensional coordinate system to the two-dimensional coordinate system in the camera image of that viewpoint according to the point cloud image fusion annotation method. Then, the coordinates of each triangle point in the triangular mesh array in the two-dimensional coordinate system are transformed to the UV texture map coordinate system to obtain the UV coordinate point set UV2.

[0026] According to a second aspect of the present invention, a 3D reconstruction and real-scene restoration system based on deep fusion of images and point clouds is provided, comprising:

[0027] The acquisition module is used to acquire point cloud data and image information from all perspectives, and also to annotate the point cloud data from all perspectives to obtain a set of annotated objects O containing all obstacles, as well as to calculate the set of ground points S and the fitted ground plane P in the point cloud scene.

[0028] The obstacle reconstruction module obtains the point set S1 under the current view in the point cloud data and the annotation object set O1 under the current view in the annotation object set O based on the matching relationship between the camera view and the deflection angle of the point cloud scanning device. Then, it performs mapping and restoration on the annotation objects based on the annotation object set O1.

[0029] The ground reconstruction module obtains the ground point set S1 under the current view and projects it onto the ground point set S2 that fits the ground plane P. Based on the ground point set S2, it performs texture mapping and restoration of the ground.

[0030] The loop module is used to traverse the point cloud data and image information of all views to complete the texturing and restoration of all views.

[0031] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the processor is configured to execute a computer management program stored in the memory to implement the steps of the above-described method for three-dimensional reconstruction and real-scene restoration based on deep fusion of images and point clouds.

[0032] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer management program is stored, wherein when the computer management program is executed by a processor, the steps of the above-described method for three-dimensional reconstruction and real-scene restoration based on deep fusion of images and point clouds are implemented.

[0033] This invention provides a method, system, electronic device, and storage medium for 3D reconstruction and realistic scene restoration based on deep fusion of images and point clouds. Utilizing computer vision technology, it constructs a 3D model that recreates a realistic scene using point cloud data. The invention performs scene restoration from point cloud data, primarily using multi-camera images as the material, and separately reconstructs and restores objects such as vehicles, buildings, road signs, etc. (hereinafter collectively referred to as obstacles) and the ground in the scene. This invention realistically restores obstacles and the ground in the scene separately from the point cloud data, providing strong reference for point cloud semantic segmentation and lane line annotation, achieving more refined scene understanding and analysis. Attached Figure Description

[0034] Figure 1 A flowchart of a method for 3D reconstruction and real-world scene restoration based on deep fusion of images and point clouds provided by this invention;

[0035] Figure 2 A flowchart of a method for 3D reconstruction and real-world scene restoration based on deep fusion of images and point clouds, provided for one embodiment;

[0036] Figure 3 A block diagram of a 3D reconstruction and real-scene restoration system based on deep fusion of images and point clouds provided by the present invention;

[0037] Figure 4 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;

[0038] Figure 5 This is a schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. Detailed Implementation

[0039] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0040] Figure 1 This invention provides a flowchart of a 3D reconstruction and real-scene restoration method based on deep fusion of images and point clouds, which mainly illustrates the concept of the method. Figure 2 This is a flowchart illustrating a method for 3D reconstruction and real-world scene restoration based on deep fusion of images and point clouds, provided in a specific embodiment. (Combined with...) Figure 1 and Figure 2 As shown, the method includes: acquiring point cloud data and image information from all perspectives, labeling point cloud data from all perspectives to obtain a set of labeled objects O containing all obstacles, and calculating the set of ground points S and the fitted ground plane P in the point cloud scene.

[0041] Based on the matching relationship between the camera viewpoint and the deflection angle of the point cloud scanning device, obtain the point set S1 under the current viewpoint in the point cloud data and the annotation object set O1 under the current viewpoint in the annotation object set O, and perform texture mapping and restoration on the annotation objects based on the annotation object set O1;

[0042] Obtain the set of points S1 under the current viewpoint and project it onto the set of ground points S2 that fits the ground plane P. Then, perform texture mapping and restoration on the ground based on the set of ground points S2.

[0043] Iterate through the point cloud data and image information from all viewpoints to complete the texturing and restoration of all viewpoints.

[0044] Understandably, given the shortcomings in the background technology, this invention proposes a method for 3D reconstruction and realistic scene restoration based on deep fusion of images and point clouds. This method realistically restores obstacles and the ground in the scene from the point cloud data, providing strong reference for point cloud semantic segmentation and lane line annotation, and achieving more refined scene understanding and analysis.

[0045] Now combined Figure 2 An example will be given to illustrate a specific implementation scenario.

[0046] It should be noted that the point cloud 3D scene reconstruction in this embodiment can be implemented based on the open-source 3D engine Three.js (hereinafter referred to as the engine). All terms mentioned in this patent (such as UV coordinate system, UV mapping, scene rendering, planar projection, normal, vector, etc.) can be instantiated using the engine. Furthermore, the "image point cloud fusion annotation," "plane fitting algorithm," "convex hull algorithm," and "Delaunay triangulation algorithm" used in the detailed implementation process described below are conclusions drawn from already implemented functions and algorithms; their specific operational processes are not included within the scope of this patent.

[0047] Based on point cloud and image fusion technology, the 3D coordinates in a point cloud scene can be transformed into 2D coordinates through a series of matrix transformations using the intrinsic and extrinsic parameters of the vehicle-mounted camera and LiDAR. Secondly, a plane fitting algorithm can be used to obtain the set of ground points S and the fitted ground plane P in this point cloud scene.

[0048] In this embodiment, six cameras are used to collect image information from multiple perspectives around the vehicle, and multiple LiDAR scanners are used to collect point cloud data from different fields of view around the vehicle.

[0049] The horizontal deflection angle of the lidar is denoted as α (clockwise is positive and counterclockwise is negative); the field of view range of N cameras and the image set are denoted as C.

[0050] For the acquired point cloud data and image information from multiple perspectives, such as Figure 2 As shown, perform the following steps:

[0051] 1. Based on the current point cloud scene, perform AI pre-annotation or manual annotation to obtain a set of annotated objects, denoted as O. It can be understood that the annotated objects here are objects such as cars, buildings, road signs, etc. in the scene, which are also collectively referred to as obstacles. The set of annotated objects O contains all the marked obstacles. The annotated objects contain the center point C of the obstacle and the set of points contained in the obstacle.

[0052] 2. Based on the point cloud scene, obtain the ground point set S and the ground fitting plane P according to the plane fitting algorithm;

[0053] 3. Take the viewpoint of any camera as the current viewpoint, obtain the parameters of that camera, and filter all point cloud data according to the horizontal tilt angle α of the LiDAR and the camera's viewpoint range. Obtain the point set S1 under the current viewpoint from all point cloud data. The specific filtering method is as follows:

[0054] a. Loop through all points in the point cloud scene from all viewpoints;

[0055] b. Calculate the arctangent value β in the horizontal direction for each point;

[0056] c. The angle of deflection of the lidar relative to the camera is γ = β + α. Determine whether the current angle of deflection γ is within the camera's field of view. If so, it is determined that the angle of deflection of the lidar is within the current field of view of the camera.

[0057] 4. Similarly, we can determine whether the labeled object is within the current camera view by using the center coordinates C of the labeled object in the scene, and filter the labeled object set O to obtain the current camera view labeled object set O1;

[0058] 5. Traverse the set of labeled objects O1 from the current camera viewpoint, and perform convex hull algorithm processing on the points of each labeled object to obtain the set of outer point arrays G1 for each labeled object;

[0059] 6. Perform Delaunay triangulation algorithm on each point group in the outer point array set G1 to obtain the triangulated mesh array set T1 of the labeled object; where every three points in the triangulated mesh array set T1 represent a triangle, and subsequently, a set (UV1) will be obtained by calculating the two-dimensional coordinates of each triangle point to the camera image of that path and then calculating the UV coordinates corresponding to each point.

[0060] 7. For each labeled object in the triangular mesh array set T1, the UV texture map coordinates of each point are calculated simultaneously according to the coordinate transformation of "point cloud image fusion annotation", so as to obtain the UV array set UV1 of the labeled object;

[0061] 8. Based on the UV array set UV1 and the triangulated mesh array set T1, the engine performs texturing and restoration on each labeled object in the labeled object set O1 from the front camera's perspective. At the same time, the camera image pixels of the texturing part are set to transparent. At this point, the texturing and restoration of the labeled objects are completed.

[0062] 9. Based on the ground point set S obtained in step 2 and the current viewpoint point set S1 in step 3, filter out the ground point set S2 under the current viewpoint; the purpose of this step is to ensure that all scene information is in the same plane when the ground is restored.

[0063] 10. Calculate the set of projected coordinates S3 of the ground points in the ground point set S2 under the current viewpoint to the fitting plane P based on the fitting plane P in step 2;

[0064] Using the projected coordinate set S3, the Delaunay triangulation algorithm is used to obtain the triangulated grid array set T2 of the ground;

[0065] 11. Repeat step 7 to perform "point cloud image fusion annotation" coordinate transformation on the triangular mesh array set T2 and calculate the UV texture map coordinate points corresponding to each triangular point in the triangular mesh array set T2 to obtain the UV coordinate point array set UV2 of the ground object.

[0066] 12. Repeat step 8 to map and restore the ground in the current view based on the UV coordinate point array set UV2 and the triangulated mesh array set T2. At this point, the point cloud scene mapping and restoration in the current camera view is completed.

[0067] 13. Repeat steps 3 to 12 to apply textures and restore the other camera views according to the annotation engine. Once completed, the restoration of the entire point cloud scene is finished.

[0068] Figure 3 A structural diagram of a 3D reconstruction and real-scene restoration system based on deep fusion of images and point clouds provided in an embodiment of the present invention is shown below. Figure 3 As shown, a 3D reconstruction and real-scene restoration system based on deep fusion of images and point clouds includes an acquisition module, an obstacle reconstruction module, a ground reconstruction module, and a looping module, wherein:

[0069] The acquisition module is used to acquire point cloud data and image information from all perspectives, and also to annotate the point cloud data from all perspectives to obtain a set of annotated objects O containing all obstacles, as well as to calculate the set of ground points S and the fitted ground plane P in the point cloud scene.

[0070] The obstacle reconstruction module obtains the point set S1 under the current view in the point cloud data and the annotation object set O1 under the current view in the annotation object set O based on the matching relationship between the camera view and the deflection angle of the point cloud scanning device. Then, it performs mapping and restoration on the annotation objects based on the annotation object set O1.

[0071] The ground reconstruction module obtains the ground point set S1 under the current view and projects it onto the ground point set S2 that fits the ground plane P. Based on the ground point set S2, it performs texture mapping and restoration of the ground.

[0072] The loop module is used to traverse the point cloud data and image information of all views to complete the texturing and restoration of all views.

[0073] It is understood that the 3D reconstruction and real scene restoration system based on deep fusion of images and point clouds provided by the present invention corresponds to the 3D reconstruction and real scene restoration method based on deep fusion of images and point clouds provided in the foregoing embodiments. The relevant technical features of the 3D reconstruction and real scene restoration system based on deep fusion of images and point clouds can be referred to the relevant technical features of the 3D reconstruction and real scene restoration method based on deep fusion of images and point clouds, and will not be repeated here.

[0074] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 4 As shown, an embodiment of the present invention provides an electronic device 400, including a memory 410, a processor 420, and a computer program 411 stored in the memory 410 and executable on the processor 420. When the processor 420 executes the computer program 411, it performs the following steps:

[0075] Acquire point cloud data and image information from all perspectives, annotate the point cloud data from all perspectives to obtain an annotated object set O containing all obstacles, and calculate the ground point set S and the fitted ground plane P in the point cloud scene.

[0076] Based on the matching relationship between the camera viewpoint and the deflection angle of the point cloud scanning device, obtain the point set S1 under the current viewpoint in the point cloud data and the annotation object set O1 under the current viewpoint in the annotation object set O, and perform texture mapping and restoration on the annotation objects based on the annotation object set O1;

[0077] Obtain the set of points S1 under the current viewpoint and project it onto the set of ground points S2 that fits the ground plane P. Then, perform texture mapping and restoration on the ground based on the set of ground points S2.

[0078] Iterate through the point cloud data and image information from all viewpoints to complete the texturing and restoration of all viewpoints.

[0079] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. (See diagram below.) Figure 5 As shown, this embodiment provides a computer-readable storage medium 500 on which a computer program 511 is stored. When the computer program 511 is executed by a processor, it performs the following steps:

[0080] Acquire point cloud data and image information from all perspectives, annotate the point cloud data from all perspectives to obtain an annotated object set O containing all obstacles, and calculate the ground point set S and the fitted ground plane P in the point cloud scene.

[0081] Based on the matching relationship between the camera viewpoint and the deflection angle of the point cloud scanning device, obtain the point set S1 under the current viewpoint in the point cloud data and the annotation object set O1 under the current viewpoint in the annotation object set O, and perform texture mapping and restoration on the annotation objects based on the annotation object set O1;

[0082] Obtain the set of points S1 under the current viewpoint and project it onto the set of ground points S2 that fits the ground plane P. Then, perform texture mapping and restoration on the ground based on the set of ground points S2.

[0083] Iterate through the point cloud data and image information from all viewpoints to complete the texturing and restoration of all viewpoints.

[0084] This invention provides a method, system, electronic device, and storage medium for 3D reconstruction and realistic scene restoration based on deep fusion of images and point clouds. Based on computer vision technology, it utilizes point cloud data to construct a 3D model that recreates a realistic scene. This invention performs scene restoration from point cloud data, primarily using multi-camera images as the material. It restores and reconstructs objects such as vehicles, buildings, road signs, etc. (hereinafter collectively referred to as obstacles) and the ground in the scene separately. Different algorithms are used to reconstruct the scene based on the points enclosed by the ground and obstacles in the point cloud scene. In the point cloud scene, after obtaining the projection coordinates of points from the image through point cloud and image fusion annotation, the object is 3D reconstructed based on the points contained in the obstacle. Convex hulls and triangular meshes generated using basic algorithms are used for meshing and texturing, thereby achieving scene restoration. The reconstructed object includes, but is not limited to, 3D bounding boxes and semantic segmentation objects. This invention realistically restores obstacles and the ground in the scene separately from the point cloud data, providing strong reference for point cloud semantic segmentation and lane line annotation, achieving more refined scene understanding and analysis, and obtaining more realistic 3D reconstruction results.

[0085] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0086] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0091] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A three-dimensional reconstruction and real scene restoration method based on image and point cloud depth fusion, characterized in that, include: Acquire point cloud data and image information from all perspectives, annotate the point cloud data from all perspectives to obtain an annotated object set O containing all obstacles, and calculate the ground point set S and the fitted ground plane P in the point cloud scene. Based on the matching relationship between the camera's viewpoint and the angle of the point cloud scanning device, the point set S1 under the current viewpoint and the annotation object set O1 under the current viewpoint in the annotation object set O containing all obstacles are obtained from the point cloud data. The annotation objects are then mapped and restored based on the annotation object set O1 under the current viewpoint; including: Obtain the points of the point cloud data under the current camera view, filter out the point set S1 under the current view from the ground point set S based on the points, and obtain the annotation object set O1 under the current view from the annotation object set O containing all obstacles based on the inverse trigonometric function. The convex hull algorithm is used to obtain the set of all outer points G1 in the set of labeled objects O1 under the current view. The set of outer points G1 is triangulated to obtain the set of triangular mesh T1. The two-dimensional coordinates of each triangle point in the set of triangular mesh T1 to the current view camera image are calculated. The two-dimensional coordinates of each triangle point are transformed to UV texture mapping coordinates to obtain the UV array set UV1. The texture mapping and restoration of the labeled objects are completed based on the triangular mesh array set T1 and the UV array set UV1. Obtain the set of points S1 under the current viewpoint and project it onto the set of ground points S2 that fits the ground plane P. Then, perform texture mapping and restoration on the ground based on the set of ground points S2. Iterate through the point cloud data and image information from all viewpoints to complete the texturing and restoration of all viewpoints. 2.The method of claim 1, wherein, The process of acquiring point cloud data and image information from all viewpoints, and labeling the point cloud data from all viewpoints to obtain a set of labeled objects O containing all obstacles, includes: Acquire image information from multiple cameras around the vehicle and point cloud data from multiple radar scans; AI pre-annotation or manual annotation is performed based on the current point cloud scene to obtain a set of annotated objects, denoted as O. Each annotated object in the set of annotated objects O contains a center point C and a set of points contained in the current annotated object. 3.The method of claim 1 or 2, wherein, The set of ground points S and the fitted ground plane P in the computational point cloud scene include: Based on the current point cloud scene, the ground point set S and the fitted ground plane P are obtained according to the plane fitting algorithm.

4. The method of claim 1 or 2, wherein, The calculation of the two-dimensional coordinates of each triangle point in the triangular mesh array T1 to the current viewpoint camera image, and the transformation of the two-dimensional coordinates of each triangle point to UV texture map coordinates to obtain the UV array set UV1, includes: The triangular mesh array of each labeled object in the triangular mesh array set T1 is transformed from the three-dimensional coordinate system to the two-dimensional coordinate system in the current view camera image according to the point cloud image fusion annotation method. Then, the coordinates of each triangle point in the triangular mesh array in the two-dimensional coordinate system are transformed to the UV texture map coordinate system to obtain the UV array set UV1.

5. The method of claim 4, wherein, The process of obtaining a set of points S1 from the current viewpoint and projecting it onto a fitted ground plane P to form a set of ground points S2, and then mapping and restoring the ground based on the set of ground points S2, includes: Obtain the ground point set S2 that is projected onto the fitted ground plane P from the current viewpoint point set S1. Triangulate the ground point set S2 to obtain the triangular mesh set T2. Calculate the two-dimensional coordinates of each triangle point in the triangular mesh set T2 to the current viewpoint camera image. Transform the two-dimensional coordinates of each triangle point to the UV texture mapping coordinates to obtain the UV coordinate set UV2. Complete the mapping and restoration of the ground based on the triangular mesh set T2 and the UV coordinate set UV2.

6. The method for 3D reconstruction and real-scene restoration based on deep fusion of images and point clouds according to claim 5, characterized in that, The calculation of the two-dimensional coordinates of each triangle point in the triangular mesh set T2 to the current viewpoint camera image, and the transformation of the two-dimensional coordinates of each triangle point to the UV texture map coordinates to obtain the UV coordinate set UV2, includes: Each triangular mesh array in the triangular mesh set T2 is transformed from the three-dimensional coordinate system to the two-dimensional coordinate system in the current viewpoint camera image according to the point cloud image fusion annotation method. Then, the coordinates of each triangle point in the triangular mesh array in the two-dimensional coordinate system are transformed to the UV texture map coordinate system to obtain the UV coordinate set UV2.

7. A 3D reconstruction and real-scene restoration system based on deep fusion of images and point clouds, characterized in that, include: The acquisition module is used to acquire point cloud data and image information from all perspectives, and also to annotate the point cloud data from all perspectives to obtain a set of annotated objects O containing all obstacles, as well as to calculate the set of ground points S and the fitted ground plane P in the point cloud scene. The obstacle reconstruction module, based on the matching relationship between the camera's viewpoint and the angle of the point cloud scanning device, obtains the point set S1 from the current viewpoint in the point cloud data and the annotation object set O1 from the current viewpoint within the annotation object set O containing all obstacles. It then performs texturing and reconstruction of the annotation objects based on the annotation object set O1 from the current viewpoint; including: Obtain the points of the point cloud data under the current camera view, filter out the point set S1 under the current view from the ground point set S based on the points, and obtain the annotation object set O1 under the current view from the annotation object set O containing all obstacles based on the inverse trigonometric function. The convex hull algorithm is used to obtain the set of all outer points G1 in the set of labeled objects O1 under the current view. The set of outer points G1 is triangulated to obtain the set of triangular mesh T1. The two-dimensional coordinates of each triangle point in the set of triangular mesh T1 to the current view camera image are calculated. The two-dimensional coordinates of each triangle point are transformed to UV texture mapping coordinates to obtain the UV array set UV1. The texture mapping and restoration of the labeled objects are completed based on the triangular mesh array set T1 and the UV array set UV1. The ground reconstruction module obtains the ground point set S1 under the current view and projects it onto the ground point set S2 that fits the ground plane P. Based on the ground point set S2, it performs texture mapping and restoration of the ground. The loop module is used to traverse the point cloud data and image information of all views to complete the texturing and restoration of all views.

8. An electronic device, characterized in that, The system includes a memory and a processor, wherein the processor is used to execute computer management programs stored in the memory to implement the steps of the three-dimensional reconstruction and real-scene restoration method based on deep fusion of images and point clouds as described in any one of claims 1-6.

Citation Information

Patent Citations

  • Terrain detection method and system for autonomous vehicle and readable storage medium

    CN114842438A

  • Method and system for sensing automated driving environment

    WO2022022694A1