Three-dimensional reconstruction method, device, electronic device, and computer-readable medium

Abstract

This disclosure provides a 3D reconstruction method, apparatus, electronic device, and computer-readable medium in the field of artificial intelligence technology, particularly computer vision, augmented reality, virtual reality, and deep learning. The method can be applied to scenarios such as autonomous driving and intelligent transportation. The specific implementation scheme involves obtaining 3D point cloud data of a scene road, determining the plane equation of at least one road plane corresponding to the scene road based on the 3D point cloud data, determining a 3D mesh model corresponding to the road plane based on the plane equation, and generating a 3D reconstruction model based on the 3D mesh model.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence technology, particularly computer vision, augmented reality, virtual reality, deep learning, and other technical fields, and can be applied to scenarios such as autonomous driving and intelligent transportation. Specifically, the present disclosure relates to a three-dimensional reconstruction method, apparatus, electronic device, and computer-readable medium. Background Art

[0002] 3D reconstruction refers to the establishment of a mathematical model suitable for computer representation and processing of three-dimensional objects, and is a key technology in virtual reality.

[0003] Three-dimensional reconstruction of roads has important applications in data labeling (such as obstacle labeling), road obstacle detection, scene roaming, etc. Summary of the Invention

[0004] The present disclosure provides a three-dimensional reconstruction method, device, electronic device, and computer-readable medium.

[0005] According to a first aspect of the present disclosure, a three-dimensional reconstruction method is provided, the method comprising:

[0006] Acquire three-dimensional point cloud data of a scene road, and determine a plane equation of at least one road plane corresponding to the scene road based on the three-dimensional point cloud data;

[0007] Determining a three-dimensional grid model corresponding to the road plane according to a plane equation of the road plane;

[0008] A three-dimensional reconstruction model is generated according to the three-dimensional mesh model.

[0009] According to a second aspect of the present disclosure, a three-dimensional reconstruction device is provided, the device comprising:

[0010] A plane equation module, configured to obtain three-dimensional point cloud data of a scene road, and determine a plane equation of at least one road plane corresponding to the scene road based on the three-dimensional point cloud data;

[0011] A three-dimensional grid module, configured to determine a three-dimensional grid model corresponding to the road plane according to a plane equation of the road plane;

[0012] The model generation module is used to generate a three-dimensional reconstruction model based on the three-dimensional grid model.

[0013] According to a third aspect of the present disclosure, an electronic device is provided, including:

[0014] at least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the three-dimensional reconstruction method.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned three-dimensional reconstruction method.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the above-mentioned three-dimensional reconstruction method when executed by a processor.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0021] Figure 1 is a flowchart of a three-dimensional reconstruction method provided by an embodiment of the present disclosure;

[0022] Figure 2 This is a flowchart of some steps of a three-dimensional reconstruction method provided by an embodiment of the present disclosure;

[0023] Figure 3 This is a flowchart of some steps of a three-dimensional reconstruction method provided by an embodiment of the present disclosure;

[0024] Figure 4 This is a flowchart of some steps of a three-dimensional reconstruction method provided by an embodiment of the present disclosure;

[0025] Figure 5 This is a flowchart of some steps of a three-dimensional reconstruction method provided by an embodiment of the present disclosure;

[0026] Figure 6 This is a flowchart of some steps of a three-dimensional reconstruction method provided by an embodiment of the present disclosure;

[0027] Figure 7 This is a flowchart of some steps of a three-dimensional reconstruction method provided by an embodiment of the present disclosure;

[0028] Figure 8This is a process diagram of a specific embodiment of a three-dimensional reconstruction method provided by an embodiment of the present disclosure;

[0029] Figure 9 is a structural schematic diagram of a three-dimensional reconstruction device provided by an embodiment of the present disclosure;

[0030] Figure 10 is a block diagram of an electronic device used to implement the 3D reconstruction method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] In some related technologies, a three-dimensional reconstruction algorithm is used to perform dense three-dimensional reconstruction of the road, thereby obtaining dense three-dimensional point cloud data of the road.

[0033] However, although the reconstructed point cloud data is dense three-dimensional point cloud data, it still cannot depict the entire road. There are still some holes without corresponding three-dimensional point cloud data. Therefore, the reconstructed point cloud data is difficult to accurately depict the texture of the road.

[0034] The three-dimensional reconstruction method, device, electronic device, and computer-readable medium provided by the embodiments of the present disclosure are intended to solve at least one of the above technical problems in the prior art.

[0035] The 3D reconstruction method provided in the embodiments of the present disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be an in-vehicle device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. The method can be implemented by a processor invoking computer-readable program instructions stored in a memory. Alternatively, the method can be executed by a server.

[0036] Figure 1 A schematic diagram of a three-dimensional reconstruction method provided by an embodiment of the present disclosure is shown in FIG. Figure 1 As shown in , the method may mainly include step S110, step S120, and step S130.

[0037] In step S110, three-dimensional point cloud data of a scene road is obtained, and a plane equation of at least one road plane corresponding to the scene road is determined based on the three-dimensional point cloud data;

[0038] In step S120, a three-dimensional grid model corresponding to the road plane is determined according to the plane equation of the road plane;

[0039] In step S130 , a three-dimensional reconstructed model is generated based on the three-dimensional mesh model.

[0040] For example, in step S110 , the three-dimensional point cloud data of the scene road may be three-dimensional point cloud data acquired by radar, sensors, etc.

[0041] In some possible implementations, the three-dimensional point cloud data of the scene road may be dense three-dimensional point cloud data obtained by performing dense three-dimensional reconstruction on the scene road.

[0042] In some possible implementations, the three-dimensional point cloud data of the scene road can be obtained by processing the road image data using a sparse three-dimensional reconstruction algorithm or a dense three-dimensional reconstruction algorithm.

[0043] In some possible implementations, the road image data may be image data generated by capturing a scene road by a camera. In other words, the road image data may be image data obtained by capturing a scene road by a camera.

[0044] In some possible implementations, the camera may be a camera mounted on a moving vehicle.

[0045] In some possible implementations, the scene road corresponds to multiple road planes.

[0046] Since the road surface environment of the scene road may be relatively complex (such as uneven road surface, turns, uphill and downhill sections, etc.), the road surface of the scene road does not constitute a road plane and does not satisfy a unified plane equation.

[0047] However, the road in the local area of ​​the scene road can be approximated as a road plane. Therefore, the scene road can be divided into small areas, and each area corresponds to a road plane.

[0048] In some possible implementations, three-dimensional point cloud data corresponding to the road plane (i.e., plane point cloud data corresponding to the road plane) is obtained from the three-dimensional point cloud data of the scene road, and the plane equation corresponding to the road plane is determined based on the three-dimensional coordinates of the point cloud in the plane point cloud data.

[0049] In some possible implementations, in step S120 , after determining the plane equation corresponding to the road plane, a mesh (three-dimensional mesh model) corresponding to the road plane is determined according to a range of a point cloud corresponding to the road plane.

[0050] The three-dimensional mesh model may be formed by a polygonal mesh composed of adjacent point clouds, wherein the polygons may include triangles, quadrilaterals, or other convex polygons.

[0051] For example, taking a triangular mesh as an example, a triangular mesh is composed of vertices, edges, and faces. Each triangular mesh consists of three vertices and three edges. Each vertex and each edge can be shared with other triangular meshes. Each triangular mesh corresponds to a face. In practical applications, a face can be represented by a list of vertices or edges.

[0052] In some possible implementations, after determining the plane equation corresponding to the road plane, the boundary points can be connected in sequence according to the coordinates of the boundary points of the point cloud range corresponding to the road plane to form a three-dimensional grid model corresponding to the road plane.

[0053] In some possible implementations, the obtained three-dimensional mesh model may be refined to obtain a finer three-dimensional mesh model.

[0054] Since the plane equation of the road plane is obtained, the holes in the plane point cloud data can be determined by the plane equation of the road plane to determine their corresponding three-dimensional point cloud data, thereby filling the holes. Therefore, the three-dimensional grid model obtained according to the plane equation of the road plane can accurately characterize the geometric characteristics of the road plane.

[0055] In some possible implementations, in step S130 , texture mapping may be performed on the three-dimensional mesh model to generate a three-dimensional reconstructed model.

[0056] In the three-dimensional reconstruction method of the embodiment of the present disclosure, the scene road is divided into road planes, and the three-dimensional mesh model corresponding to the road plane is determined by obtaining the plane equation of the road plane. The holes in the three-dimensional point cloud data are filled to achieve accurate depiction of the scene road and accurate texture restoration, thereby achieving more accurate three-dimensional reconstruction of the road scene.

[0057] The following is a detailed description of the three-dimensional reconstruction method provided by the embodiments of the present disclosure.

[0058] As described above, in some possible implementations, the three-dimensional point cloud data of the scene road can be obtained by processing the road image data using a sparse three-dimensional reconstruction algorithm or a dense three-dimensional reconstruction algorithm.

[0059] Figure 2The following is a flow chart showing the steps of processing road image data to obtain 3D point cloud data using sparse 3D reconstruction algorithm and dense 3D reconstruction algorithm. Figure 2 The step of using a sparse 3D reconstruction algorithm and a dense 3D reconstruction algorithm to process the road image data to obtain 3D point cloud data may specifically include step S210, step S220, and step S230.

[0060] In step S210, road image data is obtained, where the road image data is image data generated by capturing a scene road through a camera;

[0061] In step S220, sparse point cloud reconstruction is performed based on the road image data to determine the camera's pose information;

[0062] In step S230 , dense point cloud reconstruction is performed based on the posture information to obtain three-dimensional point cloud data of the scene road.

[0063] For example, in step S210, the road image data may be image data generated by capturing a scene road by a camera. In other words, the road image data may be image data obtained by capturing a scene road by a camera.

[0064] In some possible implementations, the camera may be a camera mounted on a moving vehicle. The moving vehicle is in the middle of a scene road, and the camera mounted on the moving vehicle takes pictures to obtain road image data.

[0065] In some possible implementations, in step S220 , a sparse point cloud reconstruction algorithm may be used to perform sparse point cloud reconstruction based on the road image data, and obtain the camera's pose information.

[0066] The camera's position information includes the camera's intrinsic parameter matrix K and the camera's extrinsic parameter matrix RT composed of the camera's rotation matrix R and translation matrix T.

[0067] In some possible implementations, a scale-invariant feature transform (SIFT) algorithm is used to extract feature points from collected road image data for matching, and the camera's position information is determined based on the matched feature points.

[0068] In some possible implementations, a sparse point cloud reconstruction is performed using an SFM (Structure from Motion) algorithm based on the camera's pose information.

[0069] In some possible implementations, the RANSAC (Random Sample Consensus) algorithm can also be used for sparse point cloud reconstruction.

[0070] The embodiments of the present disclosure do not limit the sparse point cloud reconstruction method used.

[0071] In some possible implementations, in step S230 , a dense point cloud reconstruction algorithm is used to perform dense point cloud reconstruction based on the camera's posture information to obtain dense point cloud data of the scene road as three-dimensional point cloud data of the scene road.

[0072] In some possible implementations, dense point cloud reconstruction is performed using multi-view dense matching (CMVS or PMVS).

[0073] The embodiments of the present disclosure do not limit the dense point cloud reconstruction method used.

[0074] Since dense point cloud data is obtained based on road image data, other objects besides the scene road will inevitably be captured in the process of collecting and obtaining road image data, such as road signs, road obstacles, etc. Therefore, the obtained dense point cloud data can include 3D point cloud data of other objects.

[0075] In some possible implementations, the three-dimensional point cloud data of other objects and the three-dimensional point cloud data of the scene road can be segmented by using an image segmentation method.

[0076] Figure 3 The flowchart shows the specific steps of segmenting the 3D point cloud data of other objects and the 3D point cloud data of the scene road by the image segmentation method. Figure 3 Segmenting the three-dimensional point cloud data of other objects and the three-dimensional point cloud data of the scene road by using the image segmentation method may include step S310 and step S320.

[0077] In step S310, the road image data is segmented to determine the pixels corresponding to the scene road;

[0078] In step S320 , the three-dimensional point cloud data obtained by dense point cloud reconstruction is filtered according to the pixels corresponding to the scene road to obtain the three-dimensional point cloud data of the scene road.

[0079] For example, in step S310 , a pre-trained neural network model for image segmentation may be used to perform image segmentation on the road image data to determine a mask of the scene road, and further determine the pixels corresponding to the scene road in the road image data.

[0080] The embodiments of the present disclosure do not limit the image segmentation method used.

[0081] In step S320 , the three-dimensional point cloud data corresponding to the scene road is determined based on the pixels corresponding to the scene road in the road image data and the camera's position information.

[0082] Compared with directly segmenting 3D point cloud data, image segmentation technology is more mature and the segmentation results are more accurate. Therefore, it is easy to segment the 3D point cloud data of other objects and the 3D point cloud data of scene roads through image segmentation, and the obtained segmentation results are more accurate.

[0083] In some possible implementations, GPS (Global Positioning System) information can be used as prior information to restore scale information during dense point cloud reconstruction to obtain more accurate three-dimensional point cloud data.

[0084] In some specific implementations, GPS information of the road image data is obtained, and dense point cloud reconstruction is performed using the GPS information according to the camera's position information to obtain three-dimensional point cloud data of the scene road.

[0085] In some possible implementations, the GPS information of the road image data may be obtained from the attribute information of the road image data.

[0086] Compared with directly collecting three-dimensional point cloud data, collecting road image data and obtaining three-dimensional point cloud data of scene roads by processing road image data is easier to achieve. It only requires the use of common equipment such as cameras. Therefore, it is easier to promote and apply, and the accuracy of the three-dimensional point cloud data obtained is not lower than that of the directly collected three-dimensional point cloud data.

[0087] As described above, in some possible implementations, the scene road corresponds to multiple road planes.

[0088] In some possible implementations, three-dimensional point cloud data corresponding to the road plane (i.e., plane point cloud data corresponding to the road plane) is obtained from the three-dimensional point cloud data of the scene road, and the plane equation corresponding to the road plane is determined based on the three-dimensional coordinates of the point cloud in the plane point cloud data.

[0089] Figure 4 The following is a flow chart showing the process of determining the plane equation corresponding to the road plane based on the three-dimensional coordinates of the point cloud in the plane point cloud data. Figure 4 Determining the plane equation corresponding to the road plane according to the three-dimensional coordinates of the point cloud in the plane point cloud data may include step S410 and step S420.

[0090] In step S410, plane point cloud data corresponding to the road plane is obtained from the three-dimensional point cloud data;

[0091] In step S420 , the plane equation of the road plane is fitted according to the three-dimensional coordinate data of the plane point cloud data, and equation parameters of the plane equation of the road plane are determined.

[0092] For example, in step S410 , the plane point cloud data may be determined by determining a road centerline of the road plane and calculating a distance between the three-dimensional point cloud and the road centerline.

[0093] In some possible implementations, in step S420, the plane equation of the road plane can be ; a, b, c, d are the equation parameters of the road plane.

[0094] In some possible implementations, the RANSAC algorithm can be used to fit the equation using the three-dimensional coordinate data of the planar point cloud data. , determine the values ​​of a, b, c, and d. The embodiments of the present disclosure do not limit the method of fitting the equation.

[0095] The plane equation corresponding to the plane can accurately describe the plane. By determining the plane equation parameters of the road plane by fitting the equation, the plane equation corresponding to the plane can be obtained relatively accurately, thereby achieving an accurate description of the entire road plane. Furthermore, the plane equation can be used to fill in the holes in the three-dimensional point cloud data.

[0096] Figure 5 The figure shows a flow chart of determining the plane point cloud data by determining the road center line of the road plane and calculating the distance between the three-dimensional point cloud and the road center line. Figure 5 Determining the plane point cloud data by determining the road center line of the road plane and calculating the distance between the three-dimensional point cloud and the road center line may include step S510 and step S520.

[0097] In step S510, a road center line of a road plane is determined;

[0098] In step S520, when the distance between the three-dimensional point cloud and the road center line is less than the width threshold, the three-dimensional point cloud data corresponding to the three-dimensional point cloud is determined to be plane point cloud data;

[0099] The width threshold is half of the width of the scene road.

[0100] In some possible implementations, in step S510, when the road image data is image data generated by capturing a scene road using a camera installed on a moving vehicle, the center line of the road can be determined by the position of the camera because the moving vehicle generally travels in the middle of the scene road to facilitate camera capture.

[0101] In some possible implementations, the distance between each three-dimensional point cloud and the road center line may be calculated by determining the equation of a straight line corresponding to the road center line and calculating the distance between the point and the line.

[0102] When the distance between the three-dimensional point cloud and the road center line is less than a preset width threshold, the three-dimensional point cloud is determined to be a three-dimensional point cloud included in the road plane, and the three-dimensional point cloud data corresponding to the three-dimensional point cloud is a component of the plane point cloud data.

[0103] When the distance between the three-dimensional point cloud and the road center line is not less than a preset width threshold, it means that the three-dimensional point cloud is far from the road center line and does not belong to the three-dimensional point cloud included in the road plane.

[0104] In some specific implementations, the distance between the three-dimensional point cloud and the road centerline may be determined using the following formula.

[0105] ,in, is the three-dimensional coordinate of the three-dimensional point cloud, are the three-dimensional coordinates of all points on the middle line of the road, is the three-dimensional coordinate of the starting point of the road center line, The three-dimensional coordinates of the end point of the road centerline.

[0106] The formula means to calculate the distance between all points of the three-dimensional point cloud and the middle line of the road, and determine the minimum value as the distance between the three-dimensional point cloud and the middle line of the road.

[0107] Of course, the equation of the line corresponding to the road centerline can also be determined using the 3D coordinates of the starting point of the road centerline and the 3D coordinates of the middle point of the road centerline. The distance between the 3D point cloud and the line corresponding to the line equation can then be calculated. The disclosed embodiments do not limit the algorithm used to calculate the distance between the 3D point cloud and the road centerline.

[0108] In some possible implementations, the width threshold is half of the width of the scene road. For example, if the width of the scene road is h, the width threshold is h / 2.

[0109] Since the road centerline is located in the middle of the road plane, the points on the road plane are located on both sides of the road centerline. The distance between the points on the road plane and the road centerline is obviously less than half the road width, and the distance between the points that do not belong to the road plane and the road plane is obviously greater than half the road width. Therefore, by calculating the distance between the three-dimensional point cloud and the road plane centerline, the three-dimensional point cloud belonging to the road plane can be quickly and accurately determined.

[0110] Figure 6 A schematic diagram of the process of determining the center line of the road by the position of the camera is shown. Figure 6 Determining the middle line of the road by the position of the camera may include step S610, step S620, and step S630.

[0111] In step S610, a road length threshold is determined based on the smoothness information of the scene road and the camera movement speed;

[0112] In step S620, a starting camera position and an ending camera position corresponding to each road plane are determined according to the road length threshold;

[0113] In step S630 , a road center line of the road plane is determined according to the starting camera position and the ending camera position.

[0114] In some possible implementations, in step S610 , the camera is a camera that collects road image data of a scene road, and the road image data is used to obtain three-dimensional point cloud data of the scene road.

[0115] In some possible implementations, the smoother the scene road is, the larger the road length threshold may be; and the less smooth the scene road is, the smaller the road length threshold may be.

[0116] In some possible implementations, the faster the vehicle speed and the faster the camera moves, the smaller the road length threshold may be; and the slower the vehicle speed and the slower the camera moves, the larger the road length threshold may be.

[0117] In some possible implementations, in step S620, the scene road is segmented according to the position change of the camera, and the three-dimensional point cloud data corresponding to the road image data collected by the camera from the starting camera position to the ending camera position is used as the plane point cloud data.

[0118] The segmentation should satisfy the following requirement: assuming that the road length threshold is x, the distance between the starting camera position and the corresponding ending camera position corresponding to each road plane should be less than the road length threshold.

[0119] In some specific implementations, si is the start time corresponding to the road section, ei is the end time corresponding to the road section, is the position of the camera at time si (i.e. the starting camera position), is the position of the camera at the moment ei (i.e. the final camera position), then .

[0120] In some possible implementations, in step S630 , a straight line formed by the starting camera position and the ending camera position is determined as the road center line.

[0121] In order to facilitate the collection of road image data by cameras installed on moving vehicles, the moving vehicles generally travel in the middle of the scene road and collect data along the scene road. Therefore, it is feasible and easy to determine the center line of the road by the camera position.

[0122] As described above, in some possible implementations, after the plane equation corresponding to the road plane is determined, the three-dimensional grid model corresponding to the road plane is determined according to the range of the point cloud corresponding to the road plane.

[0123] Figure 7 The schematic diagram of the process of determining the three-dimensional grid model corresponding to the road plane according to the plane equation of the road plane is shown. Figure 7 Determining the three-dimensional grid model corresponding to the road plane according to the plane equation of the road plane may include step S710, step S720, and step S730.

[0124] In step S710, the abscissa and ordinate of the boundary point are determined based on the maximum abscissa and maximum ordinate values ​​in the three-dimensional coordinate data of the plane point cloud data;

[0125] In step S720, the three-dimensional coordinates of the boundary point are determined based on the plane equation of the road plane and the horizontal coordinate and vertical coordinate of the boundary point;

[0126] In step S730 , a three-dimensional grid model corresponding to the road plane is determined based on the three-dimensional coordinates of the boundary points.

[0127] In some possible implementations, in step S710 , the minimum value of the horizontal coordinate, the maximum value of the horizontal coordinate, the minimum value of the vertical coordinate, and the maximum value of the vertical coordinate in the three-dimensional coordinate data of all the planar point cloud data are determined.

[0128] In some specific implementations, use Indicates the minimum x-coordinate value in the three-dimensional coordinate data of the plane point cloud data; use Indicates the maximum x-coordinate value in the three-dimensional coordinate data of the plane point cloud data; use Indicates the minimum y coordinate value in the three-dimensional coordinate data of the plane point cloud data; use Indicates the maximum y-coordinate value in the three-dimensional coordinate data of the planar point cloud data.

[0129] In some specific implementations, in step S720, 、 Substitute the plane equation of the road plane to determine the three-dimensional coordinates of the first boundary point; 、 Substitute the plane equation of the road plane to determine the three-dimensional coordinates of the second boundary point; 、 Substitute the plane equation of the road plane to determine the three-dimensional coordinates of the third boundary point; 、 Substitute the plane equation of the road plane to determine the three-dimensional coordinates of the fourth boundary point.

[0130] In some specific implementations, in step S730, the three-dimensional grid model corresponding to the road plane can be determined based on the three-dimensional coordinates of the first boundary point, the three-dimensional coordinates of the second boundary point, the three-dimensional coordinates of the third boundary point, and the three-dimensional coordinates of the fourth boundary point.

[0131] That is to say, the x-coordinate range and y-coordinate range of the three-dimensional point cloud of the road plane are determined according to the three-dimensional coordinate information of the plane point cloud data, and the plane equation of the road plane is determined. (The z value is calculated using the plane equation of the road plane).

[0132] The first boundary point, the second boundary point, the third boundary point, and the fourth boundary point are connected in sequence to generate two triangular facets, forming a three-dimensional mesh model corresponding to the road plane.

[0133] In some possible implementations, the obtained three-dimensional mesh model may be refined to obtain a finer three-dimensional mesh model. The embodiments of the present disclosure do not limit the refinement method for refining the three-dimensional mesh model.

[0134] Determining the 3D coordinates of the road's boundary points based on the plane equation avoids situations where the corresponding point cloud data for a road boundary point does not exist in the plane point cloud data and the road boundary points cannot be determined. Furthermore, by using the road boundary points to determine the 3D mesh model, the resulting 3D mesh model wraps the road plane as closely as possible, thereby ensuring the most comprehensive capture of the road's geometric features.

[0135] As described above, in some possible implementations, texture mapping may be performed on the three-dimensional mesh model to generate a three-dimensional reconstructed model.

[0136] In some possible implementations, a texture mapping method may be used to perform texture mapping on the acquired three-dimensional mesh model to obtain a three-dimensional mesh model with texture information, thereby determining a three-dimensional reconstructed model corresponding to the road plane.

[0137] In some possible implementations, all color information can be stored in a texture map, and when displayed, rendering is performed according to the texture coordinates of each mesh and the corresponding texture map to obtain a high-resolution color 3D reconstructed model.

[0138] The embodiments of the present disclosure do not limit the texture mapping method. For example, in some possible implementations, the open source code library mvstexturing for 3D reconstruction can be used to apply textures to 3D mesh models.

[0139] In some possible implementations, after determining the 3D reconstruction model corresponding to each road plane, the 3D reconstruction model corresponding to the scene road is determined based on the 3D reconstruction models corresponding to all the road planes.

[0140] Through texture mapping, texture information can be mapped to a three-dimensional mesh model, thereby obtaining a three-dimensional reconstructed model with texture and color information.

[0141] The following describes a three-dimensional reconstruction method provided by an embodiment of the present disclosure using a specific embodiment.

[0142] Figure 8 FIG2 shows a process diagram of a specific embodiment of the three-dimensional reconstruction method provided by the embodiment of the present disclosure. Figure 8 The three-dimensional reconstruction method provided by the disclosed embodiment may include:

[0143] Acquire multiple images of the scene and road captured by a camera installed on the vehicle. To obtain the camera's pose information and a dense point cloud of the scene and road, first perform sparse reconstruction to obtain the camera's pose, and then perform dense reconstruction to generate dense 3D point cloud data of the scene, road, and other static objects. In the reconstruction process, GPS is used as prior information to restore the scale.

[0144] In order to better obtain the three-dimensional point cloud data of the scene road and prevent other objects from affecting the three-dimensional reconstruction of the scene road, the road segmentation algorithm is used to segment the acquired image, and the mask corresponding to the scene road is obtained. Based on the segmented mask, the three-dimensional point cloud data corresponding to other static objects is filtered out in the dense reconstruction result (i.e., dense three-dimensional point cloud data), and only the three-dimensional point cloud data of the scene road is retained.

[0145] Because the road environment is complex, the data collected at one time covers a long distance, and the road may not satisfy the unified road equation. However, in a local area, the road can be approximated as a plane. In order to divide the road into small areas, the three-dimensional point cloud of the scene road needs to be segmented.

[0146] Specifically, let the position of the camera at time t be , according to the vehicle speed and the smoothness of the scene road, set the hyperparameter x, divide the camera position into segments, and the size of each segment should be less than or equal to x, that is , where si represents the start time of the i-th segment (i is a positive integer), ei represents the end time of the i-th segment, assuming that the width of the scene road is h, then for the three-dimensional point cloud data C of the scene road, each point ,have:

[0147]

[0148] That is, calculate each point The minimum value of the distance from all camera positions in the time period from si to ei is less than h / 2, then the point belongs to the i-th road plane, otherwise it does not belong to the i-th road plane.

[0149] According to the obtained segmented point cloud, plane estimation is performed separately, and the four parameters a, b, c, d in the plane equation ax+by+cz+d = 0 are estimated using the ransac algorithm. After completing the plane estimation, the mesh corresponding to the segment point cloud is generated according to the range of the point cloud, specifically the x coordinate range and y coordinate range of the segment point cloud, to determine The Z value is generated according to the plane equation. The four points are connected in sequence to generate two triangular facets. The mesh formed by them is the mesh corresponding to the plane of this section of road.

[0150] After refining the obtained mesh to generate a finer mesh, the mvstexturing method is used to generate the corresponding texture for the mesh to complete the plane texture restoration.

[0151] Based on Figure 1 The same principle as shown in the method, Figure 9 FIG. 1 shows a schematic structural diagram of a three-dimensional reconstruction device provided by an embodiment of the present disclosure, such as Figure 9 As shown, the three-dimensional reconstruction device 90 may include:

[0152] A plane equation module 910 is configured to obtain three-dimensional point cloud data of a scene road and determine a plane equation of at least one road plane corresponding to the scene road based on the three-dimensional point cloud data;

[0153] A three-dimensional grid module 920 is used to determine a three-dimensional grid model corresponding to the road plane according to the plane equation of the road plane;

[0154] The model generation module 930 is used to generate a three-dimensional reconstruction model according to the three-dimensional mesh model.

[0155] In the three-dimensional reconstruction device of the embodiment of the present disclosure, the scene road is divided into road planes, and the three-dimensional mesh model corresponding to the road plane is determined by obtaining the plane equation of the road plane. The holes in the three-dimensional point cloud data are filled to achieve accurate depiction of the scene road and accurate texture restoration, thereby achieving more accurate three-dimensional reconstruction of the road scene.

[0156] It is understandable that the above modules of the 3D reconstruction device in the embodiment of the present disclosure have the function of realizing Figure 1The functions of the corresponding steps of the three-dimensional reconstruction method in the embodiment shown in . This function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and each of the above modules can be implemented separately or integrated into multiple modules. For a detailed description of the functions of each module of the above three-dimensional reconstruction device, please refer to Figure 1 The corresponding description of the three-dimensional reconstruction method in the embodiment shown in is not repeated here.

[0157] In some possible implementations, the plane equation module 910 includes: a segmentation unit for obtaining plane point cloud data corresponding to the road plane from the three-dimensional point cloud data; a calculation unit for fitting the plane equation of the road plane based on the three-dimensional coordinate data of the plane point cloud data and determining the equation parameters of the plane equation of the road plane.

[0158] In some possible implementations, the scene road corresponds to multiple road planes, and the segmentation unit is further used to: determine the road center line of the road plane; when the distance between the three-dimensional point cloud and the road center line is less than a width threshold, determine that the three-dimensional point cloud data corresponding to the three-dimensional point cloud is plane point cloud data; the width threshold is half of the width value of the scene road.

[0159] In some possible implementations, the segmentation unit is further used to: determine a road length threshold based on the flatness information of the scene road and the camera movement speed, where the camera is a camera that collects road image data of the scene road, and the road image data is used to obtain three-dimensional point cloud data of the scene road; determine the starting camera position and the ending camera position corresponding to each road plane based on the road length threshold; and determine the road center line of the road plane based on the starting camera position and the ending camera position.

[0160] In some possible implementations, the three-dimensional grid module 920 includes: a horizontal and vertical coordinate unit, which is used to determine the horizontal coordinate and vertical coordinate of the boundary point based on the horizontal coordinate maximum value and the vertical coordinate maximum value in the three-dimensional coordinate data of the plane point cloud data; a three-dimensional coordinate unit, which is used to determine the three-dimensional coordinates of the boundary point according to the plane equation of the road plane and the horizontal coordinate and vertical coordinate of the boundary point; and a model calculation unit, which is used to determine the three-dimensional grid model corresponding to the road plane based on the three-dimensional coordinates of the boundary point.

[0161] In some possible implementations, the plane equation module 910 includes: an acquisition unit for acquiring road image data, where the road image data is image data generated by acquiring a scene road through a camera; a sparse reconstruction unit for performing sparse point cloud reconstruction based on the road image data to determine the camera's posture information; and a dense reconstruction unit for performing dense point cloud reconstruction based on the posture information to acquire three-dimensional point cloud data of the scene road.

[0162] In some possible implementations, the three-dimensional reconstruction device 90 also includes: an image segmentation module, which is used to perform image segmentation on the road image data and determine the pixels corresponding to the scene road; a filtering module, which is used to filter the three-dimensional point cloud data obtained by dense point cloud reconstruction according to the pixels corresponding to the scene road, and obtain the three-dimensional point cloud data of the scene road.

[0163] In some possible implementations, the dense reconstruction unit is further used to: obtain global positioning system information of the road image data; and reconstruct a dense point cloud using the global positioning system information according to the posture information to obtain three-dimensional point cloud data of the scene road.

[0164] In some possible implementations, the model generation module 930 is further configured to perform texture mapping on the three-dimensional mesh model to generate a three-dimensional reconstructed model.

[0165] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0166] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0167] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the three-dimensional reconstruction method provided in the embodiment of the present disclosure.

[0168] Compared with the existing technology, this electronic device divides the scene road into road planes, determines the three-dimensional grid model corresponding to the road plane by obtaining the plane equation of the road plane, fills the holes in the three-dimensional point cloud data, and achieves accurate depiction of the scene road and accurate texture recovery, thereby achieving more accurate three-dimensional reconstruction of the road scene.

[0169] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the three-dimensional reconstruction method provided by the embodiment of the present disclosure.

[0170] Compared with the existing technology, this readable storage medium divides the scene road into road planes, determines the three-dimensional grid model corresponding to the road plane by obtaining the plane equation of the road plane, fills the holes in the three-dimensional point cloud data, and achieves accurate depiction of the scene road and accurate texture recovery, thereby achieving more accurate three-dimensional reconstruction of the road scene.

[0171] The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the three-dimensional reconstruction method provided in the embodiment of the present disclosure.

[0172] Compared with the existing technology, this computer program product divides the scene road into road planes, determines the three-dimensional grid model corresponding to the road plane by obtaining the plane equation of the road plane, fills the holes in the three-dimensional point cloud data, and achieves accurate depiction of the scene road and accurate texture recovery, thereby achieving more accurate three-dimensional reconstruction of the road scene.

[0173] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0174] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. RAM 1003 may also store various programs and data required for the operation of device 1000. Computing unit 1001, ROM 1002, and RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to bus 1004.

[0175] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0176] The computing unit 1001 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the 3D reconstruction method. For example, in some embodiments, the 3D reconstruction method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the 3D reconstruction method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the 3D reconstruction method via any other suitable means (e.g., via firmware).

[0177] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0178] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0179] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0180] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0181] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0182] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0183] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0184] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A three-dimensional reconstruction method, comprising: Acquire three-dimensional point cloud data of a scene road, and determine a plane equation of at least one road plane corresponding to the scene road based on the three-dimensional point cloud data; Determining a three-dimensional grid model corresponding to the road plane according to a plane equation of the road plane; generating a three-dimensional reconstruction model according to the three-dimensional mesh model; Wherein, determining the plane equation of at least one road plane corresponding to the scene road according to the three-dimensional point cloud data includes: Acquire plane point cloud data corresponding to the road plane from the three-dimensional point cloud data; Fitting a plane equation of the road plane according to the three-dimensional coordinate data of the plane point cloud data to determine equation parameters of the plane equation of the road plane; The scene road corresponds to a plurality of road planes, and obtaining the plane point cloud data corresponding to the road plane from the three-dimensional point cloud data includes: determining a road centerline of the road plane; When the distance between the three-dimensional point cloud and the middle line of the road is less than a width threshold, determining that the three-dimensional point cloud data corresponding to the three-dimensional point cloud is plane point cloud data; the width threshold is half of the width value of the scene road; Wherein, the determining of the road center line of the road plane further includes: determining a road length threshold according to the flatness information of the scene road and a camera movement speed, wherein the camera is a camera for collecting road image data of the scene road, and the road image data is used to obtain three-dimensional point cloud data of the scene road; Determine a starting camera position and an ending camera position corresponding to each road plane according to the road length threshold; Determine a road center line of the road plane according to the starting camera position and the ending camera position; The determining of the three-dimensional grid model corresponding to the road plane according to the plane equation of the road plane includes: According to the coordinates of the boundary points of the point cloud range corresponding to the road plane, the boundary points are connected in sequence to form the three-dimensional grid model corresponding to the road plane.

2. The method according to claim 1, wherein The determining of the three-dimensional grid model corresponding to the road plane according to the plane equation of the road plane includes: Determine the abscissa and ordinate of the boundary point based on the maximum abscissa and maximum ordinate values ​​in the three-dimensional coordinate data of the plane point cloud data; Determining the three-dimensional coordinates of the boundary point according to the plane equation of the road plane and the horizontal coordinate and the vertical coordinate of the boundary point; Based on the three-dimensional coordinates of the boundary points, a three-dimensional grid model corresponding to the road plane is determined.

3. The method according to claim 1, wherein Obtain 3D point cloud data of the scene road, including: Acquire road image data, where the road image data is image data generated by capturing the scene road through a camera; Reconstructing a sparse point cloud based on the road image data to determine the camera's position information; Based on the posture information, dense point cloud reconstruction is performed to obtain three-dimensional point cloud data of the scene road.

4. The method according to claim 3, wherein: After performing dense point cloud reconstruction based on the posture information to obtain three-dimensional point cloud data of the scene road, the method further includes: Performing image segmentation on the road image data to determine pixels corresponding to the scene road; According to the pixels corresponding to the scene road, the three-dimensional point cloud data obtained by dense point cloud reconstruction is filtered to obtain the three-dimensional point cloud data of the scene road.

5. The method according to claim 3, wherein The step of reconstructing a dense point cloud according to the posture information to obtain three-dimensional point cloud data of the scene road includes: Obtaining global positioning system information of the road image data; Based on the posture information, dense point cloud reconstruction is performed using the global positioning system information to obtain three-dimensional point cloud data of the scene road.

6. The method according to claim 1, wherein Generating a three-dimensional reconstruction model according to the three-dimensional mesh model includes: Texture mapping is performed on the three-dimensional mesh model to generate a three-dimensional reconstructed model.

7. A three-dimensional reconstruction device comprising: A plane equation module, configured to obtain three-dimensional point cloud data of a scene road, and determine a plane equation of at least one road plane corresponding to the scene road based on the three-dimensional point cloud data; A three-dimensional grid module, configured to determine a three-dimensional grid model corresponding to the road plane according to a plane equation of the road plane; A model generation module, configured to generate a three-dimensional reconstruction model based on the three-dimensional grid model; Wherein, the plane equation module includes: A segmentation unit, configured to obtain planar point cloud data corresponding to the road plane from the three-dimensional point cloud data; a calculation unit, configured to fit a plane equation of the road plane according to the three-dimensional coordinate data of the plane point cloud data, and determine equation parameters of the plane equation of the road plane; The scene road corresponds to multiple road planes, and the segmentation unit is further used for: determining a road centerline of the road plane; When the distance between the three-dimensional point cloud and the middle line of the road is less than a width threshold, determining that the three-dimensional point cloud data corresponding to the three-dimensional point cloud is plane point cloud data; the width threshold is half of the width value of the scene road; Wherein, the segmentation unit is further used for: determining a road length threshold according to the flatness information of the scene road and a camera movement speed, wherein the camera is a camera for collecting road image data of the scene road, and the road image data is used to obtain three-dimensional point cloud data of the scene road; Determine a starting camera position and an ending camera position corresponding to each road plane according to the road length threshold; Determine a road center line of the road plane according to the starting camera position and the ending camera position; The three-dimensional grid module is used to connect the boundary points in sequence according to the coordinates of the boundary points of the point cloud range corresponding to the road plane to form the three-dimensional grid model corresponding to the road plane.

8. The device according to claim 7, wherein The three-dimensional grid module includes: A horizontal and vertical coordinate unit, configured to determine the horizontal and vertical coordinates of a boundary point based on the maximum horizontal and vertical coordinate values ​​in the three-dimensional coordinate data of the plane point cloud data; a three-dimensional coordinate unit, configured to determine the three-dimensional coordinates of the boundary point according to the plane equation of the road plane and the horizontal coordinate and the vertical coordinate of the boundary point; A model calculation unit is used to determine a three-dimensional grid model corresponding to the road plane based on the three-dimensional coordinates of the boundary points.

9. The device according to claim 7, wherein The plane equation module includes: An acquisition unit, configured to acquire road image data, wherein the road image data is image data generated by acquiring the scene road through a camera; a sparse reconstruction unit, configured to perform sparse point cloud reconstruction based on the road image data and determine the position information of the camera; The dense reconstruction unit is used to perform dense point cloud reconstruction according to the posture information to obtain three-dimensional point cloud data of the scene road.

10. The device according to claim 9, wherein The three-dimensional reconstruction device further includes: An image segmentation module is used to perform image segmentation on the road image data to determine pixels corresponding to the scene road; The filtering module is used to filter the three-dimensional point cloud data obtained by dense point cloud reconstruction according to the pixels corresponding to the scene road, so as to obtain the three-dimensional point cloud data of the scene road.

11. The device according to claim 9, wherein The dense reconstruction unit is further configured to: Obtaining global positioning system information of the road image data; Based on the posture information, dense point cloud reconstruction is performed using the global positioning system information to obtain three-dimensional point cloud data of the scene road.

12. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

14. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images