Three-dimensional modeling method, electronic device, system and storage medium

Abstract

The present invention discloses a three-dimensional modeling method, electronic device, system and storage medium. The method comprises: obtaining N original images respectively captured by N cameras, wherein there is a common field of view area between two adjacent original images, N≥2; performing three-dimensional reconstruction on each of the original images, and constructing a three-dimensional sub-model corresponding to each of the original images; and performing fusion processing on the three-dimensional sub-models corresponding to the two adjacent original images with the common field of view area, and obtaining a three-dimensional reconstruction model. The method enables the three-dimensional reconstruction model to fuse the three-dimensional information of the common field of view area between two adjacent original images, so as to ensure the image quality of the three-dimensional reconstruction model, reduce the computational complexity of the fusion processing, and improve the robustness.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a three-dimensional modeling method, electronic equipment, system and storage medium. Background Art

[0002] Cars are becoming an indispensable means of transportation for people to travel. With more and more vehicles on the road, the number of traffic accidents has also increased significantly. In order to reduce the occurrence of traffic accidents, modern smart cars use various detection sensors to detect and perceive the internal and external environment of the vehicle in real time, so as to further automatically control the vehicle's driving or remind the driver of possible driving hazards. For example, using fisheye cameras installed around the vehicle, a three-dimensional reconstruction model can be generated. The three-dimensional reconstruction model is a 360-degree panoramic image. The car driver can observe the image to determine whether there is any danger around the car body, so as to ensure driving safety or facilitate parking.

[0003] The inventors have discovered that the current image fusion processing is mainly composed of stitching together multiple 2D fisheye camera images. The stitched images have gaps and some information is missing, which affects the effect of the image fusion processing. Summary of the invention

[0004] The embodiments of the present invention provide a three-dimensional modeling method, an electronic device, a system and a storage medium to solve the problem of poor quality in existing image fusion processing.

[0005] An embodiment of the present invention provides a three-dimensional modeling method, including:

[0006] Obtain N original images captured by N cameras respectively, where there is a common field of view area between two adjacent original images, and N≥2;

[0007] Performing three-dimensional reconstruction on each of the original images to construct a three-dimensional sub-model corresponding to each of the original images;

[0008] The three-dimensional sub-models corresponding to two adjacent original images having a common field of view are fused to obtain a three-dimensional reconstructed model.

[0009] Preferably, the camera is an M-eye camera, and the original image includes M monocular images, M≥2;

[0010] The three-dimensional reconstruction of each of the original images to construct a three-dimensional sub-model corresponding to each of the original images includes:

[0011] Extracting corner points of the M monocular images respectively to obtain corner points corresponding to the M monocular images;

[0012] Perform feature matching on the corner points corresponding to the M monocular images to obtain corner point matching results;

[0013] Based on the corner point matching result being all the corner points that have been matched, a three-dimensional sub-model corresponding to each of the original images is constructed.

[0014] Preferably, said extracting corner points from the M monocular images respectively to obtain corner points corresponding to the M monocular images comprises:

[0015] Get the camera intrinsic parameters and camera extrinsic parameters corresponding to the M cameras;

[0016] Using the camera intrinsic parameters, perform distortion correction processing on the M monocular images to obtain M corrected images;

[0017] Using the camera extrinsic parameters, aligning the M rectified images to obtain M aligned images;

[0018] Corner points are extracted from the M aligned images respectively to obtain corner points corresponding to the M monocular images.

[0019] Preferably, the step of constructing a three-dimensional sub-model corresponding to each of the original images based on all the corner points matched by the corner point matching result includes:

[0020] Get the camera extrinsics corresponding to the M cameras;

[0021] According to the camera external parameters, for all the corner points that the corner point matching results pass, obtain the three-dimensional coordinates of the corner points corresponding to the corner points;

[0022] A three-dimensional sub-model corresponding to each of the original images is constructed according to the three-dimensional coordinates of the corner points corresponding to all the corner points.

[0023] Preferably, the fusing of the three-dimensional sub-models corresponding to the two adjacent original images having a common field of view to obtain the three-dimensional reconstructed model includes:

[0024] Determine two adjacent original images having a common field of view as a first image and a second image respectively, determine the three-dimensional sub-model corresponding to the first image as a first sub-model, and determine the three-dimensional sub-model corresponding to the second image as a second sub-model;

[0025] Acquire a first three-dimensional point cloud corresponding to the first sub-model and a second three-dimensional point cloud corresponding to the second sub-model;

[0026] Determine a target conversion relationship between a first sub-model and a second sub-model according to the first three-dimensional point cloud, the second three-dimensional point cloud, the first image, and the second image;

[0027] According to the target conversion relationship, the first sub-model and the second sub-model are fused to obtain a three-dimensional reconstructed model.

[0028] Preferably, determining the target conversion relationship between the first sub-model and the second sub-model according to the first three-dimensional point cloud, the second three-dimensional point cloud, the first image and the second image includes:

[0029] Based on the same physical space point, construct a 3D-2D matching relationship between the first three-dimensional point cloud and the second image, and construct a 3D-2D matching relationship between the second three-dimensional point cloud and the first image;

[0030] According to all the 3D-2D matching relationships, a target conversion relationship corresponding to the first sub-model and the second sub-model is obtained.

[0031] Preferably, constructing a 3D-2D matching relationship between the first three-dimensional point cloud and the second image based on the same physical space point, and constructing a 3D-2D matching relationship between the second three-dimensional point cloud and the first image, includes:

[0032] Extracting a descriptor for a first pixel in the first image to obtain a first descriptor;

[0033] Extracting a descriptor for a second pixel in the second image to obtain a second descriptor;

[0034] Calculating the difference between the first descriptor and the second descriptor to obtain a descriptor difference;

[0035] If the descriptor difference is less than a preset threshold, based on the same physical space point corresponding to the first pixel point and the second pixel point, a 3D-2D matching relationship between the first three-dimensional point cloud and the second image is constructed, and a 3D-2D matching relationship between the second three-dimensional point cloud and the first image is constructed.

[0036] Preferably, the first image includes M first monocular images, and the second image includes M second monocular images;

[0037] The step of extracting a descriptor from a first pixel in the first image to obtain a first descriptor includes:

[0038] Extract descriptors for the first pixel points in the M first monocular images to obtain M first descriptors corresponding to the same physical space point;

[0039] The extracting a descriptor from the second pixel in the second image to obtain a second descriptor includes:

[0040] Extract descriptors for the second pixel points in the M second monocular images to obtain M second descriptors corresponding to the same physical space point;

[0041] The step of calculating the difference between the first descriptor and the second descriptor to obtain the descriptor difference includes:

[0042] Calculate the differences between the M first descriptors and the M second descriptors respectively to obtain M*M descriptor differences;

[0043] If the descriptor difference is less than a preset threshold, constructing a 3D-2D matching relationship between the first three-dimensional point cloud and the second image, and constructing a 3D-2D matching relationship between the second three-dimensional point cloud and the first image, including:

[0044] If any of the M*M descriptor differences is less than a preset threshold, a 3D-2D matching relationship between the first three-dimensional point cloud and the second image is constructed, and a 3D-2D matching relationship between the second three-dimensional point cloud and the first image is constructed.

[0045] Preferably, the step of fusing the first sub-model and the second sub-model according to the target conversion relationship to obtain a three-dimensional reconstructed model includes:

[0046] Obtaining the number of point clouds corresponding to the first three-dimensional point cloud and the number of point clouds corresponding to the second three-dimensional point cloud;

[0047] Determine the sub-model with a smaller number of point clouds as a sub-model to be converted, and determine the sub-model with a larger number of point clouds as a sub-model to be fused;

[0048] According to the target conversion relationship, the three-dimensional space points in the sub-model to be converted are converted to the sub-model to be fused to obtain a three-dimensional reconstructed model.

[0049] An embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned three-dimensional modeling method when executing the computer program.

[0050] An embodiment of the present invention provides a car, comprising N cameras arranged on the car, and also comprising the above-mentioned electronic device, wherein the electronic device is connected to the N cameras.

[0051] Preferably, the N cameras include a front camera arranged at the front side of the car, a left camera arranged at the left side of the car, a right camera arranged at the right side of the car and a rear camera arranged at the rear side of the car.

[0052] Preferably, the front camera is arranged on the front bumper or windshield of the car, the left camera is arranged on the left rearview mirror of the car, the right camera is arranged on the right rearview mirror of the car, and the rear camera is arranged on the rear bumper of the car.

[0053] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned three-dimensional modeling method is implemented.

[0054] In the above-mentioned 3D modeling method, electronic device, system and storage medium, there is a common field of view area between two adjacent original images in N original images, ensuring that the two adjacent original images can be fused; construct a 3D sub-model corresponding to each original image, convert the original image in each original image into three-dimensional information, and then fuse the 3D sub-models corresponding to the two adjacent original images with the common field of view area to obtain a 3D reconstruction model, so that the 3D reconstruction model can fuse the 3D information of the common field of view area between the two adjacent original images, so as to ensure the image quality of the 3D reconstruction model, reduce the computational complexity of its fusion processing, and improve the robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0056] Figure 1 is a flow chart of a three-dimensional modeling method in one embodiment of the present invention;

[0057] Figure 2 is another flow chart of a three-dimensional modeling method in one embodiment of the present invention;

[0058] Figure 3 is another flow chart of a three-dimensional modeling method in one embodiment of the present invention;

[0059] Figure 4 is another flow chart of a three-dimensional modeling method in one embodiment of the present invention;

[0060] Figure 5 is another flow chart of a three-dimensional modeling method in one embodiment of the present invention;

[0061] Figure 6 is another flow chart of a three-dimensional modeling method in one embodiment of the present invention;

[0062] Figure 7is another flow chart of a three-dimensional modeling method in one embodiment of the present invention;

[0063] Figure 8 is another flow chart of a three-dimensional modeling method in one embodiment of the present invention;

[0064] Fig. 9 is another flow chart of a three-dimensional modeling method in one embodiment of the present invention;

[0065] Fig.10 is a schematic diagram of an electronic device in an embodiment of the present invention;

[0066] Fig.11 is a schematic diagram of a car in one embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] The three-dimensional modeling method provided in an embodiment of the present invention can be applied in an electronic device. The electronic device is connected to N cameras and is used to receive original images taken by the N cameras. All original images taken by the N cameras are fused to form a three-dimensional reconstruction model, which can ensure the quality of image processing of the formed three-dimensional reconstruction model.

[0069] In one embodiment, if Figure 1 As shown, a three-dimensional modeling method is provided, and the method is applied to an electronic device as an example for explanation. The method specifically includes the following steps:

[0070] S101: Obtain N original images captured by N cameras respectively, where there is a common field of view area between two adjacent original images, and N≥2;

[0071] S102: Perform three-dimensional reconstruction on each original image to construct a three-dimensional sub-model corresponding to each original image;

[0072] S103: Fusing the three-dimensional sub-models corresponding to two adjacent original images having a common field of view to obtain a three-dimensional reconstructed model.

[0073] Where N is the number of cameras, N ≥ 2. The original image is the original image taken by the camera, that is, the image that has not been processed.

[0074] As an example, in step S101, the electronic device is connected to N cameras, and N original images taken by the N cameras can be obtained. Since there is a common field of view area between two adjacent original images, there is a basis for image fusion between the two adjacent original images.

[0075] Furthermore, the electronic device can obtain N original images formed by N cameras shooting at the same time to ensure that the N original images are formed by N cameras shooting at the same time, so that the N original images have a basis for image fusion.

[0076] As an example, the camera can be a monocular camera, and accordingly, each monocular camera can capture one monocular image at the same time, and the original image captured by the camera includes one monocular image. The camera can also be an M-camera, M ≥ 2, for example, it can be a binocular camera or a trinocular camera; accordingly, each M-camera can capture M monocular images at the same time.

[0077] As an example, in step S102, the electronic device may use a preset 3D reconstruction algorithm to perform 3D reconstruction on the original image captured by each camera, that is, by analyzing and processing the original image, combined with computer vision knowledge, deriving the 3D information of the object in the actual environment, so as to construct a 3D sub-model corresponding to each original image. In this example, the original image captured by each camera can be 3D reconstructed to determine its corresponding 3D sub-model, and the 3D sub-model can reflect the 3D information of the target object captured in the original image.

[0078] For example, the original image collected from camera #1 will be reconstructed in three dimensions on the electronic device to generate the #1 three-dimensional sub-model. The original image collected from camera #2 will also be reconstructed in three dimensions on the electronic device to generate the #2 three-dimensional sub-model; the original image collected from camera #3 will also be reconstructed in three dimensions on the electronic device to generate the #3 three-dimensional sub-model; the original image collected from camera #4 will also be reconstructed in three dimensions on the electronic device to generate the #4 three-dimensional sub-model. At this point, the four three-dimensional sub-models (#1 three-dimensional sub-model, #2 three-dimensional sub-model, #3 three-dimensional sub-model and #4 three-dimensional sub-model) from four cameras (camera #1, camera #2, camera #3 and camera #4) are independent of each other, and the three-dimensional reconstruction method in this step is distributed three-dimensional reconstruction.

[0079] As an example, when the camera is a monocular camera, its original image includes one monocular image, and it is necessary to combine the real-time detection information of other measuring equipment to perform 3D reconstruction on the one monocular image to obtain the 3D sub-model corresponding to the original image. When the camera is an M-camera, its original image includes M monocular images, and the camera parameters of the M-camera can be used to perform 3D reconstruction on the M monocular images to determine the corresponding 3D sub-model, which can reflect the 3D information of the target object captured in the original image.

[0080] As an example, in step S103, after the electronic device constructs the three-dimensional sub-models corresponding to the N original images according to the N original images, it may fuse the three-dimensional sub-models corresponding to the two adjacent original images with a common field of view to obtain a three-dimensional reconstruction model. For example, when three original images are captured by three cameras, there is a common field of view between the first original image and the second original image, and there is a common field of view between the second original image and the third original image, then it is necessary to fuse the three-dimensional sub-model corresponding to the first original image and the three-dimensional sub-model corresponding to the second original image, and to fuse the three-dimensional sub-model corresponding to the second original image and the three-dimensional sub-model corresponding to the third original image, so that the image features of the common field of view between the two adjacent original images are fused, so as to avoid the problem of partial information missing due to gaps in the conventional image stitching process, and to help ensure the image quality of the three-dimensional reconstruction model for image processing. It can be understood that in the process of 3D sub-model fusion, only two adjacent 3D sub-models are fused each time, rather than three or more 3D sub-models. This can reduce the computational complexity of the fusion processing on the one hand, and improve the robustness on the other hand. For example, when a camera fails, the 3D sub-models corresponding to the original images captured by other cameras can still be fused.

[0081] In the three-dimensional modeling method provided in the present embodiment, there is a common field of view area between two adjacent original images among N original images, ensuring that the two adjacent original images can be fused; a three-dimensional sub-model corresponding to each original image is constructed, the original image in each original image is converted into three-dimensional information, and then the three-dimensional sub-models corresponding to the two adjacent original images with the common field of view area are fused to obtain a three-dimensional reconstructed model, which can enable the three-dimensional reconstructed model to fuse the three-dimensional information of the common field of view area between the two adjacent original images, so as to ensure the image quality of the three-dimensional reconstructed model, reduce the computational complexity of its fusion processing, and improve the robustness.

[0082] In one embodiment, the camera is an M-eye camera, and the original image includes M monocular images, where M≥2;

[0083] like Figure 2As shown, step S102, i.e., performing three-dimensional reconstruction on each original image to construct a three-dimensional sub-model corresponding to each original image, includes:

[0084] S201: extracting corner points of the M monocular images respectively to obtain corner points corresponding to the M monocular images;

[0085] S202: Perform feature matching on the corner points corresponding to the M monocular images to obtain corner point matching results;

[0086] S203: Constructing a three-dimensional sub-model corresponding to each original image based on all the corner points that have been matched as a result of the corner point matching.

[0087] Among them, corner points are local structural features in the image, which are usually described as points where the grayscale intensity changes discontinuously and are called singular feature points on the grayscale image.

[0088] As an example, in step S201, the electronic device is connected to N M cameras (M≥2), and each M camera sends M monocular images to the electronic device. After acquiring the M monocular images, the electronic device can extract corner points for each monocular image, that is, using a corner point detection algorithm based on grayscale intensity, a corner point model, and an edge profile to obtain corner points in the monocular image. The corner points are feature points with obvious feature distinction and less extraction calculation. For example, the electronic device can use but is not limited to the SIFT algorithm, the SURF algorithm, the Harris algorithm, and the FAST algorithm to extract corner points to obtain corner points corresponding to the M monocular images.

[0089] As an example, in step S202, after obtaining the corner points corresponding to the M monocular images, the electronic device may use a preset feature matching algorithm to perform feature matching on the corner points corresponding to the M monocular images to obtain a corner point matching result. In this example, the electronic device may use, but is not limited to, an optical flow tracking algorithm based on an image pyramid to perform feature matching on the corner points corresponding to the M monocular images to obtain a corner point matching result, which may reflect whether the corner points corresponding to the M monocular images are the same corner point matching result.

[0090] As an example, in step S203, after obtaining the corner point matching results of the corner points corresponding to the M monocular images, the electronic device can construct a three-dimensional sub-model corresponding to each original image based on all the corner points that have passed the matching results, so that the three-dimensional sub-model contains the three-dimensional information of all corner points that have a matching relationship in the M monocular images, which helps to ensure the image quality of subsequent image processing.

[0091] For example, after the electronic device acquires two monocular images (i.e., the left eye image and the right eye image) collected by the binocular camera, it is necessary to first extract the corner points of the left eye image and the right eye image respectively to obtain the corner points corresponding to the left eye image and the corner points corresponding to the right eye image. Then, the electronic device uses, but is not limited to, an optical flow tracking algorithm based on an image pyramid to match the corner points corresponding to the left eye image and the corner points corresponding to the right eye image one by one, and obtains the corner point matching results corresponding to all corner points. Then, the electronic device constructs a three-dimensional sub-model corresponding to each original image based on all the corner points whose corner point matching results are matched. For example, the corner points A1 and B1 in the left eye image, respectively, have the corner point matching results of the corner points A2 and B2 in the right eye image, and then based on the positional relationship between the corner points A1 and A2, and the corner points B1 and B2, the left eye image and the right eye image are reconstructed in three dimensions to obtain the three-dimensional sub-model corresponding to the original image.

[0092] In one embodiment, if Figure 3 As shown, step S201, i.e., extracting corner points of M monocular images respectively to obtain corner points corresponding to the M monocular images, includes:

[0093] S301: Obtain camera intrinsic parameters and camera extrinsic parameters corresponding to M cameras;

[0094] S302: using camera internal parameters to perform distortion correction processing on M monocular images to obtain M corrected images;

[0095] S302: Using camera extrinsic parameters, aligning the M rectified images to obtain M aligned images;

[0096] S304: Extract corner points of the M aligned images respectively to obtain corner points corresponding to the M monocular images.

[0097] Among them, the camera internal parameters corresponding to the M-eye camera include 6 parameters, 1 / dx, 1 / dy, r, u0, v0 and f, or 4 parameters, fx, fy, u0 and v0, where fx = F*Sx, fy = F*Sy, F is the focal length (i.e., f), Sx and Sy are pixels / mm (i.e., dx and dy, respectively. dx and dy represent how many length units a pixel occupies in the x-direction and y-direction, respectively, that is, the size of the actual physical value represented by a pixel, which is the key to realizing the conversion between the image physical coordinate system and the pixel coordinate system. u0 and v0 represent the horizontal and vertical pixel numbers that differ between the center pixel coordinates of the image and the pixel coordinates of the image origin.

[0098] Among them, the camera extrinsics corresponding to the M-eye camera include 3 rotation parameters and 3 translation parameters. The rotation parameters of the three axes are (ω, δ, θ), respectively. The 3*3 rotation matrices of each axis are combined (i.e., the matrices are multiplied first) to obtain R, which is a collection of the rotation information of the three axes, and its size is still 3*3; the translation parameters of the three axes of T (Tx, Ty, Tz). The 3*4 matrix formed by R and T is the key to converting to the calibration paper coordinates.

[0099] As an example, in step S301, the M cameras include M monocular cameras, and the electronic device can obtain camera intrinsic parameters of each of the M monocular cameras, and obtain camera extrinsic parameters between the M monocular cameras, where the camera intrinsic parameters and the camera extrinsic parameters are pre-calibrated parameters.

[0100] As an example, in step S302, after obtaining the original image corresponding to the M-eye camera, that is, obtaining the M monocular images sent by the same M-eye camera, the electronic device can input the camera intrinsic parameters pre-calibrated by the M-eye camera into the preset distortion correction algorithm for parameter adjustment; then use the distortion correction algorithm after parameter adjustment to perform distortion correction processing on the M monocular images respectively to obtain M corrected images after distortion correction, so as to achieve the purpose of removing image distortion. The corrected image here refers to the image after the monocular image taken by the monocular camera is subjected to distortion correction processing using the camera intrinsic parameters corresponding to the monocular camera.

[0101] As an example, in step S303, after obtaining the M corrected images with the distortion removed, the electronic device may input the camera external parameters pre-calibrated by the M cameras into a preset pixel alignment algorithm for parameter adjustment; then, the pixel alignment algorithm after parameter adjustment is used to perform pixel horizontal alignment processing on the M corrected images, and obtain M aligned images with pixel horizontal alignment, so as to facilitate the subsequent corner point extraction and corner point matching processing, and ensure the accuracy of the corner point matching processing. The aligned image here refers to the image after the corrected image is pixel horizontally aligned.

[0102] As an example, in step S304, after acquiring M aligned images with horizontal pixel alignment, the electronic device may use, but is not limited to, a corner point extraction algorithm such as the SIFT algorithm, the SURF algorithm, the Harris algorithm, and the FAST algorithm to extract corner points from the M aligned images respectively, so as to obtain corner points corresponding to each aligned image, that is, corner points corresponding to each monocular image. It is understandable that since the aligned image is an image after distortion correction and pixel alignment of the monocular image, the extracted corner points are the corner points after distortion correction and pixel alignment, which helps to ensure the accuracy of the subsequent corner point matching process.

[0103] In one embodiment, if Figure 4As shown, step S203, i.e., constructing a 3D sub-model corresponding to each original image based on all corner points matched by the corner point matching result, includes:

[0104] S401: Obtain camera extrinsics corresponding to M cameras;

[0105] S402: According to the camera external parameters, the corner point matching result is all the corner points that have been matched, and the three-dimensional coordinates of the corner points corresponding to the corner points are obtained;

[0106] S403: Construct a three-dimensional sub-model corresponding to each original image according to the three-dimensional coordinates of the corner points corresponding to all the corner points.

[0107] As an example, in step S401, the electronic device needs to obtain camera extrinsic parameters corresponding to M cameras during the three-dimensional reconstruction process, that is, obtain three rotation parameters and three translation parameters corresponding to the M cameras, so as to convert the two-dimensional feature of the corner points corresponding to the M monocular images extracted into a three-dimensional feature according to the camera extrinsic parameters to achieve three-dimensional reconstruction.

[0108] As an example, in step S402, after obtaining the camera extrinsic parameters, the electronic device can input the camera extrinsic parameters into a preset linear triangulation algorithm to adjust the parameters of the linear triangulation algorithm; then, the linear triangulation algorithm with adjusted parameters is used to process all corner points whose corner point matching results are matched, and obtain the three-dimensional coordinates of the corner points corresponding to the corner points, so as to convert the corner points in the two-dimensional space coordinate system into the three-dimensional coordinates of the corner points in the three-dimensional space coordinate system, so as to achieve the purpose of three-dimensional reconstruction.

[0109] As an example, in step S403, after obtaining the three-dimensional coordinates of all corner points, the electronic device may convert the three-dimensional coordinates of all corner points into a unified three-dimensional space coordinate system to obtain a three-dimensional sub-model corresponding to the original image.

[0110] In one embodiment, if Figure 5 As shown, step S103, that is, fusing the 3D sub-models corresponding to the two adjacent original images having the common field of view area to obtain the 3D reconstructed model, includes:

[0111] S501: Determine two adjacent original images having a common field of view as a first image and a second image respectively, determine a 3D sub-model corresponding to the first image as a first sub-model, and determine a 3D sub-model corresponding to the second image as a second sub-model;

[0112] S502: Acquire a first three-dimensional point cloud corresponding to the first sub-model and a second three-dimensional point cloud corresponding to the second sub-model;

[0113] S503: determining a target conversion relationship between the first sub-model and the second sub-model according to the first three-dimensional point cloud, the second three-dimensional point cloud, the first image, and the second image;

[0114] S504: According to the target conversion relationship, the first sub-model and the second sub-model are fused to obtain a three-dimensional reconstructed model.

[0115] As an example, in step S501, when the electronic device needs to perform fusion processing on two adjacent original images with a common field of view, the electronic device may determine the two adjacent original images with a common field of view as the first image and the second image respectively; determine the three-dimensional sub-model corresponding to the first image as the first sub-model, and determine the three-dimensional sub-model corresponding to the second image as the second sub-model. For example, the field of view of camera #1 and camera #2 is 180 degrees, and there is a common field of view between camera #1 and camera #2, then the original image captured by camera #1 and the original image captured by camera #2 have a common field of view, and when the original image captured by camera #1 and the original image captured by camera #2 need to be fused, the original image captured by camera #1 and the original image captured by camera #2 may be determined as the first image and the second image respectively; the three-dimensional sub-model #1 corresponding to the original image captured by camera #1 and the three-dimensional sub-model #2 corresponding to the original image captured by camera #2 may be determined as the first sub-model and the second sub-model respectively.

[0116] A 3D point cloud is a data set of 3D space points arranged in a regular grid. The first 3D point cloud refers to the 3D point cloud in the first sub-model. The second 3D point cloud refers to the 3D point cloud in the second sub-model.

[0117] As an example, in step S502, after determining the first sub-model and the second sub-model, the electronic device needs to respectively extract the first three-dimensional point cloud corresponding to the first sub-model and the second three-dimensional point cloud corresponding to the second sub-model, so as to perform matching conversion processing based on the first three-dimensional point cloud and the second three-dimensional point cloud.

[0118] As an example, in step S503, the electronic device may match the first image, the second image, the first three-dimensional point cloud, and the second three-dimensional point cloud, specifically matching the three-dimensional space points in the first three-dimensional point cloud with the pixel points in the second image, and matching the three-dimensional space points in the second three-dimensional point cloud with the pixel points in the first image, so as to determine the target conversion relationship corresponding to the first sub-model and the second sub-model. The target conversion relationship here refers to the three-dimensional conversion relationship between the points on the first sub-model and the points on the second sub-model. For example, the electronic device may calculate the first conversion relationship f1 of converting the three-dimensional space points on the first sub-model to the three-dimensional space points on the second sub-model, and the second conversion relationship f2 of converting the three-dimensional space points on the second sub-model to the three-dimensional space points on the first sub-model based on the three-dimensional space points of the first sub-model and the three-dimensional space points of the second sub-model corresponding to the same physical space point, and the target conversion relationship includes the first conversion relationship f1 and the second conversion relationship f2.

[0119] As an example, in step S504, when the electronic device obtains the target conversion relationship corresponding to the first sub-model and the second sub-model, it can convert all the three-dimensional space points in the first sub-model to the second sub-model according to the target conversion relationship, or convert all the three-dimensional space points in the second sub-model to the first sub-model, and obtain the three-dimensional reconstructed model, so that the three-dimensional reconstructed model includes image information of all three-dimensional space points in the first sub-model and the second sub-model, so as to avoid the problem of missing part of the information due to gaps in the conventional image stitching process, which helps to ensure the image quality of the three-dimensional reconstructed model.

[0120] In one embodiment, if Figure 6 As shown, step S503, i.e., determining the target conversion relationship between the first sub-model and the second sub-model according to the first three-dimensional point cloud, the second three-dimensional point cloud, the first image and the second image, includes:

[0121] S601: constructing a 3D-2D matching relationship between the first three-dimensional point cloud and the second image based on the same physical space point, and constructing a 3D-2D matching relationship between the second three-dimensional point cloud and the first image;

[0122] S602: According to all 3D-2D matching relationships, obtain a target conversion relationship between the first sub-model and the second sub-model.

[0123] The same physical space point refers to a space point in a real environment photographed by N cameras.

[0124] As an example, in step S601, when the electronic device performs a fusion processing on the first sub-model and the second sub-model, it is necessary to construct a 3D-2D matching relationship corresponding to the first three-dimensional point cloud and the second image based on the same physical space point, and to construct a 3D-2D matching relationship corresponding to the second three-dimensional point cloud and the first image. It can be understood that the 3D-2D matching relationship is used to reflect the matching relationship between the three-dimensional space points in the first sub-model and the second sub-model and the pixel points in the original image taken by different cameras. For example, when the pixel point x11 corresponding to the first image and the pixel point x22 corresponding to the second image point to the same physical space point, a 3D-2D matching relationship is established between the three-dimensional space point X1 in the first sub-model and the pixel point x22 in the second image, and a 3D-2D matching relationship is established between the three-dimensional space point X2 in the second sub-model and the pixel point x11 in the first image.

[0125] As an example, in step S602, after obtaining the 3D-2D matching relationship corresponding to the first sub-model and the second sub-model, the electronic device determines the 3D-2D matching relationship between the first sub-model and the second image, and the 3D-2D matching relationship between the second sub-model and the first image. Through multiple groups of 3D-2D matching relationships, the target conversion relationship between the first sub-model and the second sub-model is determined, that is, the first conversion relationship f1 of converting the three-dimensional space point on the first sub-model to the three-dimensional space point on the second sub-model and the second conversion relationship f2 of converting the three-dimensional space point on the second sub-model to the three-dimensional space point on the first sub-model are determined. The average value of the first conversion relationship f1 and the second conversion relationship f2 can be calculated and determined as the target conversion relationship corresponding to the first sub-model and the second sub-model.

[0126] In one embodiment, if Figure 7 As shown, step S601, i.e., based on the same physical space point, constructing a 3D-2D matching relationship between the first three-dimensional point cloud and the second image, and constructing a 3D-2D matching relationship between the second three-dimensional point cloud and the first image, includes:

[0127] S701: extracting a descriptor from a first pixel in a first image to obtain a first descriptor;

[0128] S702: extracting a descriptor from a second pixel in the second image to obtain a second descriptor;

[0129] S703: Calculate the difference between the first descriptor and the second descriptor to obtain a descriptor difference;

[0130] S704: If the descriptor difference is less than a preset threshold, based on the same physical space point corresponding to the first pixel point and the second pixel point, a 3D-2D matching relationship corresponding to the first three-dimensional point cloud and the second image is constructed, and a 3D-2D matching relationship corresponding to the second three-dimensional point cloud and the first image is constructed.

[0131] The first pixel point refers to a pixel point in the first image, and the second pixel point refers to a pixel point in the second image.

[0132] As an example, in step S701, the electronic device may use but is not limited to the BRIEF algorithm to extract the descriptor of the first pixel in the first image and obtain the binary-coded first descriptor, which helps to improve the matching speed in subsequent image matching. In this example, the electronic device may use but is not limited to the grayscale intensity-based, corner model-based, and edge contour-based corner detection algorithms to extract the descriptor of the first pixel in the first image and obtain the first descriptor.

[0133] As an example, in step S702, the electronic device may use but is not limited to the BRIEF algorithm to extract the descriptor of the second pixel in the second image and obtain the binary-coded second descriptor, which helps to improve the matching speed in subsequent image matching. In this example, the electronic device may use but is not limited to the grayscale intensity-based, corner model-based, and edge contour-based corner detection algorithms to extract the descriptor of the second pixel in the second image and obtain the second descriptor.

[0134] As an example, in step S703, after obtaining the first descriptor corresponding to the first image and the second descriptor corresponding to the second image, the electronic device may calculate and compare the difference between the first descriptor and the second descriptor to obtain a descriptor difference. The descriptor difference may be used to reflect the feature difference between the first pixel and the second pixel.

[0135] The preset threshold is a preset threshold used to evaluate whether a small difference is reached.

[0136] As an example, in step S704, after obtaining the descriptor difference, the electronic device may compare the descriptor difference with a preset threshold. If the descriptor difference is less than the preset threshold, it is determined that the difference between the first pixel and the second pixel is small, and the probability that the two point to the same physical space point is high. At this time, based on the same physical space point corresponding to the first pixel and the second pixel, a 3D-2D matching relationship corresponding to the first three-dimensional point cloud and the second image is constructed, and a 3D-2D matching relationship corresponding to the second three-dimensional point cloud and the first image is constructed. In this example, a 3D-2D matching relationship can be constructed between the three-dimensional space point corresponding to the first pixel in the first three-dimensional point cloud and the second pixel in the second image, and a 3D-2D matching relationship can be constructed between the three-dimensional space point corresponding to the second pixel in the second three-dimensional point cloud and the first pixel in the first image.

[0137] In one embodiment, the first image includes M first monocular images, and the second image includes M second monocular images;

[0138] like Figure 8 Step S503, i.e., based on the same physical space point, constructing a 3D-2D matching relationship between the first three-dimensional point cloud and the second image, and constructing a 3D-2D matching relationship between the second three-dimensional point cloud and the first image, includes:

[0139] S801: extracting descriptors from first pixel points in M ​​first monocular images to obtain M first descriptors corresponding to the same physical space point;

[0140] S802: extracting descriptors from second pixel points in the M second monocular images to obtain M second descriptors corresponding to the same physical space point;

[0141] S803: Calculate differences between the M first descriptors and the M second descriptors respectively to obtain M*M descriptor differences;

[0142] S804: If any of the M*M descriptor differences is less than a preset threshold, a 3D-2D matching relationship between the first three-dimensional point cloud and the second image is constructed, and a 3D-2D matching relationship between the second three-dimensional point cloud and the first image is constructed.

[0143] As an example, step S801 is a specific implementation of step S701, step S802 is a specific implementation of step S702, step S803 is a specific implementation of step S703, and step S804 is a specific implementation of step S704.

[0144] The first monocular image refers to an image captured by each monocular camera in the M cameras corresponding to the first image.

[0145] As an example, in step S801, the electronic device may use, but is not limited to, the BRIEF algorithm to perform descriptor extraction on the first pixel points in the M first monocular images to determine the M first descriptors corresponding to the same physical space point. For example, when the camera is a binocular camera, the first image includes a left-eye image and a right-eye image, and the first pixel points in the left-eye image and the right-eye image of the same physical space point (i.e., the same three-dimensional space point X1) in the first sub-model are x11 and x12, respectively, then the electronic device needs to perform descriptor extraction on the first pixel points x11 and x12 in the left-eye image and the right-eye image to obtain the two first descriptors corresponding to the same physical space point.

[0146] As an example, in step S802, the electronic device may use, but is not limited to, the BRIEF algorithm to perform descriptor extraction on the second pixel points in the M second monocular images to determine the M second descriptors corresponding to the same physical space point. For example, when the camera is a binocular camera, the second image includes a left-eye image and a right-eye image, and the second pixel points in the left-eye image and the right-eye image at the same physical space point (i.e., the same three-dimensional space point X2) in the second sub-model are x21 and x22, respectively. Then, the electronic device needs to perform descriptor extraction on the second pixel points x21 and x22 in the left-eye image and the right-eye image to obtain two second descriptors corresponding to the same physical space point.

[0147] As an example, in step S803, after obtaining the M first descriptors corresponding to the same pixel and the M second descriptors corresponding to the same pixel, the electronic device may perform difference calculations on the M first descriptors and the M second descriptors respectively to determine M*M descriptor differences. For example, the descriptor difference between the first pixel x11 and the second pixel x21, the descriptor difference between the first pixel x11 and the second pixel x22, the descriptor difference between the first pixel x12 and the second pixel x21, and the descriptor difference between the first pixel x12 and the second pixel x22 are compared.

[0148] As an example, in step S804, after obtaining M*M descriptor differences, the electronic device may compare the M*M descriptor differences with the preset thresholds respectively; if at least one of the M*M descriptor differences is less than the preset threshold, it is determined that the difference between the first pixel and the second pixel is small, and the probability that the two point to the same physical space point is high. At this time, based on the first pixel and the second pixel, a 3D-2D matching relationship corresponding to the first three-dimensional point cloud and the second image can be constructed, and a 3D-2D matching relationship corresponding to the second three-dimensional point cloud and the first image can be constructed. In this example, the electronic device can construct a 3D-2D matching relationship between the three-dimensional space point corresponding to the first pixel in the first three-dimensional point cloud and the second pixel in the second image, and construct a 3D-2D matching relationship between the three-dimensional space point corresponding to the second pixel in the second three-dimensional point cloud and the first pixel in the first image.

[0149] Furthermore, the electronic device can construct a 3D-2D matching relationship corresponding to the first three-dimensional point cloud and the second image based on the first pixel point and the second pixel point corresponding to the minimum descriptor difference, and construct a 3D-2D matching relationship corresponding to the second three-dimensional point cloud and the first image to ensure the accuracy of the 3D-2D matching relationship.

[0150] In one embodiment, if Fig. 9 As shown, step S504, that is, fusing the first sub-model and the second sub-model according to the target conversion relationship to obtain a three-dimensional reconstructed model, includes:

[0151] S901: Obtaining the number of point clouds corresponding to the first three-dimensional point cloud and the number of point clouds corresponding to the second three-dimensional point cloud;

[0152] S902: Determine the sub-model with a smaller number of point clouds as a sub-model to be converted, and determine the sub-model with a larger number of point clouds as a sub-model to be fused;

[0153] S903: According to the target conversion relationship, the three-dimensional space points in the sub-model to be converted are converted to the sub-model to be fused to obtain a three-dimensional reconstructed model.

[0154] The sub-model to be converted refers to the sub-model that needs to be converted into three-dimensional space points. The sub-model to be fused refers to the sub-model that needs to receive image information after the three-dimensional space point conversion.

[0155] As an example, in step 901, when the electronic device is fusing the first sub-model and the second sub-model, it is necessary to first count the number of point clouds corresponding to the first three-dimensional point cloud in the first sub-model, and count the number of point clouds corresponding to the second three-dimensional point cloud in the second sub-model, so as to determine the sub-model to be converted and the sub-model to be fused based on the number of point clouds corresponding to the first three-dimensional point cloud and the number of point clouds corresponding to the second three-dimensional point cloud.

[0156] As an example, in step S902, after obtaining the number of point clouds corresponding to the first three-dimensional point cloud and the number of point clouds corresponding to the second three-dimensional point cloud, the electronic device can compare the number of point clouds of the two, determine the sub-model with fewer point clouds as the sub-model to be converted, and determine the sub-model with more point clouds as the sub-model to be fused.

[0157] As an example, in step S903, after determining the sub-model to be converted and the sub-model to be fused, the 3D space points in the sub-model to be converted are converted to the sub-model to be fused according to the target conversion relationship to obtain a 3D reconstructed model. It can be understood that since the sub-model to be converted is a sub-model with a small number of point clouds, the processing operations of converting the 3D space points in the sub-model to be converted to the sub-model to be fused can be reduced, which helps to improve processing efficiency.

[0158] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0159] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the three-dimensional modeling method in the above embodiment is implemented, for example Figure 1 S101-S103 as shown, or Figures 2 to 9 As shown in , the 3D reconstructed model finally obtained can fuse the 3D information of the common field of view area between two adjacent original images to ensure the image quality of the 3D reconstructed model, reduce the computational complexity of its fusion processing, and improve the robustness. To avoid repetition, it will not be repeated here.

[0160] like Fig.10 As shown, the electronic device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data used or generated in the process of executing the three-dimensional modeling method. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a three-dimensional modeling method is implemented.

[0161] An embodiment of the present invention provides a car, the car includes N cameras, and also includes the above electronic device, the electronic device is connected to the N cameras. Fig.11As shown, the N cameras include a 1# camera arranged at the front side of the car, a 2# camera arranged at the left side of the car, a 3# camera arranged at the right side of the car, and a 4# camera arranged at the rear side of the car.

[0162] As an example, the N cameras installed on the car may be monocular cameras, binocular cameras or other M-phase cameras.

[0163] As an example, N cameras are set on a car, and the field of view of each camera is 180 degrees. At this time, N≥3, so that the N cameras can take N original images corresponding to the external environment of the car body, and there is a common field of view area between the original images taken by two adjacent cameras. The electronic device is connected to the N cameras and can fuse the N original images collected by the N cameras to obtain a three-dimensional reconstruction model. The final three-dimensional reconstruction model can fuse the three-dimensional information of the common field of view area between two adjacent original images to ensure the image quality of the three-dimensional reconstruction model, reduce the computational complexity of its fusion processing, and improve the robustness.

[0164] Preferably, the N cameras include a front camera arranged at the front side of the car, a left camera arranged at the left side of the car, a right camera arranged at the right side of the car, and a rear camera arranged at the rear side of the car.

[0165] As an example, a front camera, a left camera, a right camera and a rear camera are respectively arranged on the front, left side, right side and rear side of the car. The front camera, the left camera, the right camera and the rear camera send the original images they collect to the electronic device, so that the electronic device can perform three-dimensional reconstruction based on the four original images obtained, and can completely reconstruct the three-dimensional environment outside the car body and provide more complete 3D information of the car body environment.

[0166] In this example, the electronic device can adopt a 3D reconstruction method that combines distributed and sequential methods, that is, the original images collected by M cameras are first independently 3D reconstructed (distributed 3D reconstruction) to obtain N 3D sub-models; then, the separately established 3D sub-models are fused in a sequential manner. The beneficial effect of this method is that when fusion of 3D sub-models is performed, since two 3D sub-models are fused each time instead of three or more, the amount of image fusion calculation can be reduced; and in the process of fusion of 3D sub-models, the robustness can be improved. For example, if a camera fails, other cameras can still be fused to build a map.

[0167] For example, the electronic device performs 3D reconstruction on the original image captured by the front camera to generate a front 3D sub-model corresponding to the front camera; the electronic device performs 3D reconstruction on the original image captured by the left camera to generate a left 3D sub-model corresponding to the left camera; the electronic device performs 3D reconstruction on the original image captured by the right camera to generate a right 3D sub-model corresponding to the right camera; the electronic device performs 3D reconstruction on the original image captured by the rear camera to generate a rear 3D sub-model corresponding to the rear camera. At this point, the four 3D sub-models (front 3D sub-model, left 3D sub-model, right 3D sub-model and rear 3D sub-model) from the four cameras (front camera, left camera, right camera and rear camera) are independent of each other, and the 3D reconstruction method of this step is distributed 3D reconstruction.

[0168] For another example, since the field of view of each camera is 180 degrees, the front camera has a common field of view with the left camera and the right camera; the rear camera has a common field of view with the left camera and the right camera; but the front camera and the rear camera have no common field of view, and the left camera and the right camera have no common field of view. In addition, the fusion of 3D sub-models can start from any two cameras with a common field of view, for example, the fusion of the front camera and the left camera, the fusion of the front camera and the right camera, the fusion of the rear camera and the left camera, or the fusion of the rear camera and the right camera. The following takes the fusion of the front camera and the left camera as an example to explain the fusion process in detail: Since the front camera and the left camera have a common field of view, the front 3D sub-model from the front camera and the left 3D sub-model from the left camera are first fused to obtain a first fused model; Since the front camera and the right camera also have a common field of view, the first fused model can be fused with the right 3D sub-model from the right camera to obtain a second fused model; the first fused model contains relevant 3D reconstruction information from the front camera, the left camera and the right camera; Finally, the first fused model is fused with the rear 3D sub-model from the rear camera to obtain a 3D reconstruction model. This process is a sequential 3D fusion process.

[0169] Preferably, the front camera is arranged on the front bumper or windshield of the car, the left camera is arranged on the left rearview mirror of the car, the right camera is arranged on the right rearview mirror of the car, and the rear camera is arranged on the rear bumper of the car. Understandably, the front camera arranged on the front bumper or windshield of the car, the left camera arranged on the left rearview mirror of the car, the right camera arranged on the right rearview mirror of the car, and the rear camera arranged on the rear bumper of the car are connected to the electronic device installed on the chassis by using the communication network, so that the electronic device can perform three-dimensional reconstruction and image fusion according to the original images collected by the four cameras to obtain a three-dimensional reconstruction model, and the three-dimensional reconstruction model can completely reconstruct the three-dimensional environment outside the car body and provide more complete 3D information of the car body environment.

[0170] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the three-dimensional modeling method in the above embodiment is implemented, for example Figure 1 S101-S103 as shown, or Figures 2 to 9 As shown in , the 3D reconstructed model finally obtained can fuse the 3D information of the common field of view area between two adjacent original images to ensure the image quality of the 3D reconstructed model, reduce the computational complexity of its fusion processing, and improve the robustness. To avoid repetition, it will not be repeated here.

[0171] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0172] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0173] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A three-dimensional modeling method, characterized in that: include: Obtain N original images captured by N cameras at the same time, where there is a common field of view area between two adjacent original images, and N≥2; Performing three-dimensional reconstruction on each of the original images, and constructing a three-dimensional sub-model corresponding to each of the original images, including: when the camera is a monocular camera, the original image includes one monocular image, and in combination with real-time detection information of other measuring equipment, the one monocular image is three-dimensionally reconstructed to obtain a three-dimensional sub-model corresponding to the one monocular image; when the camera is an M-camera, the original image includes M monocular images, and the M monocular images are three-dimensionally reconstructed using the camera parameters of the M-camera itself, and the three-dimensional sub-models corresponding to the M monocular images are determined, wherein M≥2; Determine two adjacent original images having a common field of view as a first image and a second image respectively, determine the three-dimensional sub-model corresponding to the first image as a first sub-model, and determine the three-dimensional sub-model corresponding to the second image as a second sub-model; Acquire a first three-dimensional point cloud corresponding to the first sub-model and a second three-dimensional point cloud corresponding to the second sub-model; Based on the same physical space point, construct a 3D-2D matching relationship between the first three-dimensional point cloud and the second image, and construct a 3D-2D matching relationship between the second three-dimensional point cloud and the first image; According to all the 3D-2D matching relationships, obtaining a target conversion relationship corresponding to the first sub-model and the second sub-model, including: determining a first conversion relationship for converting a three-dimensional space point on the first sub-model to a three-dimensional space point on the second sub-model, and a second conversion relationship for converting a three-dimensional space point on the second sub-model to a three-dimensional space point on the first sub-model, and determining an average value of the first conversion relationship and the second conversion relationship as the target conversion relationship corresponding to the first sub-model and the second sub-model; According to the target conversion relationship, the first sub-model and the second sub-model are fused to obtain a three-dimensional reconstructed model.

2. The three-dimensional modeling method according to claim 1, characterized in that: The camera is an M-eye camera, and the original image includes M monocular images, where M≥2; The three-dimensional reconstruction of each of the original images to construct a three-dimensional sub-model corresponding to each of the original images includes: Extracting corner points of the M monocular images respectively to obtain corner points corresponding to the M monocular images; Perform feature matching on the corner points corresponding to the M monocular images to obtain corner point matching results; Based on the corner point matching result being all the corner points that have been matched, a three-dimensional sub-model corresponding to each of the original images is constructed.

3. The three-dimensional modeling method according to claim 2, characterized in that: The step of respectively extracting corner points of the M monocular images to obtain corner points corresponding to the M monocular images includes: Get the camera intrinsic parameters and camera extrinsic parameters corresponding to the M cameras; Using the camera intrinsic parameters, perform distortion correction processing on the M monocular images to obtain M corrected images; Using the camera extrinsic parameters, aligning the M rectified images to obtain M aligned images; Corner points are extracted from the M aligned images respectively to obtain corner points corresponding to the M monocular images.

4. The three-dimensional modeling method according to claim 2, characterized in that: The step of constructing a three-dimensional sub-model corresponding to each of the original images based on all the corner points matched by the corner point matching result includes: Get the camera extrinsics corresponding to the M cameras; According to the camera external parameters, for all the corner points that the corner point matching results pass, obtain the three-dimensional coordinates of the corner points corresponding to the corner points; A three-dimensional sub-model corresponding to each of the original images is constructed according to the three-dimensional coordinates of the corner points corresponding to all the corner points.

5. The three-dimensional modeling method according to claim 1, characterized in that: The step of constructing a 3D-2D matching relationship between the first three-dimensional point cloud and the second image based on the same physical space point, and constructing a 3D-2D matching relationship between the second three-dimensional point cloud and the first image, includes: Extracting a descriptor for a first pixel in the first image to obtain a first descriptor; Extracting a descriptor for a second pixel in the second image to obtain a second descriptor; Calculating the difference between the first descriptor and the second descriptor to obtain a descriptor difference; If the descriptor difference is less than a preset threshold, based on the same physical space point corresponding to the first pixel point and the second pixel point, a 3D-2D matching relationship between the first three-dimensional point cloud and the second image is constructed, and a 3D-2D matching relationship between the second three-dimensional point cloud and the first image is constructed.

6. The three-dimensional modeling method according to claim 5, characterized in that: The first image includes M first monocular images, and the second image includes M second monocular images; The step of extracting a descriptor from a first pixel in the first image to obtain a first descriptor includes: Extract descriptors for the first pixel points in the M first monocular images to obtain M first descriptors corresponding to the same physical space point; The extracting a descriptor from the second pixel in the second image to obtain a second descriptor includes: Extract descriptors for the second pixel points in the M second monocular images to obtain M second descriptors corresponding to the same physical space point; The step of calculating the difference between the first descriptor and the second descriptor to obtain the descriptor difference includes: Calculate the differences between the M first descriptors and the M second descriptors respectively to obtain M*M descriptor differences; If the descriptor difference is less than a preset threshold, constructing a 3D-2D matching relationship between the first three-dimensional point cloud and the second image, and constructing a 3D-2D matching relationship between the second three-dimensional point cloud and the first image, including: If any of the M*M descriptor differences is less than a preset threshold, a 3D-2D matching relationship between the first three-dimensional point cloud and the second image is constructed, and a 3D-2D matching relationship between the second three-dimensional point cloud and the first image is constructed.

7. The three-dimensional modeling method according to claim 1, characterized in that: The step of fusing the first sub-model and the second sub-model according to the target conversion relationship to obtain a three-dimensional reconstructed model includes: Obtaining the number of point clouds corresponding to the first three-dimensional point cloud and the number of point clouds corresponding to the second three-dimensional point cloud; Determine the sub-model with a smaller number of point clouds as a sub-model to be converted, and determine the sub-model with a larger number of point clouds as a sub-model to be fused; According to the target conversion relationship, the three-dimensional space points in the sub-model to be converted are converted to the sub-model to be fused to obtain a three-dimensional reconstructed model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the three-dimensional modeling method according to any one of claims 1 to 7 is implemented.

9. A car, comprising N cameras, characterized in that: Also included is the electronic device of claim 8, wherein the electronic device is connected to N of the cameras.

10. The automobile according to claim 9, characterized in that The N cameras include a front camera arranged at the front side of the car, a left camera arranged at the left side of the car, a right camera arranged at the right side of the car, and a rear camera arranged at the rear side of the car.

11. The automobile according to claim 9, characterized in that The front camera is arranged on the front bumper or windshield of the car, the left camera is arranged on the left rearview mirror of the car, the right camera is arranged on the right rearview mirror of the car, and the rear camera is arranged on the rear bumper of the car.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the three-dimensional modeling method according to any one of claims 1 to 7 is implemented.

Images