Three-dimensional reconstruction method and system based on deep learning
Through sparse reconstruction and inverse depth sampling strategies, the problem of poor depth map estimation in near- and far-distance scenarios is solved, reducing image matching redundancy and improving the effect of three-dimensional reconstruction.
Patent Information
- Application Number
- CN202410969168.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-12
AI Technical Summary
The three-dimensional reconstruction method based on deep learning is poor in depth map estimation in close and long-distance scenarios, and the image matching pairs are redundant due to randomly selected image matching pairs, which affects the reconstruction efficiency and quality.
By obtaining the sparse reconstruction method, the maximum depth value and minimum depth value of the 3D sparse point corresponding to the source image in the camera coordinate system is determined based on these values, and the target depth sampling interval is calculated, and the inverse depth sampling strategy is generated. Depth map, and then generate a mesh model.
The effect of three-dimensional reconstruction is improved, especially in the near- and long-distance scenes of depth estimation, reducing the redundancy of image matching pairs, and improving reconstruction efficiency and quality.
Smart Images

Figure CN118941713A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of three-dimensional reconstruction, and in particular to a three-dimensional reconstruction method, system, electronic device and storage medium based on deep learning. Background Art
[0002] At present, for three-dimensional dense reconstruction based on deep learning, the Deep Learning Multi-ViewStereo (DL-MVS) algorithm is usually used to replace the traditional stereo matching algorithm and stereo fusion algorithm to output the depth map of the scene.
[0003] In the related art, the depth map effect of the DL-MVS algorithm is poorly estimated in extremely close or relatively far scenes, and the DL-MVS algorithm selects image matching pairs by random selection, resulting in most of the selected image matching pairs being redundant.
[0004] Currently, no effective solution has been proposed to address the problem of poor three-dimensional reconstruction results based on deep learning in related technologies. Summary of the invention
[0005] The embodiments of the present application provide a deep learning-based three-dimensional reconstruction method, system, electronic device, and storage medium to at least solve the problem of poor three-dimensional reconstruction effect based on deep learning in related technologies.
[0006] In a first aspect, an embodiment of the present application provides a three-dimensional reconstruction method based on deep learning, the method comprising:
[0007] Acquire a target object image, and perform sparse reconstruction based on the target object image to obtain a sparse model of the target object, wherein the target object image includes an image pair consisting of a reference image and a source image, and the sparse model includes 3D sparse points;
[0008] Determine a maximum depth value and a minimum depth value of the 3D sparse points corresponding to the source image in the camera coordinate system, and determine a target depth sampling interval of the 3D sparse points on the image epipolar lines corresponding to each of the source images based on the maximum depth value and the minimum depth value;
[0009] Based on the inverse depth sampling strategy, a depth map corresponding to each of the reference images is obtained according to the target depth sampling interval, and a mesh model of the target object is generated according to the depth map.
[0010] In some embodiments, determining the target depth sampling interval of the 3D sparse points on the image epipolar line corresponding to each of the source images based on the maximum depth value and the minimum depth value comprises:
[0011] Acquire the internal parameter data of the target object image from the sparse model, and obtain the optical center coordinates of the source image in a pixel coordinate system according to the internal parameter data;
[0012] Based on a preset sampling number acquisition model, the depth sampling number of the 3D sparse point is obtained according to the optical center coordinates in the pixel coordinate system, the maximum depth and the minimum depth;
[0013] The target depth sampling interval is determined based on the number of depth samples, the maximum depth, and the minimum depth.
[0014] In some embodiments, the preset sampling number acquisition model includes:
[0015]
[0016] Among them, d num is the number of depth samples, d max is the maximum depth value, d min is the minimum depth value; P 1 is the optical center coordinate in the world coordinate system, (cx, cy) is the optical center coordinate in the pixel coordinate system; P 2 is the corresponding coordinate of the point (cx+1, cy) in the pixel coordinate system in the world coordinate system, L 1 norm(P 2 -P 1 ) is P 2 To P 1 L 1 norm, T and K are coordinate transformation parameters.
[0017] In some embodiments, determining the target depth sampling interval based on the number of depth samples, the maximum depth, and the minimum depth includes:
[0018] Based on a sampling interval generation model, determining the target depth sampling interval according to the maximum depth value, the minimum depth value and the number of depth samples;
[0019] The sampling interval generation model includes:
[0020]
[0021] Among them, d interval is the target depth sampling interval, d max is the maximum depth value, d min is the minimum depth value, d num is the number of depth samples.
[0022] In some embodiments, after acquiring the target object image, the method further includes:
[0023] Determine a reference image from the target object image, and obtain an angle between the reference image and a line connecting a common viewpoint to an optical center of the image to be matched, wherein the image to be matched is an image in the target object image other than the reference image;
[0024] Obtaining a common view coefficient between the reference image and the image to be matched according to the probability density of the angles corresponding to all the common view points;
[0025] Based on the common view coefficient, a source image corresponding to the reference image is determined from all the images to be matched.
[0026] In some embodiments, determining the maximum depth value and the minimum depth value of the 3D sparse point corresponding to the source image in the camera coordinate system includes:
[0027] Obtaining, from the sparse model, camera extrinsic parameters and the first coordinates of each of the 3D sparse points in the world coordinate system;
[0028] Convert the first coordinate into a second coordinate in a camera coordinate system according to the camera extrinsic parameter;
[0029] Based on the second coordinate, the depth values of the 3D sparse points are sorted to obtain a maximum depth value and a minimum depth value.
[0030] In some embodiments, the step of obtaining a depth map corresponding to each of the reference images according to the target depth sampling interval based on an inverse depth sampling strategy, and generating a mesh model of the target object according to the depth map includes:
[0031] Based on an inverse depth sampling strategy, generating a depth map of each of the reference images according to the source image and the target depth sampling interval;
[0032] Performing depth fusion according to the depth map to obtain point cloud data of the target object;
[0033] Based on the point cloud data, a mesh model of the target object is generated.
[0034] In a second aspect, an embodiment of the present application provides a three-dimensional reconstruction system based on deep learning, the system comprising: an acquisition module, an analysis module and a model generation module, wherein:
[0035] The acquisition module is used to acquire a target object image, and perform sparse reconstruction based on the target object image to obtain a sparse model of the target object, wherein the target object image includes an image pair consisting of a reference image and a source image, and the sparse model includes 3D sparse points;
[0036] The analysis module is used to determine the maximum depth value and the minimum depth value of the 3D sparse points corresponding to the source image in the camera coordinate system, and determine the target depth sampling interval of the 3D sparse points on the image epipolar line corresponding to each of the source images based on the maximum depth value and the minimum depth value;
[0037] The model generation module is used to obtain a depth map corresponding to each of the reference images according to the target depth sampling interval based on an inverse depth sampling strategy, and generate a mesh model of the target object according to the depth map.
[0038] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the three-dimensional reconstruction method based on deep learning as described in the first aspect above is implemented.
[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the deep learning-based three-dimensional reconstruction method as described in the first aspect above.
[0040] Compared with the related art, the three-dimensional reconstruction method based on deep learning provided in the embodiment of the present application acquires the target object image, performs sparse reconstruction based on the target object image, obtains the sparse model of the target object, the target object image includes an image pair consisting of a reference image and a source image, the sparse model includes 3D sparse points, determines the maximum depth value and the minimum depth value of the 3D sparse points corresponding to the source image in the camera coordinate system, determines the target depth sampling interval of the 3D sparse points on the image pole line corresponding to each source image based on the maximum depth value and the minimum depth value, obtains the depth map corresponding to each reference image based on the target depth sampling interval based on the inverse depth sampling strategy, generates the mesh model of the target object based on the depth map, and solves the problem of poor three-dimensional reconstruction effect based on deep learning. Based on the inverse depth sampling strategy, it ensures that the pixel points on the image pole line are uniform, alleviates the problem of inaccurate depth estimation caused by near depth of field and deep depth of field, and thus improves the effect of three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0042] Figure 1 is a flowchart of a deep learning-based 3D reconstruction method according to an embodiment of the present application;
[0043] Figure 2 is a schematic diagram of depth sampling according to an embodiment of the present application;
[0044] Figure 3 is another schematic diagram of depth sampling according to an embodiment of the present application;
[0045] Figure 4 is a structural block diagram of a deep learning-based 3D reconstruction system according to an embodiment of the present application;
[0046] Figure 5 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0048] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as the content disclosed in this application is insufficient.
[0049] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0050] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.
[0051] This embodiment provides a three-dimensional reconstruction method based on deep learning. Figure 1 is a flowchart of a 3D reconstruction method based on deep learning according to an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps:
[0052] Step S101 , collecting a target object image, and performing sparse reconstruction based on the target object image to obtain a sparse model of the target object, wherein the target object image includes an image pair consisting of a reference image and a source image, and the sparse model includes 3D sparse points.
[0053] Capture images of a target object that needs to be three-dimensionally reconstructed, and perform sparse reconstruction of the target object through a three-dimensional reconstruction model to obtain a sparse model, wherein the sparse model includes internal and external parameters of the target object image, corresponding 3D sparse points and 2D sparse points, wherein the three-dimensional reconstruction model includes but is not limited to a colmap model or an openmvs model.
[0054] In some embodiments, after acquiring the target object image in step S101, the method further includes:
[0055] Step S1011, determining a reference image from the target object image, and obtaining the angle between the reference image and the common viewpoint of the image to be matched and the optical center, wherein the image to be matched is the image in the target object image other than the reference image.
[0056] Step S1012: obtaining the common view coefficient between the reference image and the image to be matched according to the probability density of the angles corresponding to all common view points.
[0057] Step S1013: determining a source image corresponding to the reference image from all images to be matched based on the common view coefficient.
[0058] In the process of acquiring depth information, the input to the deep learning model is a reference image and a matching pair of source images (reference-source image). Optionally, each target object image is used as a reference image in turn, and the source image corresponding to the current reference image is determined from the remaining target object images.
[0059] Usually, it is necessary to find 10 or even 20 matching source images (source view) for each reference image (reference view) to ensure the quality of the depth map, resulting in a large number of redundant source images in the entire process.
[0060] In this embodiment, the depth probability density corresponding to the image is calculated to select a source image with a higher co-viewing relationship, so as to avoid the increase in calculation amount and the reduction in model reconstruction efficiency due to the redundancy of the source image.
[0061] The common view relationship is determined by the angle between the common viewpoint of the two images and the optical center. The calculation formula of the angle is as follows:
[0062]
[0063] Among them, θ is the angle formed by the common viewpoint of the reference image and the current image to be matched and the line connecting their corresponding optical centers, C ref is the coordinate of the optical center of the reference image camera in the world coordinate system, C src is the coordinate of the optical center of the source image camera in the world coordinate system, P intersect is the 3D point coordinate in the world coordinate system, L 1 norm() is the L of two points 1 Norm.
[0064] The probability density calculation formula for the angles corresponding to all common viewpoints of the reference image and the image to be matched is as follows:
[0065]
[0066] Where score is the probability density, n is the number of common viewpoints corresponding to the reference image and the image to be matched, i = 1, 2, ..., n; θ i is the angle corresponding to the i-th common viewpoint; θ 0and σ are preset parameters. For example, 0 =5, indicating that the standard angle between the common viewpoint and the line connecting the two optical centers is 5 degrees, that is, the difference between the two viewing angles is within 5 degrees. σ can be selected between 1 and 3, indicating that the viewing angle between 2° and 8° is in line with deep learning training.
[0067] If the probability density score corresponding to the angle between all the common view 3D points of two images is higher, the common view relationship between the two images is stronger, which is more beneficial to deep learning. By calculating the probability density of the common view angle, the image to be matched with a high common view relationship with the reference image is used as the source image, and the source view selection range corresponding to each reference image can be narrowed down, for example, to 5 or even 3. This avoids the increase in calculation amount caused by the redundancy of source images, thereby improving the efficiency of model reconstruction.
[0068] Continue to refer to Figure 1 After obtaining the sparse model of the target object, execute step S102.
[0069] Step S102, determining the maximum depth value and the minimum depth value of the 3D sparse points corresponding to the source image in the camera coordinate system, and determining the target depth sampling interval of the 3D sparse points on the image epipolar line corresponding to each source image based on the maximum depth value and the minimum depth value.
[0070] In some embodiments, determining the maximum depth value and the minimum depth value of the 3D sparse point corresponding to the source image in the camera coordinate system in step S102 includes:
[0071] Step S1021, obtaining the camera extrinsic parameters and the first coordinates of each 3D sparse point in the world coordinate system from the sparse model.
[0072] Step S1022: convert the first coordinate into a second coordinate in the camera coordinate system according to the camera external parameter.
[0073] According to the camera extrinsic parameters, the 3D sparse point coordinates in the world coordinate system output in the sparse model are converted to the camera coordinate system. The coordinate conversion formula is as follows:
[0074]
[0075] Among them, P c Represents the 3D sparse point coordinates in the camera coordinate system, represents the camera extrinsic parameter corresponding to the 3D sparse point, P w P is the coordinate of the 3D sparse point in the world coordinate system output by the sparse model. c The coordinates are (X, Y, Z), where Z is the depth value.
[0076] Step S1023: based on the second coordinate, the depth values of the 3D sparse points are sorted to obtain a maximum depth value and a minimum depth value.
[0077] The formula for obtaining the maximum depth value and the minimum depth value is as follows:
[0078] dmin,dmax=sort(P c1 , P c2 ...P cn )
[0079] Among them, dmin is the minimum depth value, dmax is the maximum depth value, P c1 , P c2 , ..., P cn represents the 3D sparse points in all camera coordinate systems. In this embodiment, sort() represents the maximum and minimum value functions.
[0080] In some embodiments, determining the target depth sampling interval of the 3D sparse points on the image epipolar line corresponding to each source image based on the maximum depth value and the minimum depth value in step S102 includes:
[0081] Step S1024, obtaining the intrinsic parameter data of the source image from the sparse model, and obtaining the optical center coordinates of the source image in the pixel coordinate system according to the intrinsic parameter data.
[0082] Step S1025, acquiring a model based on a preset number of samplings, and obtaining the number of depth samplings of the 3D sparse points according to the optical center coordinates, the maximum depth, and the minimum depth in the pixel coordinate system.
[0083] Step S1026, determining a target depth sampling interval based on the number of depth samples, the maximum depth, and the minimum depth.
[0084] Figure 2 is a schematic diagram of depth sampling according to an embodiment of the present application, such as Figure 2 As shown, I src is the source image, I ref is the reference image, the source image I src The corresponding depth sampling X 1 , ..., X n is uniform sampling, with the same sampling interval, X' 1 , ..., X' n is the depth sampling and I src The corresponding optical center is at the intersection of the epipolar line, that is, the pixel sampling point.
[0085] When the depth sampling is uniform, the sampling interval of the pixel sampling points is uneven, such as Figure 2The interval between pixel sampling points is getting smaller and smaller. For sampling points close to the camera depth plane, their corresponding features may not be in the depth map features of the source image. Figure 2 , depth sampling X 1 X' on the corresponding limit 1 Already in the source image I src If the sampling point of the depth plane far away from the camera corresponds to the source image I src The pixel spacing on the epipolar lines is getting smaller and smaller, and the features of adjacent sampling points may be very similar.
[0086] In order to avoid the problems that may be caused by the above-mentioned uniform sampling, the method of sampling inverse depth sampling in this embodiment ensures that the depth sampling points are evenly distributed on the epipolar line.
[0087] In some embodiments, the preset sampling number acquisition model includes:
[0088]
[0089] Among them, d num is the number of depth samples, d max is the maximum depth value, d min is the minimum depth value; P 1 is the optical center coordinate in the world coordinate system, (cx, cy) is the optical center coordinate in the pixel coordinate system; P 2 is the corresponding coordinate of the point (cx+1, cy) in the pixel coordinate system in the world coordinate system, L 1 norm(P 2 -P 1 ) is P 2 To P 1 L 1 norm, T and K are coordinate transformation parameters.
[0090] In some of the embodiments, step S1026 specifically includes: generating a model based on a sampling interval, and determining a target depth sampling interval according to a maximum depth value, a minimum depth value, and a number of depth samples.
[0091] The sampling interval generation model includes:
[0092]
[0093] Among them, d interval is the target depth sampling interval, d max is the maximum depth value, d min is the minimum depth value, d num is the number of depth samples.
[0094] The target depth sampling interval is the sampling interval of the depth sampling points on the corresponding epipolar lines, that is, the sampling interval of the pixel sampling points. Figure 3 is another schematic diagram of depth sampling according to an embodiment of the present application, such as Figure 3 As shown, the source image I src The corresponding depth sampling X 1 , ...X n is uneven, the pixel sampling point X' 1 , ..., X' n is uniform.
[0095] Continue to refer to Figure 1 , after determining the target depth sampling interval, execute step S103.
[0096] Step S103 , based on the inverse depth sampling strategy, a depth map corresponding to each reference image is obtained according to the target depth sampling interval, and a mesh model of the target object is generated according to the depth map.
[0097] In some embodiments, step S103 specifically includes:
[0098] Step S1031 : Based on the inverse depth sampling strategy, a depth map of each reference image is generated according to the source image and the target depth sampling interval.
[0099] According to the source image and target depth sampling interval, the pre-trained model of deep learning is used for inference to obtain the depth map of the reference image.
[0100] Step S1031, performing depth fusion according to the depth map to obtain point cloud data of the target object.
[0101] It should be noted that the depth map fusion can be performed by stereo fusion or dynamic fusion to obtain the point cloud corresponding to the target object.
[0102] Step S1031, generating a mesh model of the target object based on the point cloud data.
[0103] Optionally, the point cloud generated above is used as input and the openmvs module is used to generate a mesh model of the target object.
[0104] Through the above steps, the target object image is collected, sparse reconstruction is performed based on the target object image, and a sparse model of the target object is obtained. The target object image includes an image pair consisting of a reference image and a source image. The sparse model includes 3D sparse points. The maximum depth value and the minimum depth value of the 3D sparse points corresponding to the source image in the camera coordinate system are determined. Based on the maximum depth value and the minimum depth value, the target depth sampling interval of the 3D sparse points on the image pole line corresponding to each source image is determined. Based on the inverse depth sampling strategy, the depth map corresponding to each reference image is obtained according to the target depth sampling interval. The mesh model of the target object is generated according to the depth map, which solves the problem of poor three-dimensional reconstruction effect based on deep learning. Based on the inverse depth sampling strategy, the pixel points on the image pole line are ensured to be uniform, which alleviates the problem of inaccurate depth estimation caused by near depth of field and deep depth of field, thereby improving the effect of three-dimensional reconstruction.
[0105] Based on the deep learning dense mapping link, the input required by the deep learning model is improved, including the depth of 3D points and the selection of source views. The inverse depth sampling strategy is used to determine the depth of 3D points. The depth probability density is calculated as a judgment method to select the appropriate source view. The higher the depth probability density, the stronger the co-viewing relationship between the two, which is more conducive to obtaining a high-quality depth map.
[0106] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0107] This embodiment also provides a three-dimensional reconstruction system based on deep learning, which is used to implement the above embodiments and preferred implementations, and will not be repeated here. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0108] Figure 4 is a structural block diagram of a deep learning-based 3D reconstruction system according to an embodiment of the present application, such as Figure 4 As shown, the system includes: a collection module 41, an analysis module 42 and a model generation module 43.
[0109] The acquisition module 41 is used to acquire the target object image, and perform sparse reconstruction based on the target object image to obtain a sparse model of the target object.
[0110] The analysis module 42 is used to obtain the maximum depth value and the minimum depth value of the 3D sparse points in the sparse model in the camera coordinate system, and determine the target depth sampling interval of the 3D sparse points on the image epipolar line corresponding to each target object image based on the maximum depth value and the minimum depth value.
[0111] The model generation module 43 is used to obtain a depth map corresponding to each target object image based on the target depth sampling interval, and generate a mesh model of the target object according to the depth map.
[0112] In some embodiments, the analysis module 42 includes: a coordinate determination module, a sampling number determination module and a sampling interval determination module.
[0113] The coordinate determination module is used to obtain the internal parameter data of the target object image from the sparse model, and obtain the optical center coordinates of the source image in the pixel coordinate system based on the internal parameter data.
[0114] A sampling number determination module is used to obtain a model based on a preset sampling number, and obtain the depth sampling number of 3D sparse points according to the optical center coordinates, maximum depth and minimum depth in the pixel coordinate system;
[0115] The sampling interval determination module is used to determine the target depth sampling interval based on the number of depth samples, the maximum depth and the minimum depth.
[0116] In some embodiments, the preset sampling number acquisition model includes:
[0117]
[0118] Among them, d num is the number of depth samples, d max is the maximum depth value, d min is the minimum depth value; P 1 is the optical center coordinate in the world coordinate system, (cx, cy) is the optical center coordinate in the pixel coordinate system; P 2 is the corresponding coordinate of the point (cx+1, cy) in the pixel coordinate system in the world coordinate system, L 1 norm(P 2 -P 1 ) is P 2 To P 1 L 1 norm, T and K are coordinate transformation parameters.
[0119] In some embodiments, the sampling interval determination module includes: a model calculation module.
[0120] The model calculation module is used to generate a model based on the sampling interval and determine the target depth sampling interval according to the maximum depth value, the minimum depth value and the number of depth samples.
[0121] The sampling interval generation model includes:
[0122]
[0123] Among them, d interval is the target depth sampling interval, d max is the maximum depth value, d min is the minimum depth value, d num is the number of depth samples.
[0124] In some of the embodiments, the system further includes: an angle determination module, a common view coefficient acquisition module and a source image determination module.
[0125] The angle determination module is used to determine the reference image from the target object image and obtain the angle between the reference image and the common viewpoint to the optical center of the image to be matched, wherein the image to be matched is the image of the target object image other than the reference image.
[0126] The common view coefficient acquisition module is used to obtain the common view coefficient of the reference image and the image to be matched according to the probability density of the angles corresponding to all common view points.
[0127] The source image determination module is used to determine the source image corresponding to the reference image from all the images to be matched based on the common view coefficient.
[0128] In some of the embodiments, the analysis module 42 further includes: a first coordinate determination module, a second coordinate determination module, and a sorting module.
[0129] The first coordinate determination module is used to obtain the camera external parameters and the first coordinates of each 3D sparse point in the world coordinate system from the sparse model.
[0130] The second coordinate determination module is used to convert the first coordinate into a second coordinate in the camera coordinate system according to the camera external parameter.
[0131] The sorting module is used to sort the depth values of each 3D sparse point based on the second coordinate to obtain a maximum depth value and a minimum depth value.
[0132] In some of the embodiments, the model generation module 43 includes: a depth map generation module, a point cloud generation module and a mesh generation module.
[0133] The depth map generation module is used to generate a depth map of each reference image based on an inverse depth sampling strategy and according to source image and target depth sampling intervals.
[0134] The point cloud generation module is used to perform depth fusion based on the depth map to obtain the point cloud data of the target object.
[0135] The mesh generation module is used to generate a mesh model of the target object based on point cloud data.
[0136] Through the above steps, the acquisition module 41 acquires the target object image, performs sparse reconstruction based on the target object image, and obtains the sparse model of the target object. The analysis module 42 obtains the maximum depth value and the minimum depth value of the 3D sparse points in the sparse model in the camera coordinate system, and determines the target depth sampling interval of the 3D sparse points on the image pole line corresponding to each target object image based on the maximum depth value and the minimum depth value. The model generation module 43 obtains the depth map corresponding to each target object image based on the target depth sampling interval, and generates the mesh model of the target object according to the depth map, which solves the problem of poor three-dimensional reconstruction effect based on deep learning. Based on the inverse depth sampling strategy, it ensures that the pixel points on the image pole line are uniform, alleviates the problem of inaccurate depth estimation caused by near depth of field and deep depth of field, thereby improving the effect of three-dimensional reconstruction.
[0137] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.
[0138] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0139] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0140] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:
[0141] S1, collecting a target object image, performing sparse reconstruction based on the target object image, and obtaining a sparse model of the target object, wherein the target object image includes an image pair consisting of a reference image and a source image, and the sparse model includes 3D sparse points.
[0142] S2, determining the maximum depth value and the minimum depth value of the 3D sparse points corresponding to the source image in the camera coordinate system, and based on the maximum depth value and the minimum depth value, determining the target depth sampling interval of the 3D sparse points on the image epipolar line corresponding to each source image.
[0143] S3, based on the inverse depth sampling strategy, obtains the depth map corresponding to each reference image according to the target depth sampling interval, and generates a mesh model of the target object according to the depth map.
[0144] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.
[0145] In one embodiment, Figure 5 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 5 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 5 As shown. The electronic device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a three-dimensional reconstruction method based on deep learning is implemented.
[0146] Those skilled in the art will understand that Figure 5 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0148] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0149] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A three-dimensional reconstruction method based on deep learning, characterized in that: The method comprises: Acquire a target object image, and perform sparse reconstruction based on the target object image to obtain a sparse model of the target object, wherein the target object image includes an image pair consisting of a reference image and a source image, and the sparse model includes 3D sparse points; Determine a maximum depth value and a minimum depth value of the 3D sparse points corresponding to the source image in the camera coordinate system, and determine a target depth sampling interval of the 3D sparse points on the image epipolar lines corresponding to each of the source images based on the maximum depth value and the minimum depth value; Based on the inverse depth sampling strategy, a depth map corresponding to each of the reference images is obtained according to the target depth sampling interval, and a mesh model of the target object is generated according to the depth map.
2. The method according to claim 1, characterized in that The determining, based on the maximum depth value and the minimum depth value, a target depth sampling interval of the 3D sparse points on the image epipolar line corresponding to each of the source images comprises: Acquire the intrinsic parameter data of the source image from the sparse model, and obtain the optical center coordinates of the source image in a pixel coordinate system according to the intrinsic parameter data; Based on a preset sampling number acquisition model, the depth sampling number of the 3D sparse point is obtained according to the optical center coordinates in the pixel coordinate system, the maximum depth and the minimum depth; The target depth sampling interval is determined based on the number of depth samples, the maximum depth, and the minimum depth.
3. The method according to claim 2, characterized in that The preset sampling number acquisition model includes: Among them, d num is the number of depth samples, d max is the maximum depth value, d min is the minimum depth value; P1 is the optical center coordinate in the world coordinate system, (cx, cy) is the optical center coordinate in the pixel coordinate system; P2 is the corresponding coordinate of the point (cx+1, cy) in the pixel coordinate system in the world coordinate system, L1norm(P2-P1) is the L1 norm from P2 to P1, and T and K are coordinate conversion parameters.
4. The method according to claim 2, characterized in that: The determining the target depth sampling interval based on the number of depth samples, the maximum depth and the minimum depth comprises: Based on a sampling interval generation model, determining the target depth sampling interval according to the maximum depth value, the minimum depth value and the number of depth samples; The sampling interval generation model includes: Among them, d interval is the target depth sampling interval, d max is the maximum depth value, d min is the minimum depth value, d num is the number of depth samples.
5. The method according to claim 1, characterized in that: After acquiring the target object image, the method further includes: Determine a reference image from the target object image, and obtain an angle between the reference image and a line connecting a common viewpoint to an optical center of the image to be matched, wherein the image to be matched is an image in the target object image other than the reference image; Obtaining a common view coefficient between the reference image and the image to be matched according to the probability density of the angles corresponding to all the common view points; Based on the common view coefficient, a source image corresponding to the reference image is determined from all the images to be matched.
6. The method according to claim 1, characterized in that Determining the maximum depth value and the minimum depth value of the 3D sparse point corresponding to the source image in the camera coordinate system includes: Obtaining, from the sparse model, camera extrinsic parameters and the first coordinates of each of the 3D sparse points in the world coordinate system; Convert the first coordinate into a second coordinate in a camera coordinate system according to the camera extrinsic parameter; Based on the second coordinate, the depth values of the 3D sparse points are sorted to obtain a maximum depth value and a minimum depth value.
7. The method according to claim 1, characterized in that The step of obtaining a depth map corresponding to each of the reference images according to the target depth sampling interval based on the inverse depth sampling strategy, and generating a mesh model of the target object according to the depth map comprises: Based on an inverse depth sampling strategy, generating a depth map of each of the reference images according to the source image and the target depth sampling interval; Performing depth fusion according to the depth map to obtain point cloud data of the target object; Based on the point cloud data, a mesh model of the target object is generated.
8. A three-dimensional reconstruction system based on deep learning, characterized in that: The system includes: a collection module, an analysis module and a model generation module, wherein: The acquisition module is used to acquire a target object image, and perform sparse reconstruction based on the target object image to obtain a sparse model of the target object, wherein the target object image includes an image pair consisting of a reference image and a source image, and the sparse model includes 3D sparse points; The analysis module is used to determine the maximum depth value and the minimum depth value of the 3D sparse points corresponding to the source image in the camera coordinate system, and determine the target depth sampling interval of the 3D sparse points on the image epipolar line corresponding to each of the source images based on the maximum depth value and the minimum depth value; The model generation module is used to obtain a depth map corresponding to each of the reference images according to the target depth sampling interval based on an inverse depth sampling strategy, and generate a mesh model of the target object according to the depth map.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the deep learning-based three-dimensional reconstruction method according to any one of claims 1 to 7 is implemented.
10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the deep learning-based three-dimensional reconstruction method as described in any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Three-dimensional reconstruction method, device and equipment, readable storage medium and program product
CN121213801A
3D reconstruction methods, apparatus, equipment, readable storage media, and program products
CN121213801B
