A three-dimensional reconstruction method, device, electronic device and storage medium

Abstract

The present application provides a three-dimensional reconstruction method, device, electronic device and storage medium, which relate to the field of computer vision technology. The method includes: constructing a neighborhood facet of the seed facet based on a seed facet taken out from the facet stack of the target object; determining a consistency constraint coefficient based on the continuous judgment results of the seed facet and the neighborhood facet; adjusting the facet using a main constraint function including a first correlation coefficient, the consistency constraint coefficient and a consistency constraint function, adding the adjusted neighborhood facet corresponding to the minimum value of the main constraint function as a new seed facet to the facet stack, and repeating the process until all the seed faces in the facet stack are taken out; and reconstructing the target object in three dimensions based on all the adjusted neighborhood faces. The implementation of the present application can improve the accuracy of the three-dimensional model of the target scene (target object) restored using the neighborhood faces constructed above.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a three-dimensional reconstruction method, device, electronic device and storage medium. Background Art

[0002] With the development of oblique photography technology, 3D reconstruction has been widely used in smart cities, geological prevention and control, urban management and other fields. Among them, the main purpose of 3D reconstruction is to restore the 3D model of the target scene, and image-based 3D reconstruction is the main technical means to obtain the 3D model.

[0003] At present, the dense point cloud is mainly obtained by the patch-based 3D reconstruction algorithm (PMVS) in the Multi-View Stereo (MVS) method, and then the 3D model of the target scene is restored based on the dense point cloud. In the process of constructing neighborhood patches from seed patches and finally forming a dense point cloud (patch), the three degrees of freedom of the initial neighborhood patch are adjusted according to the correlation coefficient between the area of ​​the initial neighborhood patch projected in the reference image and the area projected in each search image, and finally a neighborhood patch that meets the correlation coefficient condition is constructed.

[0004] However, in order to meet the correlation coefficient condition, the three degrees of freedom of the initial neighborhood patch will change without constraints, which will make the consistency between the constructed neighborhood patch and the seed patch worse, and thus reduce the accuracy of the three-dimensional model of the target scene restored based on the point cloud (patch). Summary of the invention

[0005] The purpose of the present application is to provide a three-dimensional reconstruction method, device, electronic device and storage medium to improve the accuracy of the three-dimensional model of the target scene restored based on the point cloud (face patch) in order to address the deficiencies in the above-mentioned prior art.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a three-dimensional reconstruction method, the method comprising:

[0008] Taking out a seed patch from a patch stack of the target object, wherein the patch stack includes: a plurality of seed patches;

[0009] If the extracted seed patch does not construct a neighboring patch, construct a neighboring patch of the extracted seed patch according to the extracted seed patch;

[0010] Determining a consistency constraint coefficient corresponding to the continuous judgment result according to the continuous judgment result of the extracted seed face patch and the neighboring face patch;

[0011] According to the normal vector degrees of freedom of the neighborhood face patch and the consistency constraint coefficient, the neighborhood face patch is adjusted using a preset main constraint function so that the main constraint function value is minimized; wherein the main constraint function includes: a first correlation coefficient and a consistency constraint function, wherein the first correlation coefficient is a correlation coefficient of the adjusted domain face patch projected onto a plurality of image projection windows, and the consistency constraint function is used to constrain the normal vector degrees of freedom in combination with the consistency constraint coefficient;

[0012] Add the adjusted neighborhood patch corresponding to the minimum principal constraint function value as a new seed patch to the patch stack, and repeat the process until all seed patches in the patch stack are taken out and all the taken out seed patches have corresponding neighborhood patches constructed thereon;

[0013] The target object is three-dimensionally reconstructed according to all adjusted neighborhood patches.

[0014] Optionally, the determining, according to the continuous judgment result between the retrieved seed patch and the neighboring patch, a consistency constraint coefficient corresponding to the continuous judgment result includes:

[0015] If the continuity judgment result is that the extracted seed patch is continuous with the neighboring patch, then 1 is determined as the consistency constraint coefficient;

[0016] If the continuity judgment result is that the extracted seed patch is discontinuous with the neighboring patch, 0 is determined as the consistency constraint coefficient.

[0017] Optionally, before determining the consistency constraint coefficient corresponding to the continuous judgment result between the retrieved seed patch and the neighboring patch, the method further includes:

[0018] Calculating a second correlation coefficient between a first projection window and a second projection window, wherein the first projection window is a projection window of the extracted seed patch on the corresponding reference image, and the second projection window is a projection window of the neighborhood patch on the corresponding reference image;

[0019] If the second correlation coefficient is greater than or equal to a second preset threshold, determining the continuity judgment result is: the extracted seed patch is continuous with the neighboring patch;

[0020] If the second correlation coefficient is less than the second preset threshold, the continuity judgment result is determined as: the extracted seed patch is discontinuous with the neighborhood patch.

[0021] Optionally, the adjusting the neighborhood facets according to the normal vector degrees of freedom of the neighborhood facets and the consistency constraint coefficient using a preset main constraint function so that the main constraint function value is minimized includes:

[0022] According to the normal vector degree of freedom of the neighborhood face patch and the consistency constraint coefficient, the main constraint function is used to calculate the main constraint function value of the neighborhood face patch;

[0023] The positional degrees of freedom and normal vector degrees of freedom of the neighborhood facet are adjusted until the main constraint function value of the adjusted neighborhood facet is minimized; when the main constraint function value is minimized, the first correlation coefficient of the projection window of the adjusted neighborhood facet projected onto the multiple images is optimal.

[0024] Optionally, before taking out a seed patch from the patch stack of the target object, the method further includes:

[0025] Performing feature matching on multiple images of the target object to obtain multiple initial seed patches;

[0026] The patch stack including the plurality of initial seed patches is created.

[0027] Optionally, the plurality of seed patches in the patch stack are sorted according to confidence levels; and the extracting a seed patch from the patch stack of the target object comprises:

[0028] A seed dough piece with the highest confidence is selected from the dough piece stack as the removed seed dough piece.

[0029] Optionally, the method of adding the adjusted neighborhood patch corresponding to the minimum principal constraint function value as a new seed patch to the patch stack includes:

[0030] Determine whether a first correlation coefficient corresponding to when the primary constraint function value reaches a minimum is greater than or equal to a first preset threshold;

[0031] If the first correlation coefficient is greater than or equal to the first preset threshold, the adjusted neighborhood patch corresponding to the first correlation coefficient is added to the patch stack as a new seed patch.

[0032] In a second aspect, an embodiment of the present application further provides a three-dimensional reconstruction device, the device comprising:

[0033] An acquisition module is used to take out a seed patch from a patch stack of a target object, wherein the patch stack includes: a plurality of seed patches;

[0034] A construction module, configured to construct a neighboring patch of the extracted seed patch according to the extracted seed patch if the neighboring patch is not constructed by the extracted seed patch;

[0035] A determination module, configured to determine a consistency constraint coefficient corresponding to a continuous judgment result of the extracted seed face patch and the neighboring face patch according to the continuous judgment result of the extracted seed face patch and the neighboring face patch;

[0036] An adjustment module is used to adjust the neighborhood facet according to the normal vector freedom degree of the neighborhood facet and the consistency constraint coefficient, using a preset main constraint function to minimize the main constraint function value; wherein the main constraint function includes: a first correlation coefficient and a consistency constraint function, wherein the first correlation coefficient is a correlation coefficient of the adjusted domain facet projected onto a plurality of image projection windows, and the consistency constraint function is used to constrain the normal vector freedom degree in combination with the consistency constraint coefficient;

[0037] Adding a module, adding the adjusted neighborhood patch corresponding to the minimum principal constraint function value as a new seed patch to the patch stack, and repeating the process until all seed patches in the patch stack are taken out and all the taken out seed patches are constructed with corresponding neighborhood patches;

[0038] The reconstruction module is used to perform three-dimensional reconstruction of the target object according to all adjusted neighborhood patches.

[0039] Optionally, the determination module is specifically used to determine 1 as the consistency constraint coefficient if the continuity judgment result is: the extracted seed patch is continuous with the neighboring patch; if the continuity judgment result is: the extracted seed patch is discontinuous with the neighboring patch, then determine 0 as the consistency constraint coefficient.

[0040] Optionally, the determination module is also specifically used to calculate a second correlation coefficient between the first projection window and the second projection window, wherein the first projection window is the projection window of the extracted seed patch on the corresponding reference image, and the second projection window is the projection window of the neighborhood patch on the corresponding reference image; if the second correlation coefficient is greater than or equal to a second preset threshold, then the continuity judgment result is determined to be: the extracted seed patch is continuous with the neighborhood patch; if the second correlation coefficient is less than the second preset threshold, then the continuity judgment result is determined to be: the extracted seed patch is discontinuous with the neighborhood patch.

[0041] Optionally, the adjustment module is specifically used to calculate the main constraint function value of the neighborhood facet based on the normal vector degrees of freedom of the neighborhood facet and the consistency constraint coefficient, using the main constraint function; adjusting the position degrees of freedom and normal vector degrees of freedom of the neighborhood facet until the main constraint function value of the adjusted neighborhood facet is minimized; when the main constraint function value is minimized, the first correlation coefficient of the projection window of the adjusted neighborhood facet onto the multiple images is optimal.

[0042] Optionally, the device further comprises:

[0043] A matching module, used for performing feature matching on multiple images of the target object to obtain multiple initial seed patches;

[0044] A creation module is used to create the patch stack including the multiple initial seed patches.

[0045] Optionally, the plurality of seed patches in the patch stack are sorted according to confidence levels; and the acquisition module is specifically configured to select a seed patch with the highest confidence level from the patch stack as the extracted seed patch.

[0046] Optionally, the adding module is specifically used to determine whether the first correlation coefficient corresponding to when the main constraint function value reaches the minimum is greater than or equal to a first preset threshold; if the first correlation coefficient is greater than or equal to the first preset threshold, the adjusted neighborhood facet corresponding to the first correlation coefficient is added to the facet stack as a new seed facet.

[0047] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the three-dimensional reconstruction method of the first aspect mentioned above.

[0048] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the three-dimensional reconstruction method of the first aspect are executed.

[0049] The beneficial effects of this application are:

[0050] The embodiments of the present application provide a three-dimensional reconstruction method, device, electronic device and storage medium, the method comprising: taking out a seed patch from a patch stack of a target object, wherein the patch stack comprises: a plurality of seed patches; if the taken out seed patch does not construct a neighborhood patch, constructing a neighborhood patch of the taken out seed patch according to the taken out seed patch; determining a consistency constraint coefficient corresponding to a continuous judgment result between the taken out seed patch and the neighborhood patch; adjusting the neighborhood patch according to the normal vector degrees of freedom of the neighborhood patch and the consistency constraint coefficient using a preset main constraint function so that the main constraint function value is minimized; wherein the main constraint function comprises: a first correlation coefficient and a consistency constraint function, the first correlation coefficient being a correlation coefficient of the adjusted domain patch projected onto a plurality of image projection windows. The consistency constraint function is used to constrain the normal vector degrees of freedom in combination with the consistency constraint coefficient; the adjusted neighborhood patch corresponding to the minimum main constraint function value is added to the patch stack as a new seed patch, and the execution is repeated until all seed patches in the patch stack are taken out and the taken out seed patches are constructed with corresponding neighborhood patches; based on all adjusted neighborhood patches, the target object is three-dimensionally reconstructed.

[0051] The embodiment of the present application provides a three-dimensional reconstruction method, by judging the continuous results of the seed patch and the neighborhood patch, determining the value of the consistency constraint coefficient in the consistency constraint function, and using the main constraint function containing the consistency constraint function to adjust the parameters (position degrees of freedom, normal vector degrees of freedom) of the neighborhood patch, so that the value of the main constraint function is minimized, and the corresponding neighborhood patch is the optimal neighborhood patch. In other words, since the consistency constraint function in the main constraint function constrains the normal vector degrees of freedom of the neighborhood patch, the initial value of the neighborhood patch parameter is not only related to the seed patch, but also in the process of adjusting the neighborhood patch parameters, it will also be affected by the seed patch, thereby improving the consistency between the neighborhood patch and the seed patch. In addition, the accuracy of the three-dimensional model of the target scene (target object) restored based on the neighborhood patch constructed as above will also be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A schematic diagram of a process flow of a three-dimensional reconstruction method provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of a flow chart of another three-dimensional reconstruction method provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of a flow chart of another three-dimensional reconstruction method provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of a flow chart of another three-dimensional reconstruction method provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of a flow chart of another three-dimensional reconstruction method provided in an embodiment of the present application;

[0058] Figure 6 A schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of the present application;

[0059] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0061] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0062] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0063] Figure 1 A schematic diagram of a three-dimensional reconstruction method provided in an embodiment of the present application. Figure 1 As shown, the method may include:

[0064] S101. Take out a seed patch from a patch stack of a target object, wherein the patch stack includes: a plurality of seed patches.

[0065] An image collector (such as a camera) can be used to shoot a target object (such as a building) from multiple angles to collect multiple images of the target object, and multiple seed patches can be obtained based on the feature points in the multiple images. At the same time, each seed patch is placed in a patch stack according to certain rules. The patch stack is equivalent to a container that can store each seed patch according to certain rules. When constructing a neighborhood surface on its neighborhood through a seed patch, it is necessary to take a seed patch from the patch stack and delete the information of the seed patch at the same time.

[0066] Here, the concept of a patch is explained. A patch is essentially equivalent to a plane that is approximately tangent to the surface of the target object. It is a small rectangle in a three-dimensional space. A patch can be described by three-dimensional coordinate information and normal vector information. It should be noted that this application does not limit the number of seed patches in the patch stack.

[0067] S102: If the extracted seed patch has no neighboring patch, construct a neighboring patch of the extracted seed patch according to the extracted seed patch.

[0068] If the patch stack already has corresponding neighboring patches for each neighborhood of the seed patch, the seed patch can be directly deleted from the patch stack without constructing a neighborhood patch based on the seed patch; if the patch stack does not have corresponding neighboring patches for each neighborhood of the seed patch or does not have a neighborhood patch corresponding to a certain neighborhood, the neighborhood patches of its neighborhood can be constructed based on the seed patch, and the seed patch can be deleted from the patch stack, and the constructed domain patch can be placed in the patch stack.

[0069] S103: Determine a consistency constraint coefficient corresponding to the continuous judgment result according to the continuous judgment result between the extracted seed face patch and the neighborhood face patch.

[0070] According to the three-dimensional coordinate information, normal vector information of the seed patch and the positional relationship between each neighbor and the seed patch in the three-dimensional space, a neighborhood patch corresponding to each neighborhood is constructed at the same time. Since the neighborhood patch and the seed patch are adjacent in the three-dimensional space, the neighborhood patch and the seed patch have similar characteristics in three-dimensional coordinate information and normal vector information, that is, the initial three-dimensional coordinate information of the neighborhood patch is obtained by pre-setting the three-dimensional coordinate information of the seed patch, and the initial normal vector information of the neighborhood patch is consistent with the normal vector information of the seed patch.

[0071] That is to say, the initial three-dimensional coordinate information and initial normal vector information of the neighborhood patch are known information. The corresponding reference image can be obtained based on the normal vector information of the seed patch. Similarly, the corresponding initial reference image can be obtained based on the initial normal vector information of the neighborhood patch. Since the normal vector information of the seed patch is consistent with the initial normal vector information of the neighborhood patch, the reference image of the neighborhood patch and the seed patch is the same. Based on the projection information of the neighborhood patch and the seed patch in the reference image, it can be determined whether the neighborhood patch and the seed patch are continuous, and the consistency constraint coefficient (λ) is determined based on the judgment result.

[0072] S104. According to the degree of freedom of the normal vector of the neighborhood face and the consistency constraint coefficient, a preset main constraint function is used to adjust the neighborhood face so that the main constraint function value is minimized.

[0073] Among them, the main constraint function includes: a first correlation coefficient and a consistency constraint function. The first correlation coefficient is the correlation coefficient of the adjusted domain surface projected onto multiple image projection windows. The consistency constraint function is used to constrain the normal vector degrees of freedom in combination with the consistency constraint coefficient.

[0074] In order to simplify the calculation, the three-dimensional coordinate system information of the neighborhood patch can be fixed on a ray. In this way, the six degrees of freedom of the neighborhood patch (including three-dimensional coordinate information (x, y, z) and normal vector information (a, b, c)) can be simplified to three degrees of freedom (including the distance moved along the ray (z) and the normal vector azimuth (α, β)), among which the normal vector azimuth (α, β) and the normal vector information (a, b, c) have the following corresponding relationship:

[0075]

[0076] The three degrees of freedom information (z, α, β) of the neighborhood patch can be obtained according to the above-mentioned initial three-dimensional information and the initial normal vector information, and a reference image and multiple search images can be selected from the multiple images obtained by shooting according to the normal vector information (normal vector degrees of freedom) of the neighborhood patch, wherein the reference image is defined as the image corresponding to the optical axis with the smallest angle with the normal vector of the neighborhood patch among the multiple images, and the search image is defined as the image with an angle with the normal vector of the neighborhood patch less than a preset value (such as 60°) among the multiple images. Under the premise that the reference image and the search image are determined, the projection area (projection window) of the neighborhood patch in a reference image and each search image can be obtained, and the correlation coefficient of the projection area obtained by projecting the neighborhood patch to the reference image and each search image can be calculated, and each correlation coefficient is added to obtain the first correlation coefficient. At the same time, the consistency constraint function values ​​about α and β can be calculated respectively according to the normal vector azimuth (α, β).

[0077] The main constraint function includes the first correlation coefficient and the consistency constraint function that constrains the normal vector freedom (normal vector azimuth (α, β)) of the neighborhood face in combination with the consistency constraint coefficient (λ). Adjusting the neighborhood face is essentially to determine whether the neighborhood face is optimal by the change of the main constraint function. The first correlation coefficient and the consistency constraint function value jointly determine the main constraint function value, that is, when the main constraint function value reaches the minimum, the corresponding first correlation coefficient and consistency constraint function value are optimal, that is, the adjusted neighborhood face is optimal at this time.

[0078] It can be seen that here not only the first correlation coefficient of the projection window of the neighborhood patch projected onto multiple images is used to optimize the neighborhood patch, but also the normal vector freedom of the neighborhood patch is constrained to optimize the neighborhood patch. In other words, the change of the freedom of the neighborhood patch can be constrained, which can avoid the phenomenon that the obtained domain patch is a flying point when the regional textures on the target object are relatively similar.

[0079] S105. Add the adjusted neighborhood patch corresponding to the minimum principal constraint function value as a new seed patch to the patch stack, and repeat the process until all seed patches in the patch stack are taken out and all the taken out seed patches have corresponding neighborhood patches constructed thereon.

[0080] When the value of the above-mentioned main constraint function reaches the minimum, the first correlation coefficient has reached the optimal level and the consistency constraint function has also reached the optimal level, that is, the adjusted neighborhood patch has reached the optimal level. At this time, the neighborhood patch can be added to the patch stack as a new seed patch, and then steps S101 to S105 are repeated. Since each time a seed patch is taken out from the patch stack, it is equivalent to deleting the seed patch from the patch stack. When all seed patches and all neighborhood patches have completed a construction (diffusion), the number of seed patches in the patch stack is 0, which is equivalent to all seed patches in the patch stack being taken out. After executing the preset number of diffusions, each seed patch can construct a corresponding neighborhood patch in the corresponding neighborhood.

[0081] S106: Perform three-dimensional reconstruction on the target object according to all adjusted neighborhood patches.

[0082] When the number of seed patches in the patch stack is 0, all adjusted domain patches and their corresponding seed patches can be filtered, and patches that do not meet the preset rules can be eliminated to obtain multiple patches corresponding to the elimination operation. The multiple patches are placed back in the patch stack as seed patches, and steps S101 to S105 are repeated. Generally, three iterations can be performed, which can ensure the accuracy and completeness of the patches, thereby improving the accuracy of the three-dimensional model of the target scene restored based on all the patches.

[0083] In summary, in the three-dimensional reconstruction method provided by the present application, the value of the consistency constraint coefficient in the consistency constraint function is determined by judging the continuous results of the seed patch and the neighborhood patch, and the parameters (position degrees of freedom, normal vector degrees of freedom) of the neighborhood patch are adjusted by the main constraint function containing the consistency constraint function, so that the value of the main constraint function is minimized, and the corresponding neighborhood patch is the optimal neighborhood patch. In other words, since the consistency constraint function in the main constraint function constrains the normal vector degrees of freedom of the neighborhood patch, the initial value of the neighborhood patch parameter is not only related to the seed patch, but also in the process of adjusting the neighborhood patch parameters, it will also be affected by the seed patch, thereby improving the consistency between the neighborhood patch and the seed patch. In addition, the accuracy of the three-dimensional model of the target scene (target object) restored based on the neighborhood patch constructed as above will also be improved.

[0084] Figure 2 A schematic diagram of another 3D reconstruction method provided in an embodiment of the present application. Figure 2 As shown, optionally, before determining the consistency constraint coefficient corresponding to the continuous judgment result based on the continuous judgment result between the extracted seed patch and the neighborhood patch, the method further includes:

[0085] S201. Calculate a second correlation coefficient between a first projection window and a second projection window, wherein the first projection window is a projection window of the extracted seed patch on a corresponding reference image, and the second projection window is a projection window of the neighborhood patch on a corresponding reference image.

[0086] Since the three-dimensional coordinate information and normal vector information of the seed patch are known, and the neighborhood patch is a neighborhood patch in the neighborhood of the seed patch in three-dimensional space, the initial three-dimensional coordinate information and initial normal vector information of the neighborhood patch are also known. An image corresponding to the optical axis with the smallest angle between the normal vector of the seed patch can be found from the multiple images taken, and the image can be used as the reference image corresponding to the seed patch. Similarly, the reference image corresponding to the neighborhood patch can also be found. Since the initial normal vector information of the neighborhood patch is consistent with the normal vector information of the seed patch, the reference image corresponding to the seed patch and the neighborhood patch is one. The projection areas of the seed patch and the neighborhood patch in the reference image can be called the first projection window and the second projection window respectively. According to the pixel information of the first projection window and the second projection window, the second correlation coefficient of the seed patch and the neighborhood patch can be calculated by the normalized correlation coefficient algorithm.

[0087] S202: If the second correlation coefficient is greater than or equal to a second preset threshold, determine that the continuity judgment result is: the retrieved seed patch is continuous with the neighborhood patch.

[0088] S203: If the second correlation coefficient is less than the second preset threshold, determine that the continuity judgment result is: the retrieved seed patch is discontinuous with the neighborhood patch.

[0089] The second preset threshold can be set according to actual needs, such as the second preset threshold can be 0.7, which is not limited in this application. According to the relationship between the second correlation coefficient and the second preset threshold, it can be determined whether the seed patch and the neighborhood patch are continuous, and the specific value of the consistency constraint coefficient can be determined based on whether the two are continuous. When the second correlation coefficient is greater than or equal to the second preset threshold (such as 0.7), it means that the seed patch and the neighborhood patch are continuous, and then 1 can be determined as the consistency constraint coefficient; when the second correlation coefficient is less than the second preset threshold (such as 0.7), it means that the seed patch and the neighborhood patch are discontinuous, and then 0 can be determined as the consistency constraint coefficient.

[0090] Figure 3 A flowchart of another three-dimensional reconstruction method provided in an embodiment of the present application is shown below. Figure 3 As shown, optionally, the above-mentioned adjusting the neighborhood patch using a preset main constraint function according to the normal vector freedom degree of the neighborhood patch and the consistency constraint coefficient so that the main constraint function value is minimized includes:

[0091] S301. Calculate the main constraint function value of the neighborhood face patch according to the normal vector degree of freedom of the neighborhood face patch and the consistency constraint coefficient using the main constraint function.

[0092] The normal vector azimuth (α, β) of the neighborhood patch can be obtained according to the normal vector freedom of the neighborhood patch. The main constraint function is: The consistency constraint function is:

[0093] Among them, α and β represent the normal vector direction angles of the domain patch respectively; f i represents the correlation coefficient of the projection window of the neighborhood patch onto the i-th image (search image); n represents the number of images (search images); λ represents the consistency constraint coefficient; x represents the normal vector direction angle variable of the domain patch; δ is a constant.

[0094] The consistency constraint function S(x) can be used to calculate the consistency constraint function values ​​s(α) and s(β) of α and β, respectively. That is, the consistency constraint function constrains the normal vector freedom of the neighborhood patch. The correlation coefficient f of the projection window of the neighborhood patch onto the i-th image can be calculated through the normal vector azimuth (α, β) of the neighborhood patch. i , and the consistency constraint function values ​​(s(α), s(β)), and then the main constraint function values ​​of the neighborhood patch can be calculated.

[0095] S302, adjusting the positional degrees of freedom and normal vector degrees of freedom of the neighborhood face until the main constraint function value of the adjusted neighborhood face is minimized; when the main constraint function value is minimized, the first correlation coefficient of the projection window of the adjusted neighborhood face projected onto multiple images is optimal.

[0096] Among them, adjusting the positional degrees of freedom (z) and normal vector degrees of freedom (α, β) of the neighborhood patch is essentially changing the value of the first correlation coefficient and the value of the consistency constraint function s(x). When the value corresponding to the main constraint function reaches the minimum, the value of the first correlation coefficient reaches the optimal value and the consistency constraint function reaches the optimal value. In other words, the neighborhood patch corresponding to the parameters (z, α, β) at this time is the optimal patch. When the value corresponding to the main constraint function does not reach the minimum, it is necessary to continue to adjust the parameters (z, α, β) of the neighborhood patch. To restate the above content, the continuity of the current domain patch and the seed patch is judged, and the consistency constraint coefficient is determined. For other contents, refer to the above description until the neighborhood patch reaches the optimal value. It should be noted that when adjusting the neighborhood patch, the corresponding reference image and search image can be re-determined based on the adjusted normal vector information, or the corresponding reference image and search image can be directly determined using the normal vector information before the adjustment. This application does not limit it. Until the neighborhood patch reaches the optimal value.

[0097] It can be seen that when constructing neighborhood patches on each neighborhood of the seed patch in the above manner, the phenomenon of holes in the neighborhood due to similar neighborhood textures is avoided, that is, the constructed neighborhood patches will be filtered out, that is, this can improve the matching degree between the seed patch and the neighborhood patch.

[0098] Figure 4 A schematic diagram of a flow chart of another 3D reconstruction method provided in an embodiment of the present application. Figure 4 As shown, optionally, before taking out a seed patch from the patch stack of the target object, the method further includes:

[0099] S401 , performing feature matching on multiple images of the target object to obtain multiple initial seed patches.

[0100] S402: Create a patch stack including a plurality of initial seed patches.

[0101] The target object can be photographed from multiple angles by an image collector (such as a camera) to collect multiple images corresponding to the target object, and feature points in each image can be extracted by a feature extraction algorithm, wherein the feature extraction algorithm can include: Harris algorithm, DoG (Difference of Gaussian) algorithm and Sift algorithm. Each feature point is matched according to the feature point matching algorithm to obtain feature points with the same features on at least a preset number of images, wherein the preset number is at least 2. After obtaining multiple sets of matched feature points, multiple ground seed points can be obtained according to the forward intersection algorithm, and then the corresponding initial seed patch can be obtained. The initial seed patch includes at least three types of information: three-dimensional coordinate information, normal vector information and area information. The patch stack can be constructed according to the features (such as confidence) of each initial seed patch. The patch stack is equivalent to a container, and the information corresponding to each seed patch (three-dimensional coordinate information, normal vector information and area information) can be stored according to certain rules.

[0102] Optionally, multiple seed patches in the patch stack are sorted according to confidence levels, and the extracting a seed patch from the patch stack of the target object includes: selecting a seed patch with the highest confidence level from the patch stack as the extracted seed patch.

[0103] Specifically, the confidence corresponding to each seed patch can be calculated according to the confidence algorithm, and the seed patch with the highest confidence can be placed at the top layer of the patch stack, and the seed patch with the lowest confidence can be placed at the bottom layer of the patch stack. Taking out the corresponding seed patch from the top layer of the patch stack is equivalent to deleting the seed patch on that layer. When a neighboring patch is added to the patch stack as a new seed patch, the new seed patch and the remaining seed patches in the patch stack need to be reordered according to the confidence. Next time, the seed patch from the top layer of the patch stack is taken out to construct a neighboring patch, and so on, until there are no seed patches on each layer of the patch stack.

[0104] Figure 5 A flowchart of another three-dimensional reconstruction method provided in an embodiment of the present application is shown below. Figure 5 As shown, optionally, the adjusted neighborhood patch corresponding to the minimum primary constraint function value is added to the patch stack as a new seed patch, and the method includes:

[0105] S501. Determine whether a first correlation coefficient corresponding to when the primary constraint function value reaches a minimum is greater than or equal to a first preset threshold.

[0106] S502: If the first correlation coefficient is greater than or equal to the first preset threshold, the adjusted neighborhood patch corresponding to the largest first correlation coefficient is added to the patch stack as a new seed patch.

[0107] Specifically, each time the neighborhood patch is adjusted, that is, the neighborhood patch degrees of freedom (z, α, β) are adjusted, the reference image and search image corresponding to the neighborhood patch can be obtained at the same time, and the correlation coefficient between the projection area of ​​the neighborhood patch on the reference image and the projection area on each search image can be calculated according to the normalized mutual correlation coefficient algorithm. The threshold of the correlation coefficient of the projection area of ​​the neighborhood patch projected on each search image can be set in advance, and the first preset threshold corresponding to the sum of each correlation coefficient (first correlation coefficient) can also be set. When the neighborhood patch meets the main constraint function condition (the main constraint function value is the minimum), it is also necessary to determine whether the first correlation coefficient of the neighborhood patch is greater than or equal to the first preset threshold. If the neighborhood patch meets both conditions, then the neighborhood patch can be added to the patch stack. If the neighborhood patch meets the main constraint function condition but does not meet the condition that the first correlation coefficient is greater than or equal to the first preset threshold, then it is necessary to abandon the neighborhood patch, that is, not to add the neighborhood patch to the patch stack. In this way, the accuracy of the three-dimensional model of the restored target scene (target object) can be further improved.

[0108] Figure 6 A schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device may include:

[0109] The acquisition module 601 is used to take out a seed patch from a patch stack of a target object, wherein the patch stack includes: a plurality of seed patches;

[0110] A construction module 602 is used to construct a neighboring patch of the extracted seed patch according to the extracted seed patch if the neighboring patch is not constructed by the extracted seed patch;

[0111] A determination module 603 is used to determine a consistency constraint coefficient corresponding to the continuous judgment result according to the continuous judgment result between the extracted seed face and the neighboring face;

[0112] An adjustment module 604 is used to adjust the neighborhood face patch according to the normal vector freedom degree of the neighborhood face patch and the consistency constraint coefficient by using a preset main constraint function to minimize the main constraint function value;

[0113] Adding module 605 is used to add the adjusted neighborhood patch corresponding to the minimum principal constraint function value as a new seed patch to the patch stack, and repeat the process until all seed patches in the patch stack are taken out and all the taken out seed patches are constructed with corresponding neighborhood patches.

[0114] The reconstruction module 606 is used to perform three-dimensional reconstruction of the target object according to all adjusted neighborhood patches.

[0115] Optionally, determination module 603 is specifically used to determine 1 as the consistency constraint coefficient if the continuity judgment result is: the extracted seed patch is continuous with the neighborhood patch; if the continuity judgment result is: the extracted seed patch is discontinuous with the neighborhood patch, then determine 0 as the consistency constraint coefficient.

[0116] Optionally, the determination module 603 is also specifically used to calculate a second correlation coefficient between the first projection window and the second projection window, wherein the first projection window is the projection window of the extracted seed patch on the corresponding reference image, and the second projection window is the projection window of the neighborhood patch on the corresponding reference image; if the second correlation coefficient is greater than or equal to a second preset threshold, then the continuity judgment result is determined to be: the extracted seed patch is continuous with the neighborhood patch; if the second correlation coefficient is less than the second preset threshold, then the continuity judgment result is determined to be: the extracted seed patch is discontinuous with the neighborhood patch.

[0117] Optionally, the adjustment module 604 is specifically used to calculate the main constraint function value of the neighborhood facet based on the normal vector degrees of freedom of the neighborhood facet and the consistency constraint coefficient, using the main constraint function; adjusting the position degrees of freedom and the normal vector degrees of freedom of the neighborhood facet until the main constraint function value of the adjusted neighborhood facet is minimized; when the main constraint function value is minimized, the first correlation coefficient of the adjusted neighborhood facet projected onto the projection window on the multiple images is optimal.

[0118] Optionally, the device further comprises:

[0119] A matching module, used for performing feature matching on multiple images of the target object to obtain multiple initial seed patches;

[0120] A creation module is used to create a patch stack containing multiple initial seed patches.

[0121] Optionally, multiple seed patches in the patch stack are sorted according to their confidence levels, and the acquisition module 601 is specifically configured to select a seed patch with the highest confidence level from the patch stack as the removed seed patch.

[0122] Optionally, module 605 is added, which is specifically used to determine whether the first correlation coefficient corresponding to when the main constraint function value reaches the minimum is greater than or equal to a first preset threshold; if the first correlation coefficient is greater than or equal to the first preset threshold, the adjusted neighborhood facet corresponding to the largest first correlation coefficient is added to the facet stack as a new seed facet.

[0123] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0124] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0125] Figure 7This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device may include: a processor 701, a storage medium 702 and a bus 703, the storage medium 702 stores machine-readable instructions executable by the processor 701, when the electronic device is running, the processor 701 and the storage medium 702 communicate through the bus 703, and the processor 701 executes the machine-readable instructions to perform the steps of the above-mentioned three-dimensional reconstruction method. The specific implementation method and technical effect are similar and will not be repeated here.

[0126] Optionally, the present application further provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above three-dimensional reconstruction method are executed.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0130] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0131] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A three-dimensional reconstruction method, It is characterized in that The method comprises: Taking out a seed patch from a patch stack of the target object, wherein the patch stack includes: a plurality of seed patches; If the extracted seed patch does not construct a neighboring patch, construct a neighboring patch of the extracted seed patch according to the extracted seed patch; Determining a consistency constraint coefficient corresponding to the continuous judgment result according to the continuous judgment result of the extracted seed face patch and the neighboring face patch; According to the normal vector freedom degree of the neighborhood face patch and the consistency constraint coefficient, the neighborhood face patch is adjusted by using a preset main constraint function so that the main constraint function value is minimized; wherein the main constraint function includes: a first correlation coefficient and a consistency constraint function, wherein the first correlation coefficient is a correlation coefficient of the adjusted neighborhood face patch projected onto a plurality of image projection windows, and the consistency constraint function is used to constrain the normal vector freedom degree in combination with the consistency constraint coefficient; Add the adjusted neighborhood patch corresponding to the minimum principal constraint function value as a new seed patch to the patch stack, and repeat the process until all seed patches in the patch stack are taken out and all the taken out seed patches have corresponding neighborhood patches constructed thereon; The target object is three-dimensionally reconstructed according to all adjusted neighborhood patches.

2. The method according to claim 1, It is characterized in that The determining, according to the continuous judgment result of the extracted seed patch and the neighboring patch, a consistency constraint coefficient corresponding to the continuous judgment result includes: If the continuity judgment result is that the extracted seed patch is continuous with the neighboring patch, then 1 is determined as the consistency constraint coefficient; If the continuity judgment result is that the extracted seed patch is discontinuous with the neighboring patch, 0 is determined as the consistency constraint coefficient.

3. The method according to claim 2, It is characterized in that Before determining the consistency constraint coefficient corresponding to the continuous judgment result between the extracted seed patch and the neighboring patch, the method further includes: Calculating a second correlation coefficient between a first projection window and a second projection window, wherein the first projection window is a projection window of the extracted seed patch on the corresponding reference image, and the second projection window is a projection window of the neighborhood patch on the corresponding reference image; If the second correlation coefficient is greater than or equal to a second preset threshold, determining the continuity judgment result is: the extracted seed patch is continuous with the neighboring patch; If the second correlation coefficient is less than the second preset threshold, the continuity judgment result is determined as: the extracted seed patch is discontinuous with the neighborhood patch.

4. The method according to claim 1, It is characterized in that The step of adjusting the neighborhood facets according to the degree of freedom of the normal vector of the neighborhood facets and the consistency constraint coefficient by using a preset main constraint function to minimize the main constraint function value includes: According to the normal vector degree of freedom of the neighborhood face patch and the consistency constraint coefficient, the main constraint function is used to calculate the main constraint function value of the neighborhood face patch; The positional degrees of freedom and normal vector degrees of freedom of the neighborhood facet are adjusted until the main constraint function value of the adjusted neighborhood facet is minimized; when the main constraint function value is minimized, the first correlation coefficient of the projection window of the adjusted neighborhood facet projected onto the multiple images is optimal.

5. The method according to claim 1, It is characterized in that Before taking out a seed patch from the patch stack of the target object, the method further includes: Performing feature matching on multiple images of the target object to obtain multiple initial seed patches; The patch stack including the plurality of initial seed patches is created.

6. The method according to claim 1, It is characterized in that The plurality of seed patches in the patch stack are sorted according to the confidence level; The extracting a seed patch from the patch stack of the target object includes: A seed dough piece with the highest confidence is selected from the dough piece stack as the removed seed dough piece.

7. The method according to claim 1, It is characterized in that The method of adding the adjusted neighborhood patch corresponding to the minimum principal constraint function value as a new seed patch to the patch stack comprises: Determine whether a first correlation coefficient corresponding to when the primary constraint function value reaches a minimum is greater than or equal to a first preset threshold; If the first correlation coefficient is greater than or equal to the first preset threshold, the adjusted neighborhood patch corresponding to the first correlation coefficient is added to the patch stack as a new seed patch.

8. A three-dimensional reconstruction device, It is characterized in that The device comprises: An acquisition module is used to take out a seed patch from a patch stack of a target object, wherein the patch stack includes: a plurality of seed patches; A construction module, configured to construct a neighboring patch of the extracted seed patch according to the extracted seed patch if the neighboring patch is not constructed by the extracted seed patch; A determination module, configured to determine a consistency constraint coefficient corresponding to a continuous judgment result of the extracted seed face patch and the neighboring face patch according to the continuous judgment result of the extracted seed face patch and the neighboring face patch; An adjustment module, used to adjust the neighborhood face patch according to the normal vector freedom degree of the neighborhood face patch and the consistency constraint coefficient, using a preset main constraint function to minimize the main constraint function value; wherein the main constraint function includes: a consistency constraint function, and the consistency constraint function is used to constrain the normal vector freedom degree in combination with the consistency constraint coefficient; An adding module is used to add the adjusted neighborhood patch corresponding to the minimum principal constraint function value as a new seed patch to the patch stack, and repeat the process until all seed patches in the patch stack are taken out and all the taken out seed patches are constructed with corresponding neighborhood patches; The reconstruction module is used to perform three-dimensional reconstruction of the target object according to all adjusted neighborhood patches.

9. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the three-dimensional reconstruction method as described in any one of claims 1-7.

10. A storage medium, It is characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the three-dimensional reconstruction method according to any one of claims 1 to 7 are executed.

Images