Method and device for global dense matching of multi-view images of target objects for three-dimensional reconstruction
By constructing global energy functions and semantic constraints, high-precision three-dimensional reconstruction of complex objects in multi-view images is achieved, solving the problems of mismatch and incomplete reconstruction in traditional methods, and improving the degree of automation and the accuracy of the three-dimensional model.
Patent Information
- Application Number
- CN202111396551.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-23
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-11-23
AI Technical Summary
Traditional multi-view image intensive matching methods are difficult to reconstruct a three-dimensional model of non-Lambertian bodies. Especially when there are weak textures and repeated texture areas on the surface of complex objects, the obtained point clouds have errors, low accuracy and incompleteness.
By constructing a global energy function, the image is intensively matched by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by pixel by , generate three-dimensional point clouds and build a three-dimensional model of the target object to realize semantic segmentation and reconstruction.
It improves the accuracy and automation of three-dimensional reconstruction, can automatically eliminate background point clouds, directly output the target's three-dimensional model, and obtain accurate model semantic information, solving the problems of mismatch and incomplete reconstruction in traditional methods.
Smart Images

Figure CN113902875B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of three-dimensional reconstruction. Specifically, embodiments of the present application relate to a method and device for three-dimensional reconstruction of a target object by global dense matching of multiple-view images. Background Art
[0002] The three-dimensional reconstruction of the target object based on multi-view stereo images is to restore the three-dimensional model of the object from multiple overlapping images. This requires dense matching of each pixel between the multi-view images after obtaining the precise internal and external orientation elements of the image through the bundle method regional network adjustment, and generating a high-precision three-dimensional point cloud. Dense matching is a key technical link in three-dimensional reconstruction. It can restore the depth information of each pixel and is an energy optimization process. The correctness and reliability of the dense matching results are directly related to the quality of the three-dimensional reconstruction of the target object.
[0003] Traditional multi-view image dense matching and 3D reconstruction methods have difficulty reconstructing 3D models of non-Lambertian bodies (i.e., incident light has different reflections on the surface of the object, is not diffuse, and does not satisfy the image consistency constraint). On the surface of complex objects, there are a large number of weak texture areas, repeated texture areas and other difficult matching areas. The point cloud obtained by traditional dense matching methods has a large number of mismatched points, low accuracy and incomplete areas.
[0004] Traditional non-global optimized dense matching methods can only achieve geometric modeling of the target, resulting in the current matching still remaining at the low-level visual processing level. Therefore, how to improve the matching effect and ultimately improve the accuracy of the three-dimensional reconstruction results has become a technical problem that needs to be solved urgently. Summary of the invention
[0005] The purpose of the embodiments of the present application is to provide a three-dimensional reconstruction and device for multi-view images of a target object with global dense matching. Through some embodiments of the present application, the background point cloud can be automatically eliminated, the three-dimensional model of the target can be directly output, and accurate model semantic information can be obtained, which can avoid subsequent manual interpretation of the reconstructed model and improve the degree of automation of the model's three-dimensional reconstruction.
[0006] In a first aspect, some embodiments of the present application provide a method for global dense matching of multi-view images of a target object to reconstruct three-dimensional (3D) point. The method comprises: constructing a global energy function, wherein the global energy function is related to at least one of a priori model constraint of the target object, consistency constraint of object points, depth smoothness constraint of adjacent object points, classification cost of object points, category consistency constraint of adjacent object points, and depth consistency constraint of adjacent object points; performing pixel-by-pixel dense matching of images according to the global energy function to generate a matching three-dimensional point cloud, and constructing a three-dimensional model of the target object according to the matching three-dimensional point cloud to complete the reconstruction of the three-dimensional model of the target object.
[0007] Some embodiments of the present application can complete the reconstruction of the three-dimensional model by constructing a global energy function including various constraints and consistencies, at least in combination with the target category, thereby improving the accuracy of the three-dimensional model construction while also improving the degree of automation.
[0008] In some embodiments, the three-dimensional model reconstruction of the target object is completed by performing dense pixel-by-pixel matching of the image according to the global energy function to generate a matching three-dimensional point cloud and constructing a three-dimensional model of the target object according to the matching three-dimensional point cloud, including: obtaining the optimal solution of the target surface model and the target category according to the global energy function, and simultaneously achieving semantic segmentation and three-dimensional reconstruction of the target object.
[0009] Some embodiments of the present application can simultaneously obtain semantic segmentation and three-dimensional reconstruction of the target object by solving the global energy function, thereby meeting the data requirements of more occasions and improving the versatility of the technical solution.
[0010] In some embodiments, before constructing the global energy function, the three-dimensional reconstruction method also includes: obtaining an initial three-dimensional surface model of the target object; aligning the acquired prior model of the target object with the initial three-dimensional surface model to obtain a registered three-dimensional surface model; defining a depth set and a classification directory of the point cloud according to the registered three-dimensional surface model; and obtaining, according to the depth set and the classification directory, the consistency constraints of the object points in the object space, the depth smoothness constraints of the adjacent object points, the image classification costs of the object points, the category consistency constraints of the object points, the depth consistency constraints of the adjacent object points, and the gradients of the object points on the reconstructed model of the target object.
[0011] Some embodiments of the present application provide a method for obtaining a depth set and a classification directory, and obtaining parameters of constraint conditions and consistency conditions based on these variables, so that some embodiments of the present application can construct a better global energy function.
[0012] In some embodiments, constructing the global energy function includes: taking the depth and category of each point in the object space as variables, and jointly constructing the global energy function according to the prior model constraints of the target object, the consistency constraints of the object points, the depth smoothness constraints of the adjacent object points, the classification costs of the object points, the category consistency constraints of the object points and the depth consistency constraints of the adjacent object points.
[0013] Some embodiments of the present application construct a global energy function by determining two types of variables and multiple types of constraints, etc., which is enough to unify the three-dimensional reconstruction and pattern recognition problems. That is, mathematically, three-dimensional reconstruction and pattern recognition are both labelling problems, that is, a label is assigned to each point to satisfy the minimum global energy function. For three-dimensional reconstruction applications, the label is the depth of each point; and for target recognition, the label is the type of each point (such as target object and background). By unifying the two, the optimal three-dimensional reconstruction results of cultural relics and target recognition results can be obtained at the same time.
[0014] In some embodiments, the calculation formula of the global energy function is:
[0015] E(D,L)=E unary_d (D)+E binary_d (D)+E unary_l (L)+E binary_l (L)
[0016] +E c (D,L)+E priori (G(D))
[0017] Among them, E unary_d (D) is a data item with the depth set D as an unknown number; E unary_l (L) is a data item with the category set L as an unknown number; E binary_d (D) represents a smoothing term with the depth set D as an unknown number; E binary_l (L) represents a smoothing term with the category set L as an unknown number; E c (D,L) represents the interaction term between unknowns D and L; E priori (G(D)) represents the depth gradient constraint term calculated according to the prior model.
[0018] Some embodiments of the present application provide a formula for a global energy function, which makes the calculation of the value of the global energy function more objective and accurate.
[0019] In some embodiments, the depth set and classification directory of the point cloud are defined based on the registered three-dimensional surface model, including: in the initial point cloud in the object space, two label labels are assigned to each object point, namely depth and category, wherein the depth represents the geometric shape of the model, and the category represents the type of model, D is defined as the depth set of all points, and L is defined as the category set of all points.
[0020] Some embodiments of the present application provide a method for acquiring a depth set and a category set.
[0021] In some implementations, the prior model is obtained by the following principle: based on the fact that the surfaces of objects of the same type are continuous and smooth, the depth information is continuous and smooth, and there is a sudden change in depth at the junction between objects of different types, the prior model of the target object is constructed.
[0022] Some embodiments of the present application construct a priori model of the target object through characteristic attributes of the surface of objects of the same type and mutation characteristics between objects of different types, so that the obtained priori model can more accurately reflect the commonalities of objects of the same type and the differences between different objects.
[0023] In a second aspect, some embodiments of the present application provide a three-dimensional reconstruction device for global dense matching of multi-view images of a target object, wherein the multi-view image global dense matching three-dimensional reconstruction device includes: a global energy function construction module, configured to construct a global energy function, wherein the global energy function is related to at least one of a priori model constraints of the target object, consistency constraints of object points, depth smoothness constraints of adjacent object points, category consistency constraints of adjacent object points, and depth consistency constraints of adjacent object points; a three-dimensional model reconstruction module, configured to perform pixel-by-pixel dense matching of images according to the global energy function to generate a matching three-dimensional point cloud, and to construct a three-dimensional model of the target object according to the matching three-dimensional point cloud, thereby completing the reconstruction of the three-dimensional model of the target object.
[0024] In a third aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the method described in any embodiment of the first aspect above can be implemented.
[0025] In a fourth aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the method described in any embodiment of the first aspect when executing the program. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0027] Figure 1 One of the flow charts of the three-dimensional reconstruction method of the target object provided in the embodiment of the present application;
[0028] Figure 2 The second flowchart of the method for three-dimensional reconstruction of a target object provided in an embodiment of the present application;
[0029] Figure 3 A block diagram of the composition of a three-dimensional reconstruction device for a target object provided in an embodiment of the present application;
[0030] Figure 4 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0031] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0032] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0033] As described in the background technology section, the traditional non-global optimized dense matching method can only achieve geometric modeling of the target, which belongs to the low-level visual processing level. Therefore, the relevant technology requires manual judgment, analysis and induction of the target category after reconstruction, which is not conducive to the automatic processing of target object reconstruction applications. Unlike the prior art, the present application also identifies the category of the target during matching, and the matching process will also be improved due to the target category.
[0034] For example, in some embodiments of the present application, a method and device for three-dimensional reconstruction of multi-view images of a target object by global dense matching is provided. The method adds more robust semantic information constraints (such as target categories, prior models, etc.) to the dense matching of multi-view images to obtain a three-dimensional point cloud, and realizes accurate and complete three-dimensional reconstruction based on the three-dimensional point cloud. For example, according to the global energy function provided in some embodiments of the present application, dense image matching is performed to generate a three-dimensional point cloud, and then the three-dimensional point cloud is filtered, gridded, and edited to generate a digital surface model. Through deep learning and semantic analysis, the model depth and model category are respectively used as labels to be solved, thereby converting the semantic segmentation and three-dimensional reconstruction problems into labelling problems. The algorithm uses two labels as unknowns at the same time, and designs a global energy function according to image consistency constraints, depth smoothing constraints, deep learning constraints, and prior model constraints. Its optimal solution is the result of semantic segmentation and model three-dimensional reconstruction. It should be noted that the optimal solution is the optimal matching result of each point on the stereo image during the dense image matching process, and on this basis, an object point in the three-dimensional point cloud is generated by forward intersection. Through this process, a dense object three-dimensional point cloud can be obtained, which is the result of model three-dimensional reconstruction. The advantage of some embodiments of the present application is that traditional methods can only obtain three-dimensional reconstruction results, while some embodiments of the present application also optimize the classification results of the image (ie, semantic segmentation results).
[0035] Please see Figure 1 , Figure 1 A three-dimensional reconstruction method of a target object is provided for some embodiments of the present application, and the three-dimensional reconstruction method includes: S101, constructing a global energy function, wherein the global energy function is related to at least one of the prior model constraints of the target object, the consistency constraints of the object points, the depth smoothness constraints of the adjacent object points, the classification costs of the object points, the category consistency constraints of the adjacent object points, and the depth consistency constraints of the adjacent object points. And S102, densely matching the image pixel by pixel according to the global energy function to generate a matching three-dimensional point cloud, and constructing the three-dimensional model of the target object according to the matching three-dimensional point cloud to complete the reconstruction of the three-dimensional model of the target object. For example, S102 includes: based on the global energy function, densely matching the image pixel by pixel to generate a correctly matched three-dimensional point cloud (i.e., obtaining a matching three-dimensional point cloud), and then filtering and gridding the point cloud (i.e., the matching three-dimensional point cloud) to construct a three-dimensional model of the target object. This process is the process of three-dimensional reconstruction of stereo images. Some embodiments of the present application can complete the reconstruction of the three-dimensional model by constructing a global energy function including various constraints and consistencies, at least in combination with the target category, thereby improving the accuracy of the three-dimensional model construction while also improving the degree of automation.
[0036] It should be noted that, in some embodiments of the present application, before executing S101, the three-dimensional reconstruction method also includes: obtaining an initial three-dimensional surface model of the target object; aligning the acquired prior model of the target object and the initial three-dimensional surface model to obtain a registered three-dimensional surface model; defining a depth set and a classification directory of the point cloud according to the registered three-dimensional surface model; and obtaining, according to the depth set and the classification directory, the consistency constraints of the object points in the object space, the depth smoothness constraints of the adjacent object points, the image classification costs of the object points, the category consistency constraints of the object points, and the depth consistency constraints of the adjacent object points.
[0037] The above process is explained below as an example.
[0038] In some embodiments, the construction of the global energy function includes: taking the depth and category of each point in the object space as variables, and jointly constructing the global energy function according to the prior model constraints of the target object, the consistency constraints of the object points, the depth smoothness constraints of the adjacent object points, the classification costs of the object points, the category consistency constraints of the object points, and the depth consistency constraints of the adjacent object points. For example, in some embodiments of the present application, the depth set and classification directory of the point cloud defined according to the registered three-dimensional surface model include: in the initial point cloud of the object space, assigning 2 label labels to each object point, namely depth and category, wherein the depth represents the geometric shape of the model, the category represents the type to which the model belongs, defining D as the depth set of all points, and defining L as the category set of all points.
[0039] In some implementations of the present application, the prior model is obtained through the following principle: based on the fact that the surfaces of objects of the same type are continuous and smooth, the depth information is continuous and smooth, and there is a sudden change in depth at the junction between objects of different types, the prior model of the target object is constructed.
[0040] In some embodiments of the present application, generating a three-dimensional model of the target object according to the global energy function includes: obtaining an optimal solution of the target surface model and the target category according to the global energy function, and simultaneously achieving semantic segmentation and three-dimensional reconstruction of the target object.
[0041] Some embodiments of the present application can simultaneously obtain semantic segmentation and three-dimensional reconstruction of the target object by solving the global energy function, thereby meeting the data requirements of more occasions and improving the versatility of the technical solution.
[0042] Combine the following Figure 2 The three-dimensional reconstruction method of the target object in some embodiments of the present application is exemplified.
[0043] S201, start.
[0044] S202, obtaining a stereoscopic image pair of the target object. It is understandable that a stereoscopic image pair is two or more optical images with a certain degree of overlap obtained from different viewing angles for the same target object, wherein every two images with a certain degree of overlap are called a stereoscopic image pair.
[0045] S203: Acquire an initial surface of the target object according to the obtained stereo image pair.
[0046] For example, a conventional multi-view dense matching algorithm is used to quickly obtain the initial surface of the target object model. In this embodiment, a patch-based 3D multi-view stereoscopic vision algorithm (PMVS, Patch-based Multi View System) is used. That is, feature matching is initialized, patches are generated and screened, and extended patches are initialized using adjacent patches. Outlier vertices are detected and removed according to the distance between vertex neighbors to obtain a 3D model of the target object surface.
[0047] S204, obtaining a priori model of the target object.
[0048] For example, based on the fact that the surfaces of objects of the same type are often continuous and smooth, their depth information should also be continuous and smooth, while the junctions between objects of different types are often accompanied by depth mutations, a priori model of the target object is constructed.
[0049] S205, three-dimensional geometric registration of the model.
[0050] The prior model and the initial matching model obtained in S203 are subjected to three-dimensional geometric registration to achieve a one-to-one correspondence between the prior model and the initial matching model.
[0051] S206, define the depth set and classification directory of the point cloud.
[0052] For example, in the initial point cloud of the object space, two labels are assigned to each object point, namely depth and category. Depth represents the geometric shape of the model, and category represents the type of the model. Define D as the depth set of all points. Define L as the category set of all points.
[0053] The target objects to be reconstructed are classified into a classification directory S, namely:
[0054] S = {background, category 1, category 2, category 3, ...}
[0055] S207, calculating the consistency constraints of the object points.
[0056] For example, calculate the consistency constraint E of the object point p in multi-view images unary_d (D), D represents the depth set of all points, that is, for the spatial domain Ω, D = {d p ,p∈Ω}.
[0057] E unary_d (D) is expressed as:
[0058]
[0059] Among them, Cost(p,d p ) indicates that the object point p is at the corresponding depth d p The multi-view matching cost under .
[0060] S208, calculating the depth smoothing constraints of adjacent object points.
[0061] For example, the depth of adjacent object points p and q should satisfy the smoothness constraint E binary_d (D), namely:
[0062]
[0063] Where N represents a set of adjacent pixels; T[] represents a Boolean operator; P1 and P2 represent penalty terms, generally P1 <P2。
[0064] S209, calculating the image classification cost of the object point.
[0065] For example, calculate the cost E of each object point p corresponding to the classification lp in the visual image unary_d (L). L represents the classification set of all points, that is, L = {l p ,l p ∈S&p∈Ω}. The probability lp of each point p corresponding to a category in the visual image can be calculated through deep learning.
[0066] E unary_l (L) is expressed as a negative number of probability, that is:
[0067]
[0068] Where V(p) represents the set of visible images of object point p; i represents the visible image; The category of the object point p on the visible image i. Indicates that point p belongs to the category probability.
[0069] S210, calculating the category consistency constraints of adjacent object points.
[0070] For example, calculate the category consistency constraint E of adjacent object points p, q binary_l (L),
[0071]
[0072] Among them, P1' represents the penalty term, which penalizes adjacent points with inconsistent categories.
[0073] S211, calculating the depth consistency constraints of adjacent object points.
[0074] For example, when adjacent object points p and q belong to the same category, their depths should satisfy the depth consistency constraint E as close as possible. c (D,L):
[0075]
[0076] Among them, P2' represents the penalty term, which is used to punish when l p = l q When the depth changes suddenly.
[0077] S212, calculating the gradient E of the object point on the target object reconstruction model priori , the gradient should be close to the gradient of the object point on the prior model to ensure that the shape of the reconstructed model is close to the shape of the prior model.
[0078] Let G(D) represent the set of gradients of each object point, then E priori The calculation method is:
[0079]
[0080] Among them, g(d p ) represents the depth gradient on the surface model corresponding to the object point p at depth dp; g priori (p) represents the depth gradient of the object point p on the prior model; P” represents the penalty term.
[0081] S213, constructing a global energy function according to the results of the above steps.
[0082] Based on S207 to S212, a global energy function E(D, L) is constructed.
[0083] E(D,L)=E unary_d (D)+E binary_d (D)+E unary_l (L)+E binary_l (L)+E c (D,L)+E priori (G(D))
[0084] Among them, E unary_d (D) is a data item with the depth set D as an unknown number; E unary_l (L) is a data item with the category set L as the unknown number; E binary_d (D) represents the smoothing term with the depth set D as the unknown; Ebinary_l (L) represents the smoothing term with the category set L as the unknown number; E c (D,L) represents the interaction term between unknowns D and L; E priori (G(D)) represents the depth gradient constraint term calculated according to the prior model.
[0085] S214, solving the global energy function.
[0086] The global energy function in S213 is solved to simultaneously obtain the optimal geometric model and the optimal target recognition result. In this embodiment, a volumetric graph cuts method is used.
[0087] S215, generating a precise and complete three-dimensional model of the target object, that is, finally outputting a precise and complete three-dimensional model of the target object.
[0088] It should be noted that S203 can use a variety of fast dense matching methods to obtain the initial three-dimensional surface model of the target object. S205 can use a variety of three-dimensional registration methods to achieve accurate registration of the object prior model and the initial matching model obtained in step A. S213 can use a variety of global energy function solution methods to obtain the optimal dense matching and point cloud model three-dimensional reconstruction results.
[0089] It is understandable that some embodiments of the present application have at least the following beneficial effects:
[0090] First, considering that objects belonging to the same category have similar geometric appearances, some embodiments of the present application can use the prior model corresponding to each category of objects as a constraint condition, use the target type information to improve the 3D reconstruction results, and guide the target recognition process. This method can solve the mismatch problem of non-Lambertian 3D reconstruction, solve the 3D enhancement problem of missing parts of the surface of complex objects, and finally obtain a complete and accurate 3D model, and ensure that the surface of the 3D model is continuous and smooth.
[0091] Secondly, some embodiments of the present application can automatically remove background point clouds, directly output the three-dimensional model of the target, and obtain accurate model semantic information, which can avoid subsequent manual interpretation of the reconstructed model and improve the automation level of three-dimensional reconstruction of the model;
[0092] Finally, some embodiments of the present application can unify the three-dimensional reconstruction and pattern recognition problems by constructing a global energy function, that is, mathematically, three-dimensional reconstruction and pattern recognition are both labelling problems, that is, a label is assigned to each point to satisfy the minimum global energy function. For three-dimensional reconstruction applications, the label is the depth of each point; and for target recognition, the label is the type of each point (such as: target object and background). By unifying the two, the optimal three-dimensional reconstruction results of cultural relics and target recognition results can be obtained at the same time.
[0093] That is to say, some embodiments of the present application provide a "global dense matching and point cloud three-dimensional reconstruction method based on semantic information", which includes: marking each pixel, improving the dense matching result according to the semantic information constraint, and improving the semantic segmentation result according to the dense matching disparity constraint. At the same time, the optimal solution of the target surface model and the target category is obtained, and the semantic segmentation and three-dimensional reconstruction of the target object are realized synchronously. Taking the depth and category of each point as variables, according to the prior model constraints of the target object, the consistency constraints of the object point, the depth smoothing constraints of the adjacent object points, the classification cost of the object point, the category consistency constraints of the object point, and the depth consistency constraints of the adjacent object points, an optimized dense matching global energy function is jointly constructed to ensure the quality of the generated point cloud data.
[0094] Please refer to Figure 3 , Figure 3 The 3D reconstruction device of the target object provided in the embodiment of the present application is shown. It should be understood that the device is similar to the above Figure 1 or Figure 2 The method embodiment corresponds to the method embodiment and can execute each step involved in the above method embodiment. The specific functions of the device can be referred to in the above description. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system of the device. The three-dimensional reconstruction device of the target object includes: a global energy function construction module 101 and a three-dimensional model reconstruction module 102.
[0095] The global energy function construction module 101 is configured to construct a global energy function, wherein the global energy function is related to at least one of the prior model constraints of the target object, the consistency constraints of the object points, the depth smoothness constraints of adjacent object points, the category consistency constraints of adjacent object points, and the depth consistency constraints of adjacent object points.
[0096] The 3D model reconstruction module 102 is configured to perform pixel-by-pixel dense matching of images according to the global energy function to generate a matching 3D point cloud, and to construct a 3D model of the target object according to the matching 3D point cloud to complete the 3D model reconstruction of the target object.
[0097] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.
[0098] Some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which can implement the above-mentioned Figure 2 or Figure 3 The technical solution includes the method described in any embodiment.
[0099] like Figure 4 As shown, some embodiments of the present application provide an electronic device 500, which includes a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520, wherein the processor 520 reads the program from the memory 510 through a bus 530 and executes the program to implement the above Figure 2 or Figure 3 The method of any embodiment.
[0100] Processor 520 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 520 can be a microprocessor.
[0101] The memory 510 may be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code to implement some or all functions of one or more modules described in the embodiments of the present application. The processor 520 of the present disclosure embodiment may be used to execute the instructions in the memory 510 to implement Figure 2 The memory 510 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory or other memory known to those skilled in the art.
[0102] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0103] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0104] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0105] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.
[0106] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0107] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
Claims
1. A method for global dense matching of multi-view images of a target object for three-dimensional reconstruction, characterized in that: The multi-view image global dense matching 3D reconstruction method comprises: Constructing a global energy function, the construction of the global energy function includes: taking the depth and category of each point in the object space as variables, and jointly constructing the global energy function according to the prior model constraints of the target object, the consistency constraints of the object points, the depth smoothness constraints of the adjacent object points, the image classification costs of the object points, the category consistency constraints of the adjacent object points and the depth consistency constraints of the adjacent object points; the calculation formula of the global energy function is: E(D,L)=E unary_d (D)+E binary_d (D)+E unary_l (L)+E binary_l (L)+E c (D,L)+E priori (G(D)) Among them, E unary_d (D) is a data item with the depth set D as an unknown number; E unary_l (L) is a data item with the category set L as the unknown number; E binary_d (D) represents a smoothing term with the depth set D as an unknown number; E binary_l (L) represents a smoothing term with the category set L as an unknown number; E c (D,L) represents the interaction term between unknowns D and L; E priori (G(D)) represents the depth gradient constraint term calculated according to the prior model; The prior model is obtained by the following principle: based on the fact that the surface of objects of the same type is continuous and smooth, the depth information is continuous and smooth, and there is a sudden change in depth at the junction between objects of different types, the prior model of the target object is constructed; A matching three-dimensional point cloud is generated by solving the global energy function to perform dense pixel-by-pixel matching of the image, and a three-dimensional model of the target object is constructed based on the matching three-dimensional point cloud to complete the three-dimensional model reconstruction of the target object, wherein the solution of the global energy function includes an optimal semantic segmentation result and an optimal reconstruction result of the three-dimensional model.
2. The method for 3D reconstruction based on global dense matching of multi-view images as claimed in claim 1, characterized in that: The generating a matching three-dimensional point cloud by performing dense pixel-by-pixel matching of the image by solving the global energy function, and constructing a three-dimensional model of the target object according to the matching three-dimensional point cloud, comprises: The optimal solution of the target surface model and the target category is obtained according to the global energy function, and the semantic segmentation and three-dimensional reconstruction of the target object are simultaneously achieved.
3. The method for 3D reconstruction based on global dense matching of multi-view images as claimed in claim 1, characterized in that: Before constructing the global energy function, the three-dimensional reconstruction method further includes: Acquire an initial three-dimensional surface model of the target object; Registering the acquired prior model of the target object with the initial three-dimensional surface model to obtain a registered three-dimensional surface model; Defining a deep set and a classification catalog of point clouds based on the registered three-dimensional surface model; According to the depth set and the classification catalog, the consistency constraint of the object point in the object space, the depth smoothness constraint of the adjacent object point, the image classification cost of the object point, the category consistency constraint of the adjacent object point, the depth consistency constraint of the adjacent object point and the gradient of the object point on the target object reconstruction model are obtained, and the classification cost of the object point includes the image classification cost of the object point.
4. The method for 3D reconstruction based on global dense matching of multi-view images as claimed in claim 3, characterized in that: The method of defining a depth set and a classification catalog of a point cloud based on the registered three-dimensional surface model includes: in the initial point cloud in the object space, assigning two label labels to each object point, namely, depth and category, wherein the depth represents the geometric shape of the model, and the category represents the type of the model, defining D as the depth set of all points, and defining L as the category set of all points.
5. A device for global dense matching of multi-view images of a target object for three-dimensional reconstruction, characterized in that: The multi-view image global dense matching 3D reconstruction device comprises: The global energy function construction module is configured to construct a global energy function, wherein the construction of the global energy function includes: taking the depth and category of each point in the object space as variables, and constructing the global energy function according to the prior model constraints of the target object, the consistency constraints of the object points, the depth smoothness constraints of the adjacent object points, the image classification costs of the object points, the category consistency constraints of the adjacent object points, and the depth consistency constraints of the adjacent object points; the calculation formula of the global energy function is: E(D,L)=E unary_d (D)+E binary_d (D)+E unary_l (L)+E binary_l (L)+E c (D,L)+E priori (G(D)) Among them, E unary_d (D) is a data item with the depth set D as an unknown number; E unary_l (L) is a data item with the category set L as the unknown number; E binary_d (D) represents a smoothing term with the depth set D as an unknown number; E binary_l (L) represents a smoothing term with the category set L as an unknown number; E c (D,L) represents the interaction term between unknowns D and L; E priori (G(D)) represents the depth gradient constraint term calculated according to the prior model; The three-dimensional model reconstruction module is configured to generate a matching three-dimensional point cloud by performing dense pixel-by-pixel matching of the image by solving the global energy function, and to construct a three-dimensional model of the target object based on the matching three-dimensional point cloud to complete the three-dimensional model reconstruction of the target object, wherein the solution of the global energy function includes an optimal semantic segmentation result and an optimal reconstruction result of the three-dimensional model.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 4 can be implemented.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method described in any one of claims 1 to 4 can be implemented.
