Image-based three-dimensional model reconstruction method and system, and storage medium

Abstract

The invention provides an image-based three-dimensional model reconstruction method and system and a storage medium, and the method comprises the steps: carrying out the photographing of a to-be-reconstructed target through a camera, and obtaining at least two multi-view images of the to-be-reconstructed target; performing feature extraction according to each multi-view image to obtain feature points of each multi-view image; matching the feature points of the at least two multi-view images to obtain multiple groups of feature point pairs; determining the position of each target feature point according to the position of each group of feature point pairs and the parameters of the camera; and performing three-dimensional model reconstruction on the to-be-reconstructed target according to the position of each target feature point. According to the method, feature matching is performed according to the multi-view images, and model reconstruction is performed in combination with the positions of the feature point pairs and the camera parameters, so that the three-dimensional model can be accurately reconstructed at low cost.

Description

Technical Field

[0001] The present application belongs to the field of model reconstruction, and in particular to an image-based three-dimensional model reconstruction method, system and storage medium. Background Art

[0002] At present, 3D reconstruction technology can achieve high-precision and high-efficiency 3D model reconstruction in many fields, including urban planning, historical heritage protection, virtual reality (VR), augmented reality (AR) and medical imaging. In recent years, due to the development of science and technology, the demand for 3D model reconstruction in various fields has become increasingly strong.

[0003] As the name suggests, 3D model reconstruction is to reconstruct a realistic 3D model of an object from point clouds, pictures, 3D contour lines and other data. During the reconstruction process, different processing algorithms will be used for 3D reconstruction of different data.

[0004] There are good research results in the existing technology on the rapid reconstruction of three-dimensional shapes based on images, but these are only laboratory research results and cannot be commercialized at present. Other three-dimensional model reconstruction technologies are costly and have a long service life. They cannot save costs while ensuring accurate reconstruction of three-dimensional models. Therefore, the present application provides an image-based three-dimensional model reconstruction method, which performs feature matching according to multi-view images and reconstructs the model based on the position of feature point pairs and camera parameters, which is conducive to low-cost and accurate reconstruction of three-dimensional models. Summary of the invention

[0005] Embodiments of the present application provide an image-based three-dimensional model reconstruction method, system, and storage medium.

[0006] In a first aspect, an embodiment of the present application provides a method for reconstructing a three-dimensional model based on an image, the method comprising:

[0007] Shooting the target to be reconstructed by a camera to obtain at least two multi-view images of the target to be reconstructed;

[0008] Perform feature extraction according to each of the multi-view images to obtain feature points of each of the multi-view images;

[0009] Matching feature points of at least two of the multi-view images to obtain multiple groups of feature point pairs, wherein a group of feature point pairs is the same target feature point on the target to be reconstructed, and the feature point positions of the target feature points in different multi-view images are different;

[0010] Determining the position of each target feature point according to the position of each group of feature point pairs and the parameters of the camera;

[0011] A three-dimensional model is reconstructed for the target to be reconstructed according to the position of each target feature point.

[0012] Optionally, before photographing the target to be reconstructed by a camera to obtain at least two multi-view images of the target to be reconstructed, the method further includes:

[0013] The camera is calibrated in each azimuth viewing angle of the target to be reconstructed to obtain parameters of the camera, wherein the parameters of the camera include internal and external parameters of the camera.

[0014] Optionally, photographing the target to be reconstructed by a camera to obtain at least two multi-view images of the target to be reconstructed includes:

[0015] The camera is used to shoot at various azimuth viewing angles of the target to be reconstructed to obtain at least two two-dimensional multi-view images.

[0016] Optionally, performing feature extraction according to each of the multi-view images to obtain feature points of each of the multi-view images includes:

[0017] Performing feature detection according to the pixel points of each of the multi-view images to obtain pixel point information of each of the multi-view images;

[0018] Feature extraction is performed according to the pixel point information of each of the multi-view images and preset feature parameters to obtain feature points of each of the multi-view images.

[0019] Optionally, the multi-view images have the same size, a coordinate system is established with the center of each multi-view image, and the coordinates of feature points of each multi-view image are obtained, the feature points carry feature point information, and the feature point information includes feature point color, and the feature points of at least two multi-view images are matched to obtain multiple groups of feature point pairs, including:

[0020] Feature points of at least two of the multi-view images are matched according to the feature point information, and feature points with consistent feature point information are taken as a group of feature point pairs to obtain multiple groups of feature point pairs, wherein the feature points with consistent feature point information are the same target feature point.

[0021] Optionally, determining the position of each target feature point according to the position of each group of feature point pairs and the parameters of the camera includes:

[0022] A triangulation calculation is performed according to the position of each group of feature point pairs and the parameters of the camera to determine the position of each target feature point.

[0023] Optionally, before matching the feature points of at least two of the multi-view images to obtain a plurality of groups of feature point pairs, the method further includes:

[0024] Corresponding epipolar lines in at least two of the multi-view images are adjusted to be in a horizontal state and on the same horizontal scanning line, and feature points of the multi-view images are corrected.

[0025] In a second aspect, an embodiment of the present application provides an image-based three-dimensional model reconstruction system, the system comprising:

[0026] An image acquisition module, used to photograph a target to be reconstructed through a camera to acquire at least two multi-view images of the target to be reconstructed;

[0027] A feature extraction module, used to extract features from each of the multi-view images to obtain feature points of each of the multi-view images;

[0028] A feature matching module, used for matching feature points of at least two of the multi-view images to obtain multiple groups of feature point pairs, wherein a group of feature point pairs is the same target feature point on the target to be reconstructed, and the feature point positions of the target feature points in different multi-view images are different;

[0029] A target feature point determination module, used to determine the position of each target feature point according to the position of each group of feature point pairs and the parameters of the camera;

[0030] The model reconstruction module is used to reconstruct a three-dimensional model of the target to be reconstructed according to the position of each target feature point.

[0031] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the computer program.

[0032] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method as described above is implemented.

[0033] The present application performs feature matching based on multi-view images and reconstructs the model in combination with the position of feature point pairs and camera parameters, which is conducive to low-cost and accurate reconstruction of the three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 is a schematic diagram of an image-based three-dimensional model reconstruction method provided in an embodiment of the present application;

[0036] Figure 2 The embodiment of this application provides Figure 1 Schematic diagram of the specific process of S200;

[0037] Figure 3 is a structural schematic diagram of an image-based three-dimensional model reconstruction system provided in an embodiment of the present application;

[0038] Figure 4 It is a structural diagram of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0040] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0041] It should also be understood that the term “and / or” used in the specification and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0042] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0043] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0044] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0045] From our long-term follow-up research in the relevant technical fields at home and abroad, we can see that there are many international institutions such as Microsoft, Autodesk, Stanford University and MIT that have made good research achievements in the field of rapid reconstruction of 3D shapes based on images, but these are only laboratory research results and are not commercially available. Microsoft once provided 3D reconstruction services based on images online, but due to the large number of user visits and insufficient technology, it could not bear the heavy technical services and soon shut down the corresponding servers. At present, there are also market promotions of 3D reconstruction systems based on images such as Canadian company FOTO3D, but they require a lot of manual interaction and have very high requirements for the shooting environment and shooting accuracy of the photos, so the market recognition is not high.

[0046] In China, except for Peking University, Tsinghua University, Institute of Automation of Chinese Academy of Sciences, Beihang University, Hong Kong Polytechnic University, Beijing Science and Technology Everbright Company, etc., this technology is almost at the academic stage, and there is little substantive in-depth research. In terms of marketization and productization, there are only Autodesk 123D Catch and Beijing Science and Technology Everbright 3DCloud platform. However, the research focuses of these scientific research institutions are different. For example, Beihang University mainly focuses on the research of virtual scenes used in the military field, and Hong Kong Polytechnic University mainly focuses on the research of human face stereo synthesis, etc.

[0047] There are good research results in the prior art on the rapid reconstruction of three-dimensional shapes based on images. For example, in three-dimensional software modeling, many excellent modeling software can be seen on the market at present, and the more well-known ones are 3DMAX, Maya, etc. Their common feature is to use some basic geometric elements, such as cubes, spheres, etc., through a series of geometric operations, such as translation, rotation, stretching, and Boolean operations, to construct complex geometric scenes. This method requires operators to have rich professional knowledge and be proficient in using modeling software. In addition, the operation is complicated and the cycle is long. At the same time, the three-dimensional model of the final component is not very realistic. For another example, a three-dimensional scanner (3Dimensional Scanner) is also called a three-dimensional digitizer (3Dimensional Digitizer). At present, three-dimensional scanners mainly use technologies such as lasers and structured light to obtain three-dimensional coordinate information through feedback from emitted light, while texture color is basically obtained by the camera on the device. This method requires expensive hardware equipment such as three-dimensional scanners, and due to the limitations of the technology itself, some materials and colors are affected by the reflection or refraction, absorption, etc. of light, resulting in many loopholes in the obtained three-dimensional model and the inability to complete the scanning. Such as human hair, dark clothing, and transparent objects, etc. In addition, 3D scanners can only obtain the position information of objects today. The texture features on the surface of objects still mostly require a lot of manual work to complete. The whole process is costly and time-consuming.

[0048] The method adopted in this application is to reconstruct the three-dimensional model based on the image. Image-Based Modeling and Rendering (IBMR) is an extremely active research field in the current computer graphics community. Compared with traditional geometry-based modeling and rendering, IBMR technology has many unique advantages. Image-based modeling and rendering technology provides us with the most natural way to obtain photo realism. Using IBMR technology, modeling becomes faster and more convenient, and a high rendering speed and a high degree of realism can be obtained. The latest research progress of IBMR has achieved many fruitful results and may fundamentally change our understanding and concept of computer graphics. Since the image itself contains rich scene information, it is naturally easy to obtain a photo-realistic scene model from the image. The main purpose of image-based modeling is to restore the three-dimensional geometric structure of the scene from a two-dimensional image. Restoring the three-dimensional shape of the scene from a two-dimensional image originally belonged to the content of computer graphics and computer vision. Due to its broad application prospects, researchers in computer graphics and computer vision are now full of interest in this field. Compared with the traditional method of using modeling software or 3D scanners to obtain three-dimensional models, the image-based modeling method is low-cost, highly realistic, and highly automated, so it has broad application prospects. This method is simple to operate, highly automated, low-cost, highly realistic in texture and color, and is not restricted by time and space. For example, the domestic 3DCloud runs in the form of a cloud. As long as the photos are uploaded to the cloud, the 3D model can be automatically generated. It is mainly used in many application fields such as 3D display, 3D printing, film and television media, advertising production, virtual reality, etc., and with factors such as low cost, it has a good future development prospect.

[0049] In view of this, the present application performs feature matching based on multi-view images, and reconstructs the model based on the position of feature point pairs and camera parameters, which is conducive to low-cost and accurate reconstruction of the three-dimensional model.

[0050] Figure 1 is a schematic diagram of an image-based three-dimensional model reconstruction method provided in an embodiment of the present application, the method comprising:

[0051] S100 , photographing a target to be reconstructed by a camera to obtain at least two multi-view images of the target to be reconstructed.

[0052] To reconstruct a 3D model of a target based on an image, it is first necessary to obtain an image of the target to be reconstructed. In the embodiment of the present application, at least two multi-view images of the target to be reconstructed are obtained. Multi-view geometry mainly studies the use of geometric methods to restore 3D objects through several 2D images. In short, it studies 3D reconstruction, which is mainly used in computer vision. 3D reconstruction is the process of restoring the 3D geometric structure information of an object from images of a static object from different viewpoints.

[0053] Specifically, photographing the target to be reconstructed by a camera to obtain at least two multi-view images of the target to be reconstructed includes: photographing the target to be reconstructed by the camera at various azimuth viewing angles to obtain at least two two-dimensional multi-view images.

[0054] Multi-view images refer to a collection of images of the same scene taken from different perspectives. These images capture the same object or scene from different positions and angles, providing multiple perspective information about the object or scene. Multi-view images play an important role in computer vision and 3D reconstruction. By analyzing the relationship between these images, the 3D geometric structure information of the object can be restored. The sources of multi-view images are diverse, and can come from data of different forms, different sites, different perspectives, different features, and different modalities. For example, photos of a person taken from different angles, data scanned by different devices, audio, vision, and text in a video, and other different modal data can all constitute multi-view images. Technically, the processing of multi-view images involves complex geometric and visual calculations. By analyzing the correspondence between different views, these images can be used to estimate the position and posture of the camera, and then restore the 3D structure of the scene.

[0055] Camera calibration is an extremely important step in the process of three-dimensional reconstruction. Therefore, the method described in the present application, before photographing the target to be reconstructed with a camera and obtaining at least two multi-view images of the target to be reconstructed, the method also includes: calibrating the camera in the scenes of various azimuth viewing angles of the target to be reconstructed to obtain parameters of the camera, wherein the parameters of the camera include internal and external parameters of the camera.

[0056] In the embodiments of the present application, accurate camera calibration is crucial for image-based 3D reconstruction tasks, and the internal and external parameters of the camera are determined through camera calibration. For applications where the reconstruction accuracy of 3D scenes or models is high and the shooting environment can be customized on demand, offline calibration technology can generally better meet user needs; on the contrary, if scene analysis and reconstruction are required from images or video sequences where the environment cannot be customized or calibration information is missing, only online calibration technology can be used. The research on camera calibration is divided into three parts: (1) calibration based on scene structure information, (2) self-calibration based on camera active information (pure rotation), and (3) self-calibration that does not rely on scene structure and camera active information.

[0057] The first camera calibration method is suitable for offline camera calibration. Offline camera calibration technology requires accurate camera intrinsic and extrinsic parameters as input and prerequisites for the reconstruction algorithm. In one possible implementation, a three-dimensional calibration object with a non-coplanar special calibration mark is used to provide the correspondence between image points and their corresponding three-dimensional space points and calculate the calibration parameters. In another possible implementation, at least two different views of a planar calibration pattern are required for calibration. Through the calibration algorithm, the projection matrix of the camera can be calculated, and the three-dimensional measurement information of the scene can be provided. Without giving the absolute translation, rotation and scaling parameters of the real scene, measurement reconstruction at the similarity transformation level can be achieved.

[0058] The second camera calibration method is suitable for online camera calibration. In many cases, such as when there is a lack of calibration equipment or the camera parameters are constantly changing, there is not enough data to support offline camera calibration. For multi-view 3D reconstruction of such scenes, online camera calibration technology is required. The main difference between online and offline calibration frameworks lies in the method of calibrating the camera or estimating camera parameters. In most literature, offline calibration technology is called self-calibration. Self-calibration methods can be roughly divided into two categories: self-calibration based on scene constraints and self-calibration based on geometric constraints.

[0059] Among them, the appropriate scene constraints for self-calibration based on scene constraints can often greatly simplify the difficulty of self-calibration. For example, parallel lines that are widely present in buildings or man-made scenes can help provide the information of the elimination points and elimination lines of sight in the three main orthogonal directions, and can provide the algebraic solution or numerical solution of the camera internal parameters based on this. The solution of the elimination point can be performed by voting and searching for the maximum value. In one possible implementation, a Gaussian sphere can be used to construct the solution space. In another possible implementation, a geometric parameter estimation method based on the three main orthogonal directions of the elimination point. In another possible implementation, the constraint of the absolute quadratic curve is constructed from the position of the elimination point and the calibration matrix is ​​solved by Cox decomposition.

[0060] Among them, self-calibration based on geometric constraints does not require external scene constraints, but only relies on the inherent geometric constraints between multiple views to complete the calibration task. In one possible implementation, the theory and algorithm of self-calibration are used using absolute quadratic surfaces. In one possible implementation, the camera parameters are solved based on the Kruppa equation. The progressive self-calibration technique is used to upgrade from projective reconstruction to metric reconstruction. A major difficulty of the self-calibration technique is that it is not unlimitedly applicable to any image or video sequence. In fact, there are specific motion sequences or spatial feature distributions that lead to the degeneration and singular solutions of the self-calibration solution framework.

[0061] S200 , performing feature extraction according to each of the multi-view images to obtain feature points of each of the multi-view images.

[0062] In image-based reconstruction technology, it is necessary to perform feature matching on multi-view images. First, feature points need to be extracted, and feature detection needs to be performed before feature extraction. Features in the traditional sense are defined as image regions or positions with large brightness or chromaticity changes in at least one specific direction. In one possible implementation, the first-order derivative can be used to estimate the local cross-correlation value. This method can give a robust detection result but lacks positioning accuracy in some cases. In one possible implementation, the product of gradient and curvature is used to characterize corner features and detect corners. The SUSAN detector uses a combination of the size, center and moment information of the feature region to detect corners. In one possible implementation, the scale-invariant feature detection operator SIFT can be used. The advantage of SIFT is that it can effectively extract features that are invariant to a certain extent, such as rotation and scaling, thereby greatly reducing the dependence of the feature detection algorithm on the environment and image quality.

[0063] Specifically, Figure 2 The embodiment of this application provides Figure 1 The specific process diagram of S200 is as follows: Figure 2 As shown, the extracting features according to each of the multi-view images to obtain feature points of each of the multi-view images includes:

[0064] S201, performing feature detection according to pixel points of each of the multi-view images to obtain pixel point information of each of the multi-view images;

[0065] S202: Perform feature extraction according to pixel point information of each of the multi-view images and preset feature parameters to obtain feature points of each of the multi-view images.

[0066] In an embodiment of the present application, feature detection is performed on the pixels of each of the multi-view images through the possible implementation methods described above to obtain pixel information of each of the multi-view images, wherein the pixel information includes pixel position, pixel color (RGB three-channel information) and pixel clarity, etc., and features are extracted according to preset feature parameters to determine which pixels are feature points. For example, 10,000 pixels are detected in a multi-view image, and the pixel information of these pixels is extracted respectively, and the pixel information is screened according to the preset feature information to determine feature points from the pixels, wherein the feature points carry feature point information, including the position of the feature points, the color of the feature points, etc. Among them, the multi-view images are of the same size, and a coordinate system is established with the center of each of the multi-view images, and the feature point coordinates of each of the multi-view images are obtained, which is the position information of the feature points.

[0067] In an embodiment of the present application, the feature points of the multi-view images need to be corrected before the feature points are matched. This is because the pixel points of each multi-view image taken under multiple viewing angles may also deviate. For example, the depth information of the pixel points cannot be obtained or the pixel information has errors. In view of this, the method described in the present application also includes: adjusting the corresponding polar lines in at least two of the multi-view images to a horizontal state and to be on the same horizontal scanning line, and correcting the feature points of the multi-view images.

[0068] Image correction technology can be used to adjust the corresponding epipolar lines in the two views to a horizontal state and on the same horizontal scan line, so that the depth information can be restored using the traditional dual-view algorithm based on horizontal parallax. In this framework, Markov random field modeling can be used and the solution can be achieved using an optimization algorithm based on graph theory, which is conducive to obtaining more accurate feature point information, thereby making the 3D model of the target to be reconstructed more accurate.

[0069] S300, matching feature points of at least two of the multi-view images to obtain multiple groups of feature point pairs, wherein a group of feature point pairs is the same target feature point on the target to be reconstructed, and the feature point positions of the target feature points in different multi-view images are different.

[0070] In image-based reconstruction technology, both sparse feature matching and dense feature matching can be considered. Generally, the specific choice needs to be made according to the application background and scene characteristics. The performance of the feature matching algorithm will be affected by lens distortion, light environment, scene occlusion and other unknown image noise. There are currently two main approaches to solving the matching problem. The first approach is to detect a feature record in a key frame and use a tracking algorithm to track this feature set in subsequent frames. The second approach is to perform feature detection independently in multiple views and establish matching feature pairs through data association. This can be achieved through a simple regional correlation algorithm, or by defining an objective function that describes similarity and optimizing it through various means. For situations where dense scene reconstruction is required, dense multi-view matching is also required. The performance of the dense multi-view matching algorithm directly affects the final reconstruction quality. When the image sampling points are dense enough, optical flow technology can be used to simulate the pixel or feature displacement between adjacent images.

[0071] Specifically, matching the feature points of at least two of the multi-view images to obtain multiple groups of feature point pairs includes:

[0072] Feature points of at least two of the multi-view images are matched according to the feature point information, and feature points with consistent feature point information are taken as a group of feature point pairs to obtain multiple groups of feature point pairs, wherein the feature points with consistent feature point information are the same target feature point.

[0073] In an embodiment of the present application, the feature point information includes the feature point color, and the feature points of at least two multi-view images are matched according to the feature point information. Since for a target feature point on a target to be reconstructed, the position of the target feature point in different multi-view images may be different, for example, there is a target feature point P on the reconstructed target, which appears as feature point A on the left view image and as feature point B on the right view image. When it is detected that the feature information of feature point A and feature point B is consistent, feature point A and feature point B are determined to be a set of feature point pairs. In fact, both are target feature points P on the target to be reconstructed, so feature point A and feature point B are matched and determined to be a set of feature point pairs, thereby obtaining the coordinates of feature point A and feature point B, respectively.

[0074] S400, determining the position of each of the target feature points according to the position of each group of feature point pairs and the parameters of the camera.

[0075] Specifically, determining the position of each target feature point according to the position of each group of feature point pairs and the parameters of the camera includes:

[0076] A triangulation calculation is performed according to the position of each group of feature point pairs and the parameters of the camera to determine the position of each target feature point.

[0077] Triangulation algorithms are commonly used in the field of computer vision and 3D reconstruction. They are used to derive 3D structural information in a scene from images or visual data from multiple perspectives. Triangulation algorithms use the principles of visual geometry to measure and analyze feature points or corresponding points from multiple perspectives to calculate the positions of these points in 3D space. The triangulation of 3D structures can also be simulated by optical flow with point-to-point correspondence information. Under the assumption of dense spatial sampling, optical flow can be effectively approximated by coefficient feature displacement.

[0078] In the embodiment of the present application, the triangulation calculation is to perform triangulation calculation for each set of matched feature point pairs using the pixel coordinates of the feature point in two viewing angles and the internal and external parameters of the camera, and derive the positions of the two feature points in three-dimensional space. For example, according to the step of S300: there is a target feature point P on the reconstructed target, which is represented as feature point A on the left view image and as feature point B on the right view image. When the feature information of feature point A and feature point B is detected to be consistent, feature point A and feature point B are determined to be a set of feature point pairs. In fact, both are target feature points P on the target to be reconstructed. Feature point A and feature point B are matched and determined as a set of feature point pairs, thereby obtaining the coordinates of feature point A and feature point B respectively. According to the step of camera calibration before S100, the intrinsic parameters and extrinsic parameters of the camera in each multi-view are obtained. According to the coordinates of the position of each set of feature point pairs, the coordinates of feature point A and feature point B and the intrinsic and external parameters of the camera, triangulation calculation is performed to determine the position of each target feature point.

[0079] S500: reconstructing a three-dimensional model of the target to be reconstructed according to the position of each target feature point.

[0080] In the embodiment of the present application, the position of the target feature point on the target to be reconstructed is determined through S100-S400, and the three-dimensional model of the target to be reconstructed is reconstructed. The present application performs feature matching based on multi-view images and combines the position of the feature point pair and camera parameters to reconstruct the model, which is conducive to low-cost and accurate reconstruction of the three-dimensional model.

[0081] See also Figure 3 , shows a structural schematic diagram of an image-based three-dimensional model reconstruction system provided in an embodiment of the present application. For the sake of ease of explanation, only the parts related to the embodiment of the present application are shown.

[0082] The image-based three-dimensional model reconstruction system 600 may specifically include the following modules:

[0083] An image acquisition module 610 is used to photograph a target to be reconstructed by a camera to acquire at least two multi-view images of the target to be reconstructed;

[0084] A feature extraction module 620, configured to extract features from each of the multi-view images to obtain feature points of each of the multi-view images;

[0085] A feature matching module 630 is used to match feature points of at least two of the multi-view images to obtain multiple groups of feature point pairs, wherein a group of feature point pairs is the same target feature point on the target to be reconstructed, and the feature point positions of the target feature points in different multi-view images are different;

[0086] A target feature point determination module 640 is used to determine the position of each target feature point according to the position of each group of feature point pairs and the parameters of the camera;

[0087] The model reconstruction module 650 is used to reconstruct a three-dimensional model of the target to be reconstructed according to the position of each target feature point.

[0088] The system 600 may specifically include the following modules:

[0089] The camera calibration module is used to calibrate the camera in the scenes of various azimuth viewing angles of the target to be reconstructed to obtain the parameters of the camera, wherein the parameters of the camera include the internal and external parameters of the camera.

[0090] The image acquisition module 610 is specifically configured to acquire at least two two-dimensional multi-view images by shooting the target to be reconstructed at various azimuth viewing angles through the camera.

[0091] The feature extraction module 620 may specifically include the following units:

[0092] A feature detection unit 621, configured to perform feature detection according to pixel points of each of the multi-view images to obtain pixel point information of each of the multi-view images;

[0093] The feature point acquisition unit 622 is used to perform feature extraction according to the pixel point information of each of the multi-view images and preset feature parameters to obtain the feature points of each of the multi-view images.

[0094] The multi-view images have the same size, a coordinate system is established with the center of each multi-view image, and the coordinates of the feature points of each multi-view image are obtained, wherein the feature points carry feature point information, and the feature point information includes the feature point coordinates and the feature point color;

[0095] The feature matching module 630 is specifically used to match the feature points of at least two of the multi-view images according to the feature point information, and take the feature points with consistent feature point information as a group of feature point pairs to obtain multiple groups of feature point pairs, wherein the feature points with consistent feature point information are the same target feature point.

[0096] The target feature point determination module 640 is specifically configured to perform triangulation calculation according to the position of each group of feature point pairs and the parameters of the camera to determine the position of each target feature point.

[0097] The image-based three-dimensional model reconstruction system provided in the embodiment of the present application can be applied in the aforementioned method embodiment. For details, please refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0098] Figure 4 Schematic diagram of the structure of the terminal device provided in the embodiment of the present application. Figure 4 As shown, the terminal device 700 of this embodiment includes: at least one processor 710 ( Figure 4 Only one is shown in the figure), a memory 720, and a computer program 721 stored in the memory 720 and executable on the at least one processor 710, wherein the processor 710 implements the steps in the above method embodiment when executing the computer program 721.

[0099] The terminal device 700 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device may include, but is not limited to, a processor 710 and a memory 720. Those skilled in the art will appreciate that Figure 4 It is only an example of the terminal device 700 and does not constitute a limitation on the terminal device 700. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0100] The processor 710 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0101] In some embodiments, the memory 720 may be an internal storage unit of the terminal device 700, such as a hard disk or memory of the terminal device 700. In other embodiments, the memory 720 may also be an external storage device of the terminal device 700, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device 700. Further, the memory 720 may also include both an internal storage unit of the terminal device 700 and an external storage device. The memory 720 is used to store an operating system, an application program, a boot loader (Boot Loader), data and other programs, such as the program code of the computer program, etc. The memory 720 may also be used to temporarily store data that has been output or is to be output.

[0102] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0103] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0104] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0105] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0106] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0107] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0108] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0109] The present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed through a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing.

[0110] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for reconstructing a three-dimensional model based on an image, characterized in that: The method comprises: Shooting the target to be reconstructed by a camera to obtain at least two multi-view images of the target to be reconstructed; Perform feature extraction according to each of the multi-view images to obtain feature points of each of the multi-view images; Matching feature points of at least two of the multi-view images to obtain multiple groups of feature point pairs, wherein a group of feature point pairs is the same target feature point on the target to be reconstructed, and the feature point positions of the target feature points in different multi-view images are different; Determining the position of each target feature point according to the position of each group of feature point pairs and the parameters of the camera; A three-dimensional model is reconstructed for the target to be reconstructed according to the position of each target feature point.

2. The method according to claim 1, characterized in that: Before photographing the target to be reconstructed by a camera to obtain at least two multi-view images of the target to be reconstructed, the method further includes: The camera is calibrated in each azimuth viewing angle of the target to be reconstructed to obtain parameters of the camera, wherein the parameters of the camera include internal and external parameters of the camera.

3. The method according to claim 1, characterized in that: The step of photographing the target to be reconstructed by a camera to obtain at least two multi-view images of the target to be reconstructed includes: The camera is used to shoot at various azimuth viewing angles of the target to be reconstructed to obtain at least two two-dimensional multi-view images.

4. The method according to claim 1, characterized in that: The extracting features according to each of the multi-view images to obtain feature points of each of the multi-view images comprises: Performing feature detection according to the pixel points of each of the multi-view images to obtain pixel point information of each of the multi-view images; Feature extraction is performed according to the pixel point information of each of the multi-view images and preset feature parameters to obtain feature points of each of the multi-view images.

5. The method according to claim 1, characterized in that: The multi-view images have the same size, a coordinate system is established with the center of each multi-view image, and the coordinates of the feature points of each multi-view image are obtained, the feature points carry feature point information, and the feature point information includes the color of the feature point. The feature points of at least two multi-view images are matched to obtain multiple groups of feature point pairs, including: Feature points of at least two of the multi-view images are matched according to the feature point information, and feature points with consistent feature point information are taken as a group of feature point pairs to obtain multiple groups of feature point pairs, wherein the feature points with consistent feature point information are the same target feature point.

6. The method according to claim 1, characterized in that: Determining the position of each target feature point according to the position of each group of feature point pairs and the parameters of the camera includes: A triangulation calculation is performed according to the position of each group of feature point pairs and the parameters of the camera to determine the position of each target feature point.

7. The method according to claim 1, characterized in that: Before matching the feature points of at least two of the multi-view images to obtain a plurality of feature point pairs, the method further includes: Corresponding epipolar lines in at least two of the multi-view images are adjusted to be in a horizontal state and on the same horizontal scanning line, and feature points of the multi-view images are corrected.

8. An image-based three-dimensional model reconstruction system, characterized in that: The system comprises: An image acquisition module, used to photograph a target to be reconstructed through a camera to acquire at least two multi-view images of the target to be reconstructed; A feature extraction module, used to extract features from each of the multi-view images to obtain feature points of each of the multi-view images; A feature matching module, used for matching feature points of at least two of the multi-view images to obtain multiple groups of feature point pairs, wherein a group of feature point pairs is the same target feature point on the target to be reconstructed, and the feature point positions of the target feature points in different multi-view images are different; A target feature point determination module, used to determine the position of each target feature point according to the position of each group of feature point pairs and the parameters of the camera; The model reconstruction module is used to reconstruct a three-dimensional model of the target to be reconstructed according to the position of each target feature point.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images