Image three-dimensional reconstruction method and device, storage medium and electronic device

Abstract

The embodiment of the invention provides an image three-dimensional reconstruction method and device, a storage medium and an electronic device, and the method comprises the steps: obtaining image data, and carrying out the feature extraction of the image data, and obtaining image features; performing natural mark detection on an environment scene corresponding to the image data to obtain a natural mark of the environment scene; according to the image features and the natural marks, extracting a target plane in the environment scene; and performing three-dimensional reconstruction on the image data and the target plane by taking the naturally marked three-dimensional position information as plane constraint information. Through the method and the device, the problems of high data acquisition cost and high algorithm complexity during three-dimensional reconstruction in related technologies are solved, and the effect of reducing the three-dimensional reconstruction data acquisition cost and the algorithm complexity is achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer vision, and in particular, to a method and device for three-dimensional reconstruction of an image, a storage medium, and an electronic device. Background Art

[0002] Traditional 3D reconstruction methods are usually based on devices such as stereo cameras or laser scanners to collect a large amount of 3D point clouds or image data, and then convert these data into 3D models through 3D reconstruction algorithms. In order to achieve pure visual 3D reconstruction, one or more high-precision cameras are needed for image acquisition and 3D reconstruction. And the camera needs to be calibrated to obtain the camera's intrinsic and extrinsic parameters. At the same time, 3D reconstruction needs to be performed in a flat scene, and there must be enough plane structures in the scene. Plane constraints are based on plane structures, so there must be enough plane structures in the scene. And 3D reconstruction requires a lot of calculations and storage, so sufficient computing resources and storage space are required. In particular, a large number of feature extraction, matching and optimization operations are required in the 3D reconstruction process, which places high demands on computing resources.

[0003] In summary, when performing three-dimensional reconstruction in related technologies, there are problems of high data acquisition cost and high algorithm complexity. Summary of the invention

[0004] The embodiments of the present invention provide a method and device for three-dimensional reconstruction of an image, a storage medium and an electronic device, so as to at least solve the problems of high data acquisition cost and high algorithm complexity when performing three-dimensional reconstruction in related technologies.

[0005] According to one embodiment of the present invention, a method for three-dimensional reconstruction of an image is provided, comprising: acquiring image data, and performing feature extraction on the image data to obtain image features; performing natural marker detection on an environmental scene corresponding to the image data to obtain natural markers of the environmental scene; extracting a target plane in the environmental scene based on the image features and the natural markers; and performing three-dimensional reconstruction of the image data and the target plane using the three-dimensional position information of the natural markers as plane constraint information.

[0006] According to another embodiment of the present invention, a three-dimensional reconstruction device of an image is provided, comprising: an image acquisition module, used to acquire image data, and perform feature extraction on the image data to obtain image features; a natural marking module, used to perform natural marking detection on an environmental scene corresponding to the image data, and obtain natural markings of the environmental scene; a plane extraction module, used to extract a target plane in the environmental scene based on the image features and the natural markings; and a three-dimensional reconstruction module, used to perform three-dimensional reconstruction of the image data and the target plane using the three-dimensional position information of the natural markings as plane constraint information.

[0007] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0008] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0009] The present invention provides a method for three-dimensional reconstruction of an image, which obtains image data and extracts features from the image data to obtain image features; performs natural marker detection on the environment scene corresponding to the image data to obtain natural markers of the environment scene; extracts the target plane in the environment scene according to the image features and the natural markers; uses the three-dimensional position information of the natural markers as plane constraint information to perform three-dimensional reconstruction of the image data and the target plane. The method solves the problem of high data acquisition cost and high algorithm complexity in the related art when performing three-dimensional reconstruction, and achieves the effect of reducing the data acquisition cost and algorithm complexity of three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a hardware structure block diagram of a computer terminal of a three-dimensional image reconstruction method according to an embodiment of the present invention;

[0011] Figure 2 is a flow chart of a method for three-dimensional reconstruction of an image according to an embodiment of the present invention;

[0012] Figure 3 is a structural block diagram of a three-dimensional reconstruction device for an image according to an embodiment of the present invention;

[0013] Figure 4 is a structural block diagram of a three-dimensional reconstruction device for an image according to an embodiment of the present invention;

[0014] Figure 5is a structural block diagram of a three-dimensional reconstruction device for an image according to an embodiment of the present invention;

[0015] Figure 6 is a structural block diagram of a three-dimensional reconstruction device for an image according to an embodiment of the present invention;

[0016] Figure 7 It is a flow diagram of a planar constraint three-dimensional reconstruction method based on a nature marker according to a scenario embodiment of the present invention;

[0017] Figure 8 is a flow chart of a three-dimensional reconstruction method of an image according to a scene embodiment of the present invention;

[0018] Fig. 9 It is a schematic diagram of a combined feature tracking algorithm according to a scenario embodiment of the present invention. DETAILED DESCRIPTION

[0019] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0020] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0021] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a computer terminal of a three-dimensional image reconstruction method according to an embodiment of the present invention. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0022] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the three-dimensional reconstruction method of the image in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0023] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0024] In this embodiment, a three-dimensional reconstruction method of an image running on the above-mentioned computer terminal is provided. Figure 2 is a flow chart of a method for three-dimensional reconstruction of an image according to an embodiment of the present invention. Figure 2 As shown, the three-dimensional reconstruction method includes the following steps:

[0025] Step S202, acquiring image data, and performing feature extraction on the image data to acquire image features;

[0026] Step S204, performing natural mark detection on the environment scene corresponding to the image data to obtain natural marks of the environment scene;

[0027] In an exemplary embodiment, natural marker detection includes at least one of the following: corner detection; edge detection; plane detection.

[0028] In actual implementation, natural marker extraction can be automatically completed through computer vision and deep learning algorithms without human intervention, thus achieving 3D reconstruction faster and more efficiently.

[0029] In an exemplary embodiment, after step S204, the method further includes: extracting features from the natural markers, and performing feature matching retrieval on the natural markers and the environmental scene based on the feature extraction to confirm that the natural markers come from the environmental scene.

[0030] In the actual implementation process, the natural markers are detected and extracted. After obtaining the physical size, the image feature points are extracted and a detection tree is generated based on the image feature points. The image data is searched to confirm that the natural markers come from the environment scene. At the same time, feature extraction is also performed on the collected scene image information, followed by feature matching and retrieval to confirm that the collected scene image information belongs to the original image data.

[0031] Step S206, extracting the target plane in the environment scene according to the image features and natural markers;

[0032] In an exemplary embodiment, after step S206, the method further includes: triangulating the target plane, and performing fitting optimization on the target plane using a nonlinear optimization algorithm according to the triangulation.

[0033] In the actual implementation process, after the plane information is stored, the plane is extracted and recognized in combination with the extracted scene image information. After the target plane is determined, spatial triangulation based on plane constraints is also involved, wherein the plane constraint refers to the three-dimensional position information of the extracted natural marker, and the specific three-dimensional position information may include position (coordinate) information and posture information. In the specific implementation process, the target plane in the above embodiment can also be fitted and optimized according to the result of triangulation, that is, the above optimization step, so that the target plane better satisfies the plane constraint, so that the pixel points and the target plane can be used for more accurate three-dimensional reconstruction.

[0034] Step S208: Using the three-dimensional position information of the natural marker as plane constraint information, three-dimensionally reconstruct the image data and the target plane.

[0035] In an exemplary embodiment, before step S208, the method further includes: performing combined feature tracking on the target plane to obtain plane constraint information of all image data in the video image, wherein the combined feature tracking includes point feature tracking and area feature tracking.

[0036] In an exemplary embodiment, combined feature tracking is performed on a target plane, including: using a corner feature point detection algorithm to obtain an image gradient change rate of a pixel point in image data, and performing point feature tracking on a pixel point whose image gradient change rate is greater than or equal to an image gradient change rate threshold; selecting an image area in the current frame image data that is equal to the pixel grayscale average value of a preset recognition local template as a target area, and performing regional feature tracking on the target area.

[0037] Through the above steps, image data is obtained, and image features are obtained by extracting features from the image data; natural marker detection is performed on the environment scene corresponding to the image data to obtain the natural markers of the environment scene; the target plane in the environment scene is extracted according to the image features and the natural markers; the three-dimensional position information of the natural markers is used as plane constraint information to perform three-dimensional reconstruction of the image data and the target plane. Through the present invention, the problems of high data acquisition cost and high algorithm complexity in the related art when performing three-dimensional reconstruction are solved, and the effect of reducing the data acquisition cost and algorithm complexity of three-dimensional reconstruction is achieved.

[0038] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0039] In this embodiment, a three-dimensional reconstruction device for an image is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0040] Figure 3 is a structural block diagram of a three-dimensional reconstruction device for an image according to an embodiment of the present invention. Figure 3 As shown, the three-dimensional reconstruction device 30 includes: an image acquisition module 310, which is used to acquire image data and perform feature extraction on the image data to obtain image features; a natural marking module 320, which is used to perform natural marking detection on the environmental scene corresponding to the image data to obtain the natural marking of the environmental scene; a plane extraction module 330, which is used to extract the target plane in the environmental scene according to the image features and the natural marking; and a three-dimensional reconstruction module 340, which is used to use the three-dimensional position information of the natural marking as plane constraint information to perform three-dimensional reconstruction on the image data and the target plane.

[0041] In an exemplary embodiment, Figure 4 is a structural block diagram of a three-dimensional reconstruction device for an image according to an embodiment of the present invention. Figure 4 As shown, the three-dimensional reconstruction device 40 includes Figure 3 In addition to all the modules shown, it also includes: a combined tracking module 410, which is used to perform combined feature tracking on the target plane to obtain plane constraint information of all image data in the video image, wherein the combined feature tracking includes point feature tracking and area feature tracking.

[0042] In an exemplary embodiment, Figure 5 is a structural block diagram of a three-dimensional reconstruction device for an image according to an embodiment of the present invention. Figure 5 As shown, the three-dimensional reconstruction device 50 includes Figure 4 In addition to all the modules shown, it also includes: a matching retrieval module 510, which is used to extract features from natural markers, and perform feature matching retrieval on natural markers and environmental scenes based on the feature extraction to confirm that the natural markers come from the environmental scenes.

[0043] In an exemplary embodiment, Figure 6 is a structural block diagram of a three-dimensional reconstruction device for an image according to an embodiment of the present invention. Figure 6 As shown, the three-dimensional reconstruction device 60 includes Figure 5 In addition to all the modules shown, the present invention further includes: a triangulation module 610, which is used to triangulate the target plane, so as to perform fitting optimization on the target plane by using a nonlinear optimization algorithm based on the triangulation.

[0044] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0045] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0046] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0047] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0048] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0049] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0050] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0051] In order to enable those skilled in the art to better understand the technical solution of the present invention, it is described below in conjunction with specific scenario embodiments.

[0052] Scenario Example 1

[0053] Traditional 3D reconstruction methods are usually based on devices such as stereo cameras or laser scanners to collect a large amount of 3D point cloud or image data, and then convert these data into 3D models through 3D reconstruction algorithms. Compared with plane-constrained 3D reconstruction based on natural features, traditional methods have the following technical defects: 1. High data acquisition cost: Traditional 3D reconstruction methods require the use of professional equipment for data acquisition. These devices are expensive and difficult to obtain. At the same time, the acquisition process also requires a lot of time and human resources. 2. Uneven data density: The 3D point cloud or image data collected by traditional methods often have uneven density, especially in complex scenes or severe occlusions, which are prone to missing data or overlapping data. 3. High algorithm complexity: Traditional 3D reconstruction algorithms often require a lot of computing resources and time, especially when processing large-scale data, the computational complexity will increase dramatically, resulting in slow reconstruction speed. 4. Large restrictions on scenes: Traditional 3D reconstruction methods usually require preprocessing or restrictions on scenes, such as calibrating cameras, placing markers or reference points, etc. These restrictions may cause certain restrictions and troubles to actual application scenarios. The visual reconstruction solution based on QR code can restore the coordinate movement of the camera because the scale of the QR code is known. However, when reconstructing an unknown environment in an indoor scene, most of the scene needs to be covered with QR code, which affects the protection of indoor information.

[0054] In order to convert multiple two-dimensional images or point cloud data into three-dimensional scenes, three-dimensional reconstruction technology can often be used. Its application environment is very wide, such as architecture and cultural heritage protection: by collecting multiple images or point cloud data, reconstructing the three-dimensional model of the building or cultural heritage for protection and restoration; e-commerce: by collecting images or point cloud data of goods from multiple angles, reconstructing the three-dimensional model of the goods for online display and sales; games and virtual reality: by collecting multiple images or point cloud data, reconstructing the three-dimensional model of the virtual scene for games and virtual reality applications. And because three-dimensional reconstruction technology has a wide range of applications in architecture, e-commerce, games, robots, medicine and other fields, its application scenarios are also constantly expanding and deepening, and its requirements for the robustness of scene perception are getting higher and higher. In some museum scenes where labeling cannot be performed, laser equipment cannot be used, the signal is poor, and the surrounding light is dim, a new three-dimensional reconstruction method is needed.

[0055] Therefore, in the scene embodiment of the present invention, for indoor scenes, physical measurements are performed on known natural markers, and the information is stored as image data retrieval library information before three-dimensional reconstruction. At the same time, the scale-invariant feature transform (SIFT) feature descriptor is used to extract features of the nature marker. Next, the program acceleration (FBOW) is used for image retrieval, and the fast library for approximate nearest neighbors (FLANN) algorithm is used for feature matching. When the target information is successfully identified and determined to appear in the reconstructed image information, the physical scale of the target information is read. Then, the target area is tracked by combining features, and the relative coordinates obtained by tracking are used to perform simultaneous constraint equations and perform nonlinear optimization. From to obtain the coordinate movement of the camera when the target nature marker is observed. The three-dimensional scene is reconstructed according to the coordinate movement. The combined features include point features and regional features. In the scene embodiment of the present invention, the three-dimensional reconstruction of the image is realized mainly by using the technology of assisting the three-dimensional reconstruction of the nature marker tracking algorithm and using the combined features to perform reverse combined tracking of the nature marker. The core idea is to use the planar constraint information in the scene to guide the 3D reconstruction process, thereby improving the accuracy and robustness of the reconstruction.

[0056] Natural markers refer to geometric features commonly found in scenes, such as corners, edges, planes, etc. Compared with traditional manual marker methods, natural markers are more common and easier to detect, which can greatly simplify the 3D reconstruction process. In the plane-constrained 3D reconstruction method based on natural markers, natural markers are mainly used to extract plane information in the scene, and this plane information is used to guide the 3D reconstruction process.

[0057] The basic process of this method includes steps such as image acquisition, natural marker detection, plane extraction, plane-constrained 3D reconstruction and optimization processing. In the image acquisition stage, a camera or other sensor can be used to acquire multiple perspective images of the scene. In the natural marker detection stage, a specific algorithm is used to detect natural markers in the scene, such as corners or edges. In the plane extraction stage, natural markers and other geometric features are used to extract planes in the scene, and these planes are classified and segmented. In the plane-constrained 3D reconstruction stage, the extracted plane constraint information is used to guide the 3D reconstruction algorithm and generate a 3D model. Finally, in the optimization stage, the generated 3D model is optimized to improve the accuracy and completeness of the model.

[0058] Figure 7 is a flow chart of a planar constraint 3D reconstruction method based on a nature marker according to a scenario embodiment of the present invention, such as Figure 7 As shown, it includes data acquisition, natural marker detection and plane-constrained 3D reconstruction.

[0059] 1. Natural Marker Detection

[0060] Natural marker detection includes corner detection, edge detection, plane detection, etc. It can be implemented using traditional computer vision algorithms or deep learning algorithms. The results of natural marker detection will be used for subsequent plane extraction and segmentation.

[0061] 2. Plane Constrained 3D Reconstruction

[0062] Plane-constrained 3D reconstruction refers to reconstructing the 3D model in the scene through image information from multiple perspectives and 3D position information of natural markers under the constraints of natural markers. Plane-constrained 3D reconstruction mainly includes the following steps:

[0063] (1) Plane extraction and segmentation: According to the position information of natural markers, the planes in the scene are segmented, and the planes are fitted and optimized.

[0064] (2) Feature point matching: The correspondence between multiple perspective images is obtained through the feature point matching algorithm.

[0065] (3) 3D reconstruction: Using multiple perspective images and the 3D position information of natural markers, the 3D geometric information of the scene is obtained through a 3D reconstruction algorithm.

[0066] (4) Optimization processing: Optimize the generated 3D model to improve the accuracy and completeness of the model.

[0067] In actual applications, the specific implementation of plane-constrained 3D reconstruction may be different, but the above steps are its basic components.

[0068] Scenario Example 2

[0069] In this scenario embodiment, the process of the three-dimensional reconstruction method of an image according to the embodiment of the present invention is described in detail. Figure 8 is a flow chart of a three-dimensional reconstruction method of an image according to a scene embodiment of the present invention, such as Figure 8 As shown, the three-dimensional reconstruction method of the image includes the following steps:

[0070] Step S801, data acquisition and preprocessing: First, it is necessary to shoot a video before 3D reconstruction. Video shooting needs to consider factors such as lighting conditions, camera position and angle, speed and direction of moving objects. These factors will affect the quality and availability of video data, thereby affecting the results of 3D reconstruction. The video needs to be processed to improve the quality and accuracy of the data. Video processing can include removing noise, adjusting contrast and brightness, removing motion blur, etc.

[0071] In the actual implementation process, the above data collection includes collecting and entering the known environment plane information of the scene, and collecting the scene image information, which can be specifically performed by shooting a video. Then, data preprocessing is performed.

[0072] Step S802, detection and extraction of natural markers: Automatically detect and extract natural markers through computer vision and deep learning algorithms to determine the position and posture information of the markers, including extracting feature points, generating retrieval trees, and storing plane information.

[0073] In the actual implementation process, after the known environmental plane information of the scene is collected and recorded in step S801, the natural markers are detected and extracted to obtain the physical size, where the physical size can be the physical size of the environmental plane, etc.

[0074] In the actual implementation process, the natural markers are detected and extracted. After obtaining the physical size, the image feature points are extracted and a detection tree is generated based on the image feature points. The image data is searched to confirm that the natural markers come from the environment scene. At the same time, feature extraction is also performed on the collected scene image information, followed by feature matching and retrieval to confirm that the collected scene image information belongs to the original image data.

[0075] Step S803, Extraction and segmentation of plane constraints: According to the position and attitude information of natural markers, extract and segment the planes in the scene, such as the ground, walls, etc. The source of the plane information is as follows:

[0076] 1) In three-dimensional space, a plane can be represented as ax + by + cz + d = 0, where (a, b, c) is the normal vector of the plane, (x, y, z) is a point on the plane, and d is the distance. Given a set of n three-dimensional points {x1, x2,..., xn}, methods such as RANSAC can be used to obtain a set of inliers {x1', x2',..., xm'}, where m < n. These inliers can be fitted to a plane P. The method of fitting the plane can use the least squares method, that is, minimize the following formula:

[0077] E = Σ(axi + byi + czi + d)^2

[0078] where (xi, yi, zi) is the coordinate of the i-th inlier, and E is the objective function.

[0079] 2) Perform triangulation: Given a set of n three-dimensional points {x1, x2,..., xn} and m planes {P1, P2,..., Pm}, where each plane Pi is represented by its parameters (ai, bi, ci, di), a space triangulation method based on plane constraints can be used for three-dimensional reconstruction. The goal of this method is to minimize the following formula:

[0080] E = Σ(||x_i - P_j||^2 - dj)^2

[0081] where x_i is the coordinate of the i-th three-dimensional point, P_j is the j-th plane, and dj is the distance of the j-th plane. By minimizing the objective function E, a set of optimal three-dimensional points {x1', x2',..., xn'} can be obtained, and these points can be used for three-dimensional reconstruction.

[0082] 3) Optimization:

[0083] During the triangulation process, a non-linear optimization algorithm can be used to optimize the reconstruction result to meet the plane constraints. The goal of optimization is to minimize the following formula:

[0084] E = Σ(||x_i - P_j||^2 - dj)^2 + λΣ(||P_j||^2 - 1)^2

[0085] Where x_i is the coordinate of the i-th 3D point, P_j is the j-th plane, d_j is the distance of the j-th plane, and λ is the regularization coefficient. By minimizing E, a set of optimal 3D points {x1', x2', ..., xn'} and planes {P1', P2', ..., Pm'} can be obtained, and these points and planes can be used for more accurate 3D reconstruction. Among them, the optimization of points and planes is also the optimization of the target plane in the above embodiment.

[0086] In the actual implementation process, after the plane information is stored, the plane is extracted and recognized in combination with the extracted scene image information. After the target plane is determined, spatial triangulation based on plane constraints is also involved, wherein the plane constraint refers to the three-dimensional position information of the extracted natural marker, and the specific three-dimensional position information may include position (coordinate) information and posture information. In the specific implementation process, the target plane in the above embodiment can also be fitted and optimized according to the result of triangulation, that is, the above optimization step, so that the target plane better satisfies the plane constraint, so that the pixel points and the target plane can be used for more accurate three-dimensional reconstruction.

[0087] Step S804: perform reverse combination tracking on the nature marker using the combination features.

[0088] In the actual implementation process, the combined plane tracking based on the fusion feature performs the reverse combined tracking, which can ensure that the corresponding image data is obtained more accurately during the tracking process of the video data, and at the same time can reduce the computational complexity.

[0089] Fig. 9 is a schematic diagram of a combined feature tracking algorithm according to an embodiment of the present invention, such as Fig. 9 As shown, the template image T corresponds to the nature marker to be identified, and T(x) is the corresponding image pixel information. The original image I is the image frame captured by the corresponding hardware device. I(x) is the corresponding image pixel information. The template image T(x) and the original image I(x) are transformed through the transformation function Wrap(x:p) to minimize the difference between the transformed image I(W(x:p)) and the template image. Among them, x refers to the pixel coordinate under the image coordinate system, p refers to the transformation parameter, and W:x→W(x;p) refers to a function that transforms the image coordinates. Then the residual of the similarity of the square difference is:

[0090]

[0091] As a nonlinear least squares problem, the variable to be optimized is p, |||| 2represents the two-norm operation, and Δp represents the change in the variable p to be optimized. Derivative the expression, and then set the derivative function to 0 to solve for p, and the Taylor expansion of the residual expression at p is:

[0092]

[0093] Since the reverse combined tracking algorithm considers the linearization of Δp, the minimization formula of the objective function is:

[0094]

[0095] in, Represents the difference between image pixels, which is characterized by and Neither of them has anything to do with P, that is, these two items can be calculated in advance before iterative optimization, and the Hessian matrix and the inverse of the Hessian matrix can be directly calculated based on the two, which can greatly improve the optimization speed. Therefore, it is also the most commonly used form. For the parameter update of the reverse combination algorithm, the following form is used:

[0096]

[0097] Combined features are used to optimize the reverse combined tracking. The pixel points with large image gradient change rate are obtained by using the corner feature point detection algorithm on the image. Then these corner feature points are tracked in the plane image, using a simplified transformation function:

[0098]

[0099] Among them, u and v represent plane coordinates, T represents transposition, and p1 and p2 represent the changes in the corresponding plane coordinates.

[0100] After simplifying the tracking, the plane area before and after the tracking is affine transformed, and then the calculation result of the affine transformation is used as the initial value of the calculation of the plane reverse combined tracking W(x:p).

[0101] Step S805, 3D reconstruction and modeling: 3D modeling and reconstruction of the scene is performed using a 3D reconstruction algorithm through the point cloud data and plane constraint information in the 3D reconstruction. The 3D model is then evaluated and optimized to improve the accuracy and precision of the model.

[0102] In the actual implementation process, the 3D reconstruction results are optimized, including removing unreasonable points, adjusting the model topology, etc., and then the reconstruction results are output to ensure the accuracy and precision of the model.

[0103] Scenario Example 3

[0104] In the above scenario embodiments, the principle and process of the three-dimensional reconstruction method of the image are specifically introduced. In the third scenario embodiment, an example is given in combination with an actual scenario. The details are as follows:

[0105] Step 1: 3D reconstruction is the process of recovering the 3D structure of a scene from a 2D image or video. Typically, 3D reconstruction requires using multiple images or videos, matching the features in these images or videos through computer vision algorithms, and then estimating the 3D structure of the scene. The goal of 3D reconstruction is to create an accurate 3D model of a scene, which can be applied to fields such as virtual reality, augmented reality, architecture, and urban planning.

[0106] Simultaneous Localization and Mapping (SLAM) refers to the simultaneous positioning and mapping, estimating the position of a robot or mobile device in an unknown environment and the three-dimensional structure of the scene in real time. SLAM uses sensors (such as cameras, lidar, and inertial measurement units) to collect environmental information, and uses algorithms to achieve real-time positioning and mapping of robots or mobile devices in the environment. The purpose of SLAM is to achieve autonomous navigation and environmental understanding, and can be applied to robotics, autonomous driving, drones and other fields.

[0107] 3D reconstruction focuses on recovering 3D scene structures from static images or videos, while SLAM focuses on real-time positioning and mapping in dynamic environments. The two technologies have some similarities in implementation, but their application areas and purposes are different.

[0108] Therefore, the method of modeling the target scene such as a museum or an art gallery adopts three-dimensional reconstruction. The three-dimensional reconstruction collects video images of the scene, pays attention to the collection technique and speed, and pre-processes the image information after the collection by removing noise, adjusting contrast and brightness, removing motion blur and other technical means. The scene using the present invention includes a large number of indoor plane images. The physical dimensions of the plane image, including length and width, can be obtained by manual measurement. The target plane image is imaged and its physical dimensions are collected.

[0109] Step 2: Extract features from the plane image, including:

[0110] (1) Scale space construction: First, the SIFT algorithm constructs a scale space by using the Gaussian difference function to detect image features at different scales. In the scale space, the image is convolved with different Gaussian kernels to obtain a series of image pyramids, each of which represents a different scale.

[0111] (2) Key point detection: Next, the SIFT algorithm searches for key points with stable features in the scale space. First, at each pixel point in each scale space, the Gaussian gradient and direction of the pixel point are obtained by performing Gaussian weighted averaging on the surrounding pixels. Then, the algorithm detects local extreme points in each pyramid layer, that is, in the 3x3x3 pixel block around the current pixel point, a pixel point is found whose value is the maximum or minimum in the pixel block. This pixel point is the local extreme point. Finally, by comparing the gradient size and direction of the pixel points in the neighborhood, key points with stable features are screened out.

[0112] (3) Direction assignment: After determining the key points, the SIFT algorithm assigns a main direction to each key point for subsequent feature description. Specifically, the algorithm calculates the gradient direction histogram in the neighborhood around the key point, then selects the largest direction as the main direction of the key point, and selects additional directions relative to this direction.

[0113] (4) Feature description: Finally, the SIFT algorithm describes the features of each key point for subsequent feature matching. Specifically, the algorithm constructs 16x16 pixel blocks based on the main direction in the neighborhood around the key point and calculates the gradient amplitude and direction of the pixels in each pixel block. Then, each pixel block is divided into 4x4 sub-blocks, and the pixels in each sub-block are rotated according to the main direction to form an 8-dimensional gradient histogram. Finally, the gradient histograms of all sub-blocks are connected in series to form a 128-dimensional feature vector.

[0114] And in the feature extraction process, the KD-Tree algorithm is used to construct a retrieval tree for the features of the target plane image. The process is:

[0115] Construct KD-Tree: All descriptors are used as input data of KD-Tree, and KD-Tree is constructed recursively. In the process of constructing KD-Tree, one dimension (i.e. one dimension of the feature vector) is selected as the splitting dimension each time, and the median of the current dimension is selected as the splitting point. In this way, each node divides its child nodes into two subsets, the left child node contains data less than or equal to the splitting point, and the right child node contains data greater than the splitting point.

[0116] Query nearest neighbor: For a given query descriptor, determine the location of the query point in the KD-Tree by recursively visiting the KD-Tree. Then, starting from the current node, calculate the distance between the query point and the current node by a distance metric (such as Euclidean distance), as well as the distance from the shortest distance between the query point and the current node to the hyperplane of the node. Based on these distances, the subtree where the nearest neighbor of the query point is located can be determined, and the nearest neighbor is recursively found in the subtree.

[0117] Select the nearest neighbor: Finally, traverse the entire KD-Tree and select the nearest neighbor point as the matching point of the query point. The nearest neighbor point can be determined by calculating the distance between descriptors (such as Euclidean distance or cosine similarity).

[0118] Step 3: Plane matching of video images. First, use the feature matching algorithm to find the corresponding feature points or feature descriptors in the two images. Then, based on the correspondence between the camera intrinsic parameters and the feature points or feature descriptors, calculate the position and posture of the camera in three-dimensional space (such as the rotation matrix and translation vector of the camera). Finally, the pose estimation results can be further optimized using pose optimization algorithms (such as the Levenberg-Marquardt algorithm) to improve accuracy and robustness. And calculate warp (x: p) based on the plane matching results and use it as the initial value of the tracking algorithm for optimization.

[0119] Step 4: Perform reverse combined tracking based on combined features on the identified plane in the video image, and use the tracking results to optimize the 3D reconstruction. The reverse combined tracking based on combined features must first obtain the pixel points with large image gradient change rate through the corner feature point detection algorithm of the image. Then track these corner feature points in the plane image, and for each pixel, select a local window of size n×n in the first frame image. Then calculate the grayscale gradient of all pixels in the two windows at the corresponding window position in the second frame image. Construct an n×n Jacobian matrix, in which each row contains the x, y partial derivatives and time derivatives of a pixel in the window in the first frame image, with a total of n×n rows. Calculate the pseudo-inverse matrix of the Jacobian matrix, that is, (JTJ)-1JT, where J represents the transpose of the Jacobian matrix and T represents the transpose of the matrix. Calculate the grayscale change of all pixels in the window in the first frame image, that is, I(x+dx,y+dy,t+dt)-I(x,y,t) in the equation, where dx and dy represent the displacement of the pixel in the x and y directions, and dt represents the time interval between two frames of images. The grayscale change is formed into an n×1 vector, and the product of the vector and the pseudo-inverse matrix of the Jacobian matrix is ​​calculated to obtain the pixel displacement vector (dx,dy). Then perform reverse combined tracking.

[0120] (1) Select a local template to be identified in the reference frame image and calculate the grayscale average of the template.

[0121] (2) At the corresponding template position in the current frame image, calculate the grayscale gradient of all pixels in the two templates.

[0122] (3) Construct an n×n Jacobian matrix, where each row contains the x and y partial derivatives of a pixel in the template in the reference frame image, with a total of n×n rows.

[0123] (4) Calculate the pseudo-inverse matrix of the Jacobian matrix, that is, (JTJ)-1JT, where J represents the transpose of the Jacobian matrix and T represents the transpose of the matrix.

[0124] (5) Calculate the grayscale change of all pixels in the template in the current frame image, that is, I(x+dx,y+dy)-I(x,y) in the optical flow equation, where dx and dy represent the displacement of the pixel in the x and y directions.

[0125] (6) The grayscale change is formed into an n×1 vector, and the product of the vector and the pseudo-inverse matrix of the Jacobian matrix is ​​calculated to obtain the pixel displacement vector (dx, dy).

[0126] (7) Update the position and shape of the template, add the displacement vector to the position of the template in the current frame image, and then recalculate the grayscale average of the template.

[0127] (8) Repeat steps (2) to (7) until the position and shape of the template are stable or the preset number of iterations is reached.

[0128] Step 5: Use plane parameters and point cloud data to build a plane constraint optimization model, and optimize the reconstruction results by minimizing the plane constraint error. Common plane constraint optimization methods include energy function-based methods, graph optimization-based methods, and iterative closest point-based methods.

[0129] In actual implementation, the physical size of the natural marker can be extracted by replacing the manual measurement of the plane image with that of a 3D laser rangefinder.

[0130] In summary, the three-dimensional reconstruction method of an image provided by an embodiment of the present invention can be widely used in environmental scenes such as architectural design, cultural relics protection, virtual reality and intelligent transportation. Specifically, for building reconstruction: the three-dimensional reconstruction technology based on natural feature planes can be used for three-dimensional reconstruction of buildings, such as urban buildings, historical buildings, etc. By extracting the natural feature planes of the building, such as walls, roofs, etc., a three-dimensional model of the building can be constructed, which can be used in urban planning, cultural heritage protection, etc. For terrain reconstruction: the three-dimensional reconstruction technology based on natural feature planes can be used for three-dimensional reconstruction of terrain, such as mountains, rivers, lakes, etc. By extracting the natural feature planes of the terrain, such as mountains, water systems, etc., a three-dimensional model of the terrain can be constructed, which can be used in geological exploration, natural resource management, etc. For industrial design: the three-dimensional reconstruction technology based on natural feature planes can be used for three-dimensional modeling in industrial design, such as automobile shells, machine parts, etc. By extracting the natural feature planes of the design object, its three-dimensional model can be constructed, which can be used in design, manufacturing, etc. For visual effects and game development: the three-dimensional reconstruction technology based on natural feature planes can be used for three-dimensional modeling in visual effects and game development, such as movie special effects, game scenes, etc. By extracting the natural feature planes in the scene, its three-dimensional model can be constructed, which can be used for visual effects and game development. The three-dimensional reconstruction method of the embodiment of the present invention has the following advantages: 1. High-precision three-dimensional reconstruction: This technology can realize the accurate extraction and segmentation of the planes in the scene through the constraints of natural markers, thereby realizing high-precision three-dimensional reconstruction. This is very important for application scenarios that require precise measurement and modeling, such as digital protection of cultural relics, three-dimensional measurement of buildings, and virtual reality game development. 2. Save time and effort: Compared with traditional three-dimensional reconstruction methods, the plane-constrained three-dimensional reconstruction technology based on natural markers can greatly save time and labor costs. The detection and extraction of natural markers can be automatically completed by computer vision and deep learning algorithms without manual intervention. Therefore, this technology can realize three-dimensional reconstruction faster and more efficiently. 3. Wide applicability: The plane-constrained three-dimensional reconstruction technology based on natural markers is applicable to a variety of different application scenarios, such as digital protection of cultural relics, three-dimensional measurement of buildings, virtual reality game development, intelligent transportation systems, etc. Therefore, this technology has strong versatility and application value. 4. Strong real-time performance: Since the plane-constrained 3D reconstruction technology based on natural markers can quickly achieve 3D reconstruction, it can realize application scenarios with strong real-time performance, such as virtual reality games and intelligent transportation systems. This can improve the practicality and user experience of the application. 5. High security: This technology can ensure the security and integrity of data through encryption and verification of natural markers. This is very important for application scenarios that require data privacy and security, such as intelligent transportation systems and other fields.In summary, the plane-constrained 3D reconstruction technology based on natural markers has many beneficial effects such as high precision, time and labor saving, wide applicability, strong real-time performance and high security, which can bring important technical and economic value to many application scenarios.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method of an image, characterized in that: include: Acquire image data, and perform feature extraction on the image data to acquire image features; Performing natural mark detection on the environment scene corresponding to the image data to obtain the natural mark of the environment scene; Extracting a target plane in the environment scene according to the image features and the natural markers; The three-dimensional position information of the natural marker is used as plane constraint information to perform three-dimensional reconstruction on the image data and the target plane.

2. The method according to claim 1, characterized in that Before using the three-dimensional position information of the natural marker as plane constraint information to perform three-dimensional reconstruction on the image data and the target plane, the method further includes: Combined feature tracking is performed on the target plane to obtain the plane constraint information of all the image data in the video image, wherein the combined feature tracking includes point feature tracking and area feature tracking.

3. The method according to claim 2, characterized in that Performing combined feature tracking on the target plane includes: Using a corner feature point detection algorithm to obtain an image gradient change rate of a pixel point in the image data, and performing point feature tracking on the pixel point whose image gradient change rate is greater than or equal to an image gradient change rate threshold; An image area having an average pixel grayscale value equal to that of a preset local recognition template is selected from the image data of the current frame as a target area, and regional feature tracking is performed on the target area.

4. The method according to claim 1, characterized in that: The natural marker detection includes at least one of the following: Corner detection; Edge detection; Plane detection.

5. The method according to claim 1, characterized in that After extracting the target plane in the environment scene according to the image features and the natural markers, the method further includes: The target plane is triangulated, and a nonlinear optimization algorithm is used to perform fitting optimization on the target plane according to the triangulation.

6. The method according to claim 1, characterized in that After obtaining the natural markers of the environment scene, the method further includes: Feature extraction is performed on the natural marker, and feature matching retrieval is performed on the natural marker and the environmental scene based on the feature extraction to confirm that the natural marker comes from the environmental scene.

7. A three-dimensional reconstruction device for an image, characterized in that: include: An image acquisition module is used to acquire image data and perform feature extraction on the image data to acquire image features; A natural marking module, used to perform natural marking detection on the environment scene corresponding to the image data to obtain the natural marking of the environment scene; A plane extraction module, used for extracting a target plane in the environment scene according to the image features and the natural markers; The three-dimensional reconstruction module is used to use the three-dimensional position information of the natural marker as plane constraint information to perform three-dimensional reconstruction on the image data and the target plane.

8. The device according to claim 7, characterized in that Also includes: The combined tracking module is used to perform combined feature tracking on the target plane to obtain the plane constraint information of all the image data in the video image, wherein the combined feature tracking includes point feature tracking and area feature tracking.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the method described in any one of claims 1 to 6 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method described in any one of claims 1 to 6 is implemented.

Images