A three-dimensional reconstruction optimization method and device based on virtual image plane feature adjustment

Abstract

The application discloses a three-dimensional reconstruction optimization method based on virtual image plane feature adjustment, comprising the following steps: acquiring a multi-view real image set, calculating the matching suitability of each real image pair in the set; planning a virtual photography path between the real image pair with the highest matching suitability, generating an initial virtual image based on the virtual photography path; extracting features from the real image pair and the initial virtual image, adjusting the feature pixels of the real image pair and the initial virtual image based on the extracted features; the application can effectively improve the positioning accuracy and matching reliability of the feature pixels, and provides stable and accurate feature matching pixels for three-dimensional reconstruction.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction optimization method, apparatus, equipment and medium based on virtual image plane feature adjustment. Background Technology

[0002] Visual 3D scene reconstruction is one of the core tasks in computer vision, robot navigation, augmented reality and other fields. Its goal is to recover the 3D geometric structure and texture information of a scene or object from a set of 2D images and generate a 3D model that can be used for measurement, analysis and display. This technology has extremely important application value in urban modeling, cultural relic protection, autonomous driving, virtual reality and other fields.

[0003] Feature pixel matching is a crucial step in constructing the entire 3D model. The accuracy, robustness, and completeness of feature pixel matching directly determine the quality of the final 3D model. However, in practical applications, when the viewpoint differences between input images are too large, the performance of traditional feature matching methods will drop sharply, thus affecting subsequent 3D reconstruction steps.

[0004] Existing methods for addressing matching difficulties caused by significant viewpoint differences mainly fall into two categories. The first category utilizes geometric constraints and viewpoint transformations to reduce imaging differences. These methods are typically based on planar structural projection, achieving initial alignment between images through projection transformations. While this can improve matching coverage, its performance is highly dependent on the presence and consistency of planar structures in the scene. It is prone to matching errors or projection failures when dealing with complex structures such as curved facades and arc-shaped buildings. The second category is based on data-driven feature extraction and matching methods, which are increasingly being applied to SfM tasks with significant viewpoint differences, demonstrating strong representation potential. However, they still face bottlenecks in efficiency and stability in large-scale engineering applications. Therefore, there is a lack of a high-precision, high-reliability feature matching optimization method for 3D reconstruction of multi-view images with large viewpoint differences. Summary of the Invention

[0005] This invention provides a three-dimensional reconstruction method based on virtual images, which can effectively improve the positioning accuracy and matching reliability of feature pixels, and provide stable and accurate feature matching pixels for three-dimensional reconstruction.

[0006] In a first aspect, embodiments of the present invention provide a three-dimensional reconstruction optimization method based on virtual image plane feature adjustment, comprising: Obtain a set of real images from multiple perspectives and calculate the matching suitability of each real image pair in the set; A virtual photography path is planned between the real image pairs with the highest matching suitability, and an initial virtual image is generated based on the virtual photography path; Feature extraction is performed on the real image pair and the initial virtual image, and the feature pixels of the real image pair and the initial virtual image are adjusted based on the extracted features.

[0007] Furthermore, the acquisition of a set of real images from multiple perspectives, and the calculation of the matching suitability of each real image pair in the set, includes: A set of real images from multiple perspectives is obtained, and the interior and exterior orientation elements of each image in the set are recovered using the SfM method to generate a sparse 3D point cloud. For all real image pairs in the real image set, calculate the baseline length, overlap, and viewing angle difference. Real image pairs with a viewing angle difference greater than a preset viewing angle difference threshold are selected. Based on the baseline length, the overlap, the viewing angle difference, and the preset weight coefficient, the matching suitability of each real image pair is calculated.

[0008] Furthermore, generating the initial virtual image based on the virtual photography path includes: Based on the virtual photography path, the camera pose of each virtual image plane is generated sequentially; The camera pose is rendered using a 3D model to generate an initial virtual image.

[0009] Furthermore, after generating the initial virtual image based on the virtual photography path, the process further includes: Using a pre-trained diffusion model, the initial virtual image is repaired, super-resolution is applied, and artifact removal is performed to generate a high-quality virtual image.

[0010] Furthermore, The real image pair includes a first real image and a second real image. Therefore, the step of extracting features from the real image pair and the initial virtual image, and adjusting the feature pixels of the real image pair and the initial virtual image based on the extracted features, includes: Step 1: Extract features from the first real image, the second real image, and the initial virtual image. The features include pixel coordinates and feature descriptors. Step 2: Calculate the feature matching relationship between each real image and the initial virtual image to obtain initial feature matching pairs; Step 3: Based on the initial feature matching pairs, use the dense optical flow algorithm to calculate the dense correspondence between the feature pixels of each real image and the initial virtual image; Step 4: Based on the dense correspondence, project the feature pixels of the first real image onto the initial virtual image to obtain the feature point coordinates corresponding to the initial virtual image, and correct the feature pixels of the second real image based on the feature point coordinates. Step 5: Based on the adjusted pixel coordinates of the feature pixels, re-extract the corresponding feature descriptors; Step 6: Repeat steps 2 to 5 above until the preset iteration termination condition is met.

[0011] Furthermore, The calculation of the feature matching relationship between each real image and the initial virtual image to obtain the initial feature matching pair includes: For each real image, all feature points are obtained. Based on the similarity function, the feature points that match the real image are found. The matched feature points are recorded as positive matching pairs. For all feature points in the initial virtual image, the feature points that match the real image are found based on the similarity function. The two feature points are recorded as negative matching pairs. Feature pairs that appear in both the forward matching pair and the reverse matching pair are retained as initial feature matching pairs.

[0012] Furthermore, the method also includes: After each iteration, the uncertainty of the corrected feature pixels is calculated.

[0013] Secondly, embodiments of the present invention provide a three-dimensional reconstruction optimization device based on virtual image plane feature adjustment, comprising: The matching suitability calculation module is used to acquire a set of real images from multiple perspectives and calculate the matching suitability of each real image pair in the set. The virtual image generation module is used to plan a virtual photography path between the real image pairs with the highest matching suitability, and generate an initial virtual image based on the virtual photography path. The feature pixel adjustment module is used to extract features from the real image pair and the initial virtual image, and adjust the feature pixels of the real image pair and the initial virtual image based on the extracted features.

[0014] Thirdly, embodiments of the present invention provide an electronic device, comprising: Memory, used to store computer programs; A processor for executing the computer program; Wherein, when the processor executes the computer program, it implements the three-dimensional reconstruction optimization method based on virtual image plane feature adjustment as described in any of the first aspects above.

[0015] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed, implements the three-dimensional reconstruction optimization method based on virtual image plane feature adjustment as described in any of the first aspects above.

[0016] Compared with existing technologies, the 3D reconstruction optimization method based on virtual image plane feature adjustment provided by this invention has the following advantages: It acquires a set of real images from multiple perspectives and calculates the matching suitability of each real image pair in the set; it plans a virtual photography path between the real image pairs with the highest matching suitability and generates an initial virtual image based on the virtual photography path; it extracts features from the real image pairs and the initial virtual image, and adjusts the feature pixels of the real image pairs and the initial virtual image based on the extracted features; this invention can effectively improve the positioning accuracy and matching reliability of feature pixels, providing stable and accurate feature matching pixels for 3D reconstruction. Attached Figure Description

[0017] To more clearly illustrate the technical features of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a three-dimensional reconstruction optimization method based on virtual image plane feature adjustment provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a three-dimensional reconstruction optimization device based on virtual image plane feature adjustment provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0022] In a first aspect, embodiments of the present invention provide a three-dimensional reconstruction optimization method based on virtual image plane feature adjustment, see [link to previous section]. Figure 1 This is a flowchart illustrating an embodiment of a three-dimensional reconstruction optimization method based on virtual image plane feature adjustment provided by the present invention.

[0023] like Figure 1 As shown, the method includes the following steps: S1: Obtain a set of real images from multiple perspectives and calculate the matching suitability of each real image pair in the set; S2: Plan a virtual photography path between the real image pairs with the highest matching suitability, and generate an initial virtual image based on the virtual photography path; S3: Extract features from the real image pair and the initial virtual image, and adjust the feature pixels of the real image pair and the initial virtual image based on the extracted features.

[0024] In practice, a collection of real images from multiple perspectives (e.g., a series of images of the same object taken from different positions and angles) is acquired to ensure that there is a certain overlap between the images. The matching suitability of any two real images in the collection is calculated, and the most suitable real image pair for generating virtual images is selected. A feature transfer bridge is built between the image pairs with the highest matching suitability. One or more progressive virtual photography paths are planned with the camera poses of the two as the starting and ending points, and the initial virtual image is generated based on the virtual photography path.

[0025] Multi-level features are extracted from real images and initial virtual images to establish feature correspondence between real and virtual images. Using virtual images as a bridge, the coordinate data of feature points are corrected through the bidirectional flow of features between real and virtual images, eliminating matching errors and making feature matching between real image pairs more accurate and stable.

[0026] In summary, this invention calculates the matching suitability of each pair of images in a multi-view real image set, accurately selects the most suitable image pair, laying a good foundation for subsequent operations, plans virtual photography paths and generates initial virtual images to ensure that the virtual image generation conforms to the actual shooting logic, extracts features from real images and initial virtual images and adjusts feature pixels, effectively improving the positioning accuracy and matching reliability of feature pixels, and finally achieves high-precision and high-reliability feature matching under large viewpoint differences, providing stable and reliable feature matching pixels for 3D reconstruction, and can more realistically and accurately restore the 3D structure of the target scene.

[0027] In one optional implementation, acquiring a set of real images from multiple perspectives and calculating the matching suitability of each real image pair in the set includes: A collection of real images from multiple perspectives is obtained, and the interior and exterior orientation elements of each image in the collection are recovered using the SfM method to generate a sparse 3D point cloud. For all real image pairs in the real image set, calculate the baseline length, overlap, and viewing angle difference. Real image pairs with a viewing angle difference greater than a preset viewing angle difference threshold are selected. Based on the baseline length, the overlap, the viewing angle difference, and the preset weight coefficient, the matching suitability of each real image pair is calculated.

[0028] Specifically, a set of real images from multiple perspectives is acquired, and the Structure from Motion (SfM) method is used to recover the interior orientation elements (including focal length) of each image in the set. Principal point coordinates Distortion parameters ) and exterior orientation elements (including rotation matrices) Translation vector ,in Simultaneously, sparse 3D point clouds are generated. By utilizing sparse 3D point cloud alignment, the geometric relationship of all images under a unified world coordinate system is established.

[0029] For all real image pairs in the real image set, calculate their baseline length, overlap, and viewpoint difference angle, where the baseline length is... The baseline length refers to the distance between the optical centers of the cameras that captured the two images. A baseline that is too long or too short will affect the image matching results. An excessively long baseline may result in a small overlap between the images, making feature matching difficult; a baseline that is too short may make the differences between the images indistinct, making it difficult to distinguish different feature points. The formula is as follows: ; in, and The optical centers of the camera are the two images.

[0030] Overlap ratio refers to the proportion of the overlapping area between two images to the total area of ​​the images. The higher the overlap ratio, the more feature points can be matched between the images, and the higher the reliability of the matching. Conversely, if the overlap ratio is too low, there may be insufficient matching points, which will affect the accuracy of the matching.

[0031] Angle of view The angle between the viewing angles of two images is considered. When the angle difference is too large, the shape and texture of objects in the images will change significantly, increasing the difficulty of matching. If the angle difference is too small, it may not provide enough information for effective matching. The formula is as follows: ; in, and is the direction vector of the camera's optical axis.

[0032] Real image pairs with a viewing angle difference greater than a preset viewing angle difference threshold are selected. For each selected real image pair, the matching suitability is calculated based on baseline length, overlap, viewing angle difference, and a preset weighting coefficient. The specific calculation formula is as follows: ; in, As the normalization factor, For the weighting coefficients, satisfying , Used to prioritize image pairs with high overlap. Used to prioritize image pairs with appropriate baseline distances. Used to prioritize image pairs whose viewing angle differences are close to a threshold.

[0033] This embodiment first uses the SfM method to recover the interior and exterior orientation elements of each image and generate a sparse 3D point cloud, providing basic data support for subsequent analysis and enabling a more accurate grasp of the spatial information of the image. Then, it obtains the baseline length, overlap, and viewing angle difference of the real image pair. Based on these parameters and pre-set weight coefficients, it calculates the matching suitability. By comprehensively considering multiple factors, it can accurately select the most suitable image pair for subsequent processing, laying a good foundation for subsequent virtual image generation and 3D reconstruction operations.

[0034] In one optional implementation, generating the initial virtual image based on the virtual photography path includes: Based on the virtual photography path, the camera pose of each virtual image plane is generated sequentially; The camera pose is rendered using a 3D model to generate an initial virtual image.

[0035] Specifically, each path consists of a series of consecutive virtual image plane poses: ; in, Indicates the first The pose of a virtual camera.

[0036] Along virtual path The camera pose for each virtual image plane is generated sequentially, and the viewpoint change between adjacent poses is controlled within a preset threshold. Internally, to ensure continuity: ; Meanwhile, the distance between the optical centers of adjacent virtual cameras satisfies the following formula, which is used to limit the movement step size of the virtual cameras and avoid position jumps: ; Using a tilted 3D model Or 3D Gaussian sputtering model Based on the virtual camera pose and internal reference Perform perspective projection rendering to generate an initial virtual image.

[0037] This embodiment generates the camera pose of each virtual image plane sequentially according to the virtual photography path, which can accurately simulate the camera position and posture that conforms to the actual shooting logic. Then, the 3D model is used to render the camera pose to generate the initial virtual image, making the generated initial virtual image more realistic and accurate in terms of content, structure and spatial representation. This provides a solid and reliable foundation for subsequent adjustments to the initial virtual image based on real images and the final high-quality 3D reconstruction.

[0038] In one optional implementation, after generating the initial virtual image based on the virtual photography path, the method further includes: Using a pre-trained diffusion model, the initial virtual image is repaired, super-resolution is applied, and artifact removal is performed to generate a high-quality virtual image.

[0039] Specifically, the initial virtual image generated in the above embodiments still has some defects. Only after processing with a diffusion model can it serve as a reliable bridge for feature transfer. It is a generative AI model based on denoising. After pre-training, it has the ability to repair defects and restore real details. The model generates high-quality virtual images by performing operations such as denoising, improving image resolution, enhancing texture details, and eliminating rendering artifacts on the initial virtual image.

[0040] This embodiment repairs and fills in any defects or missing parts that may exist in the initial virtual image. Super-resolution processing can significantly improve the image's clarity and detail, while artifact removal effectively eliminates unrealistic traces or interference information in the image, ultimately generating a high-quality virtual image. This provides a more accurate, reliable, and visually better foundation for subsequent 3D reconstruction and other work based on the image, helping to improve the final quality of the entire process.

[0041] In one alternative implementation, The real image pair includes a first real image and a second real image. Therefore, the step of extracting features from the real image pair and the initial virtual image, and adjusting the feature pixels of the real image pair and the initial virtual image based on the extracted features, includes: Step 1: Extract features from the first real image, the second real image, and the initial virtual image. The features include pixel coordinates and feature descriptors. Step 2: Calculate the feature matching relationship between each real image and the initial virtual image to obtain initial feature matching pairs; Step 3: Based on the initial feature matching pairs, use the dense optical flow algorithm to calculate the dense correspondence between the feature pixels of each real image and the initial virtual image; Step 4: Based on the dense correspondence, project the feature pixels of the first real image onto the initial virtual image to obtain the feature point coordinates corresponding to the initial virtual image, and correct the feature pixels of the second real image based on the feature point coordinates. Step 5: Based on the adjusted pixel coordinates of the feature pixels, re-extract the corresponding feature descriptors; Step 6: Repeat steps 2 to 5 above until the preset iteration termination condition is met.

[0042] Specifically, feature extraction is performed on real images and initial virtual images, including pixel coordinates and feature descriptors. Pixel coordinates record the spatial location of each feature point, while descriptors (such as SIFT, SURF, or deep learning features) encode semantic information such as local texture and edges, providing a basis for subsequent matching.

[0043] Based on the extracted features, the feature matching relationship between the real image and the initial virtual image is calculated to obtain the initial feature matching pairs and form a sparse correspondence.

[0044] Based on sparse matching pairs and their uncertain weights, dense matching algorithms (such as PDCNet) are used to calculate dense correspondences, as shown below: ; in, For optical flow field, This represents the matching confidence score for position (u,v). is the initial feature matching pair, and w is the uncertainty weight.

[0045] Based on the dense correspondence, the feature pixels of the first real image are projected onto the initial virtual image to obtain more accurate feature point positions on the virtual image, as shown below: ; in, These are the feature pixels projected onto the virtual image. These are the feature pixels of the real image.

[0046] Then, by utilizing the dense optical flow between the second real image and the virtual image, the feature points on the virtual image are projected onto the second real image. The original feature point coordinates of the second real image are compared with the more accurate coordinates obtained through the virtual image, and the feature point coordinate data of the second real image are updated.

[0047] Based on the adjusted pixel coordinates of the feature pixels, the image patch is cropped with the new coordinates as the center, and the feature descriptors are re-extracted to ensure that the feature descriptors are consistent with the new positions.

[0048] Repeat the matching, projection, adjustment, and re-extraction steps to gradually refine the alignment accuracy, and finally obtain the adjusted feature pixels.

[0049] This embodiment uses virtual images as an intermediary to transfer feature points from the first real image to the second real image, thereby correcting the coordinates of the feature points in the second real image. At the same time, it continuously iterates to make the position of the feature points more and more accurate.

[0050] In one alternative implementation, The calculation of the feature matching relationship between each real image and the initial virtual image to obtain the initial feature matching pair includes: For each real image, all feature points are obtained. Based on the similarity function, the feature points that match the real image are found. The matched feature points are recorded as positive matching pairs. For all feature points in the initial virtual image, the feature points that match the real image are found based on the similarity function. The two feature points are recorded as negative matching pairs. Feature pairs that appear in both the forward matching pair and the reverse matching pair are retained as initial feature matching pairs.

[0051] Specifically, after obtaining the feature descriptors of real and virtual images, a similarity measurement method is used to measure the degree of similarity between them. A similarity threshold is preset. When the cosine similarity of the descriptors of a pair of feature points in the real and virtual images exceeds the threshold, it is preliminarily considered that the pair of feature points are matched.

[0052] After completing the preliminary matching based on cosine similarity, a forward matching check is first performed. That is, for each feature point in the real image, a matching feature point is found in the virtual image (judged by the cosine similarity exceeding the threshold), and these matching pairs are recorded. This step starts from the perspective of the real image and finds the feature points that may correspond to it in the virtual image.

[0053] Next, a reverse matching check is performed, which is the opposite of the forward matching check. For each feature point in the virtual image, a matching feature point is found in the real image, and these matching pairs are recorded. This step starts from the perspective of the virtual image and looks for feature points in the real image that may correspond to it.

[0054] Only when a pair of feature points is determined to be a match in both the forward matching check (real image → virtual image) and the reverse matching check (virtual image → real image) is the pair of feature points retained as the final reliable match pair. This bidirectional consistency check method can effectively eliminate those erroneous match pairs that are only matched in one direction, because a true match pair should have corresponding feature points in both directions.

[0055] This embodiment calculates the feature matching relationship between real and virtual images and performs a two-way consistency check, which can effectively obtain a relatively reliable initial matching pair, laying a solid foundation for subsequent image processing.

[0056] In an optional implementation, the method further includes: After each iteration, the uncertainty of the corrected feature pixels is calculated.

[0057] Specifically, each iteration corrects pixel positions using methods such as dense optical flow and local re-extraction of features, which may introduce new errors, such as optical flow estimation errors and local feature extraction noise. Therefore, after each iteration, the uncertainty of the corrected feature pixels is calculated and updated, which not only ensures the accuracy of feature matching but also improves the robustness and efficiency of the iteration process.

[0058] Secondly, embodiments of the present invention provide a three-dimensional reconstruction optimization device based on virtual image plane feature adjustment, see [link to relevant documentation]. Figure 2 This is a schematic diagram of an embodiment of a three-dimensional reconstruction optimization device based on virtual image plane feature adjustment provided by the present invention.

[0059] like Figure 2 As shown, the device includes: The matching suitability calculation module 21 is used to obtain a set of real images from multiple perspectives and calculate the matching suitability of each real image pair in the set. Virtual image generation module 22 is used to plan a virtual photography path between the real image pairs with the highest matching suitability, and generate an initial virtual image based on the virtual photography path; The feature pixel adjustment module 23 is used to extract features from the real image pair and the initial virtual image, and adjust the feature pixels of the real image pair and the initial virtual image based on the extracted features.

[0060] In one optional implementation, acquiring a set of real images from multiple perspectives and calculating the matching suitability of each real image pair in the set includes: A collection of real images from multiple perspectives is obtained, and the interior and exterior orientation elements of each image in the collection are recovered using the SfM method to generate a sparse 3D point cloud. For all real image pairs in the real image set, calculate the baseline length, overlap, and viewing angle difference. Real image pairs with a viewing angle difference greater than a preset viewing angle difference threshold are selected. Based on the baseline length, the overlap, the viewing angle difference, and the preset weight coefficient, the matching suitability of each real image pair is calculated.

[0061] In one optional implementation, generating the initial virtual image based on the virtual photography path includes: Based on the virtual photography path, the camera pose of each virtual image plane is generated sequentially; The camera pose is rendered using a 3D model to generate an initial virtual image.

[0062] In one optional implementation, after generating the initial virtual image based on the virtual photography path, the method further includes: Using a pre-trained diffusion model, the initial virtual image is repaired, super-resolution is applied, and artifact removal is performed to generate a high-quality virtual image.

[0063] In one alternative implementation, The real image pair includes a first real image and a second real image. Therefore, the step of extracting features from the real image pair and the initial virtual image, and adjusting the feature pixels of the real image pair and the initial virtual image based on the extracted features, includes: Step 1: Extract features from the first real image, the second real image, and the initial virtual image. The features include pixel coordinates and feature descriptors. Step 2: Calculate the feature matching relationship between each real image and the initial virtual image to obtain initial feature matching pairs; Step 3: Based on the initial feature matching pairs, use the dense optical flow algorithm to calculate the dense correspondence between the feature pixels of each real image and the initial virtual image; Step 4: Based on the dense correspondence, project the feature pixels of the first real image onto the initial virtual image to obtain the feature point coordinates corresponding to the initial virtual image, and correct the feature pixels of the second real image based on the feature point coordinates. Step 5: Based on the adjusted pixel coordinates of the feature pixels, re-extract the corresponding feature descriptors; Step 6: Repeat steps 2 to 5 above until the preset iteration termination condition is met.

[0064] In one alternative implementation, The calculation of the feature matching relationship between each real image and the initial virtual image to obtain the initial feature matching pair includes: For each real image, all feature points are obtained. Based on the similarity function, the feature points that match the real image are found. The matched feature points are recorded as positive matching pairs. For all feature points in the initial virtual image, the feature points that match the real image are found based on the similarity function. The two feature points are recorded as negative matching pairs. Feature pairs that appear in both the forward matching pair and the reverse matching pair are retained as initial feature matching pairs.

[0065] In an optional embodiment, the device is further configured to: After each iteration, the uncertainty of the corrected feature pixels is calculated.

[0066] Thirdly, embodiments of the present invention provide an electronic device, see [link to previous document]. Figure 3 The diagram shown is a structural schematic of an electronic device provided in an embodiment of the present invention.

[0067] like Figure 3 As shown, the device includes: Memory 31 is used to store computer programs; Processor 32 is used to execute the computer program; When the processor 32 executes the computer program, it implements the three-dimensional reconstruction optimization method based on virtual image plane feature adjustment as described in any of the above embodiments.

[0068] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 32 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0069] The processor 32 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0070] The memory 31 can be used to store the computer programs and / or modules. The processor 32 implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory 31 and calling the data stored in the memory 31. The memory 31 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 31 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0071] It should be noted that the aforementioned electronic devices include, but are not limited to, processors and memory, as will be understood by those skilled in the art. Figure 3 The structural diagram is merely an example of the electronic device described above and does not constitute a limitation on the electronic device. It may include more components than shown in the diagram, or combine certain components, or use different components.

[0072] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed, implements the three-dimensional reconstruction optimization method based on virtual image plane feature adjustment as described in any of the above embodiments.

[0073] It should be understood that the present invention can implement all or part of the processes in the above-described 3D reconstruction optimization method based on virtual image plane feature adjustment, or it can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-described 3D reconstruction optimization method based on virtual image plane feature adjustment. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0074] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. It should be noted that, for those skilled in the art, several equivalent obvious modifications and / or equivalent substitutions can be made without departing from the technical principles of the present invention, and these obvious modifications and / or equivalent substitutions should also be considered within the scope of protection of the present invention.

Claims

1. A three-dimensional reconstruction optimization method based on virtual image plane feature adjustment, characterized in that, include: Obtain a set of real images from multiple perspectives and calculate the matching suitability of each real image pair in the set; A virtual photography path is planned between the real image pairs with the highest matching suitability, and an initial virtual image is generated based on the virtual photography path; Feature extraction is performed on the real image pair and the initial virtual image, and the feature pixels of the real image pair and the initial virtual image are adjusted based on the extracted features.

2. The 3D reconstruction optimization method based on virtual image plane feature adjustment as described in claim 1, characterized in that, The process of acquiring a set of real images from multiple perspectives and calculating the matching suitability of each real image pair in the set includes: A collection of real images from multiple perspectives is obtained, and the interior and exterior orientation elements of each image in the collection are recovered using the SfM method to generate a sparse 3D point cloud. For all real image pairs in the real image set, calculate the baseline length, overlap, and viewing angle difference. Real image pairs with a viewing angle difference greater than a preset viewing angle difference threshold are selected. Based on the baseline length, the overlap, the viewing angle difference, and the preset weight coefficient, the matching suitability of each real image pair is calculated.

3. The 3D reconstruction optimization method based on virtual image plane feature adjustment as described in claim 1, characterized in that, The generation of the initial virtual image based on the virtual photography path includes: Based on the virtual photography path, the camera pose of each virtual image plane is generated sequentially; The camera pose is rendered using a 3D model to generate an initial virtual image.

4. The 3D reconstruction optimization method based on virtual image plane feature adjustment as described in claim 1, characterized in that, After generating the initial virtual image based on the virtual photography path, the process further includes: Using a pre-trained diffusion model, the initial virtual image is repaired, super-resolution is applied, and artifact removal is performed to generate a high-quality virtual image.

5. The 3D reconstruction optimization method based on virtual image plane feature adjustment as described in claim 1, characterized in that, The real image pair includes a first real image and a second real image. Therefore, the step of extracting features from the real image pair and the initial virtual image, and adjusting the feature pixels of the real image pair and the initial virtual image based on the extracted features, includes: Step 1: Extract features from the first real image, the second real image, and the initial virtual image. The features include pixel coordinates and feature descriptors. Step 2: Calculate the feature matching relationship between each real image and the initial virtual image to obtain initial feature matching pairs; Step 3: Based on the initial feature matching pairs, use the dense optical flow algorithm to calculate the dense correspondence between the feature pixels of each real image and the initial virtual image; Step 4: Based on the dense correspondence, project the feature pixels of the first real image onto the initial virtual image to obtain the feature point coordinates corresponding to the initial virtual image, and correct the feature pixels of the second real image based on the feature point coordinates. Step 5: Based on the adjusted pixel coordinates of the feature pixels, re-extract the corresponding feature descriptors; Step 6: Repeat steps 2 to 5 above until the preset iteration termination condition is met.

6. The 3D reconstruction optimization method based on virtual image plane feature adjustment as described in claim 5, characterized in that, The calculation of the feature matching relationship between each real image and the initial virtual image to obtain the initial feature matching pair includes: For each real image, all feature points are obtained. Based on the similarity function, the feature points that match the real image are found. The matched feature points are recorded as positive matching pairs. For all feature points in the initial virtual image, the feature points that match the real image are found based on the similarity function. The two feature points are recorded as negative matching pairs. Feature pairs that appear in both the forward matching pair and the reverse matching pair are retained as initial feature matching pairs.

7. The 3D reconstruction optimization method based on virtual image plane feature adjustment as described in claim 5, characterized in that, The method further includes: After each iteration, the uncertainty of the corrected feature pixels is calculated.

8. A three-dimensional reconstruction optimization device based on virtual image plane feature adjustment, characterized in that, include: The matching suitability calculation module is used to acquire a set of real images from multiple perspectives and calculate the matching suitability of each real image pair in the set. The virtual image generation module is used to plan a virtual photography path between the real image pairs with the highest matching suitability, and generate an initial virtual image based on the virtual photography path. The feature pixel adjustment module is used to extract features from the real image pair and the initial virtual image, and adjust the feature pixels of the real image pair and the initial virtual image based on the extracted features.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program; Wherein, when the processor executes the computer program, it implements the three-dimensional reconstruction optimization method based on virtual image plane feature adjustment as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the three-dimensional reconstruction optimization method based on virtual image plane feature adjustment as described in any one of claims 1 to 7.

Images