Incremental 3D reconstruction method and device based on panoramic image
Through the incremental three-dimensional reconstruction method of panoramic images, the panoramic camera and perspective slicing technology are used to solve the problems of low efficiency and large errors in complex scenes, and efficient and accurate three-dimensional model construction is achieved.
Patent Information
- Application Number
- CN202411412109.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-10-11
AI Technical Summary
The existing three-dimensional reconstruction methods have problems such as low data acquisition efficiency, high computational complexity, large error and poor robustness in complex or repeated texture scenarios. In particular, the coordinated working scheme of planar perspective cameras and multi-sensors are difficult to apply in large scenarios.
Using an incremental three-dimensional reconstruction method based on panoramic images, images are collected and perspective segmented by panoramic cameras are used to construct the reference coordinate system using the matching point pairs of initialized images and candidate images, a three-dimensional model is generated, and the beam adjustment optimization is reduced. error.
It improves the accuracy and robustness of three-dimensional reconstruction, reduces the computational complexity and memory requirements, reduces the number of error feature matching times, and realizes efficient data acquisition and accurate three-dimensional model construction.
Smart Images

Figure CN118918273B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of computer vision and three-dimensional reconstruction, and in particular to a method and device for incremental three-dimensional reconstruction based on a panoramic image. Background Art
[0002] The importance of 3D reconstruction in cutting-edge technology fields such as urban planning, augmented reality (AR), virtual reality (VR) and mixed reality (MR) is becoming increasingly prominent. The development of these fields is inseparable from high-precision 3D models. At present, the most common 3D reconstruction scheme uses a flat perspective camera to collect images. Due to its limited field of view, it is necessary to shoot from multiple angles to obtain a full scene map, and the collection process is time-consuming and laborious. In contrast, panoramic cameras make up for the shortcomings of flat perspective cameras with their ultra-large viewing angles (360 degrees horizontally and 180 degrees vertically), and improve image collection efficiency.
[0003] Existing 3D reconstruction methods rely on the collaboration of multiple sensors (such as multiple panoramic cameras, depth cameras, lidar, etc.). Multi-source data fusion acquisition increases hardware costs while increasing system complexity and maintenance difficulties. Panoramic cameras are prone to large 3D reconstruction errors or even failure to reconstruct due to incorrect feature matching in scenes with repeated textures. In addition, there are also 3D reconstruction solutions that combine planar perspective cameras with SLAM. However, due to the narrow field of view of the camera, a large amount of data needs to be collected to realize the complete scene, which is difficult to apply in large or complex scenes. Therefore, an improved solution is urgently needed to solve these technical problems. Summary of the invention
[0004] In view of this, the present application proposes a panoramic image-based incremental three-dimensional reconstruction method and device, which reduces the model calculation complexity and memory requirements while efficiently collecting data, reduces the number of erroneous feature matches in complex scenes, especially repeated texture scenes, and improves the accuracy and robustness of three-dimensional reconstruction.
[0005] In a first aspect, the present application proposes a method for incremental 3D reconstruction based on a panoramic image, comprising the following steps:
[0006] Acquire at least one panoramic image from video data of a scene to be reconstructed, perform perspective segmentation on each of the at least one panoramic image, obtain a perspective view set corresponding to the at least one panoramic image and a priori pose of each perspective view, and calculate a first matching point pair set between any two perspective views that are associated with each other in the perspective view set;
[0007] Selecting at least two initialization images from the perspective image set, constructing updated poses of the at least two initialization images in a reference coordinate system based on the prior poses of the at least two initialization images, then initializing and generating a three-dimensional model including a three-dimensional point set in the reference coordinate system based on a first matching point pair set between the at least two initialization images and the updated poses of the at least two initialization images, and adding the at least two initialization images to the registered image set;
[0008] Selecting a candidate image from the remaining image set in the perspective image set, obtaining an updated pose of the candidate image in the reference coordinate system based on a first matching point pair set between the candidate image and the registered image set, generating incremental three-dimensional points based on the first matching point pair set between the candidate image and the registered image set and the updated pose of the candidate image to update the three-dimensional point set, obtaining an incrementally updated three-dimensional model, and adding the candidate image to the registered image set;
[0009] The three-dimensional model is optimized by bundle adjustment based on the constraints of the pose error and reprojection error of the registered image set, and the steps from selecting candidate images to performing bundle adjustment are repeatedly executed until all the perspective view sets have been added to the registered image set.
[0010] Optionally, acquiring at least one panoramic image from video data of a scene to be reconstructed, perspective segmenting each panoramic image in the at least one panoramic image, and obtaining a set of perspective images corresponding to the at least one panoramic image and a priori pose of each perspective image include:
[0011] Extracting video frames from video data of the scene to be reconstructed to obtain at least one panoramic image, and performing 360-degree perspective segmentation on the at least one panoramic image according to a set segmentation angle and direction to obtain a perspective image set corresponding to the at least one panoramic image;
[0012] The pose of the at least one panoramic image is estimated by using a SLAM algorithm according to feature matching between the video frames, and the prior pose of each perspective image is calculated based on the pose of the at least one panoramic image.
[0013] Optionally, calculating a first set of matching point pairs between any two perspective images having an associated relationship in the perspective image set includes:
[0014] Building an association relationship between any two perspective images based on a time sequence and a common field of view relationship between any two perspective images in the perspective image set;
[0015] By utilizing the feature matching between any two perspective images in the perspective image set having an associated relationship, a first matching point pair set between any two perspective images in the perspective image set having an associated relationship is calculated.
[0016] Optionally, selecting at least two initialization images from the perspective image set, constructing updated poses of the at least two initialization images in a reference coordinate system based on the prior poses of the at least two initialization images, and then initializing and generating a three-dimensional model including a three-dimensional point set in the reference coordinate system based on a first matching point pair set between the at least two initialization images and the updated poses of the at least two initialization images, comprises:
[0017] Selecting a first image and a second image from the perspective image set in chronological order as the initialization images, taking the position of the first image as the origin, and constructing the reference coordinate system according to the orientation of the first image;
[0018] Obtain updated poses of the first image and the second image in the reference coordinate system respectively by calculating the prior poses of the first image and the second image;
[0019] A triangulation calculation is performed based on a first matching point pair set between the first image and the second image and an updated pose of the first image and the second image in the reference coordinate system to reconstruct a three-dimensional point set of two-dimensional matching points of the first image and the second image in the reference coordinate system, and a three-dimensional model including the three-dimensional point set is initialized and generated.
[0020] Optionally, selecting a candidate image from the remaining image set in the perspective image set, and obtaining an updated pose of the candidate image in the reference coordinate system based on a first matching point pair set between the candidate image and the registered image set, comprises:
[0021] Selecting the best image from the remaining image set as the candidate image according to a common field of view relationship and a common panoramic image source relationship between the remaining image set in the perspective image set and the registered image set;
[0022] Obtaining a second set of matching point pairs between the candidate image and the three-dimensional point set based on a first set of matching point pairs between the candidate image and each registered image in the registered image set;
[0023] Based on a second set of matching point pairs between the candidate image and the three-dimensional point set and the prior pose of the candidate image, an updated pose of the candidate image in the reference coordinate system is obtained by optimizing using a random sample consensus algorithm.
[0024] Optionally, selecting the best image from the remaining image set as the candidate image according to the common field of view relationship and the common panoramic image source relationship between the remaining image set in the perspective image set and the registered image set includes:
[0025] Whether the remaining image sets in the perspective image set have a common field of view relationship with the registered image set is used as a first condition, and whether the remaining image sets in the perspective image set have a common panoramic image source relationship with the registered image set is used as a second condition;
[0026] The best image is selected as the candidate image from the remaining images that simultaneously satisfy the first condition and the second condition; if there is no remaining image that simultaneously satisfies the first condition and the second condition, the best image is selected as the candidate image from the remaining images that satisfy the first condition.
[0027] Optionally, the generating incremental three-dimensional points based on the first matching point pair set between the candidate image and the registered image set and the updated pose of the candidate image to update the three-dimensional point set to obtain the incrementally updated three-dimensional model includes:
[0028] A triangulation calculation is performed based on the first matching point pair set between the candidate image and each registered image in the registered image set and the updated posture of the candidate image in the reference coordinate system, and incremental three-dimensional points of the two-dimensional matching points of the candidate image and the registered image set in the reference coordinate system are reconstructed to update the three-dimensional point set and obtain an incrementally updated three-dimensional model.
[0029] Optionally, the performing bundle adjustment optimization on the three-dimensional model based on the constraints of the pose error and the reprojection error of the registered image set includes:
[0030] Taking minimizing the sum of the pose error and the reprojection error of the registered image subset associated with the candidate image in the registered image set as the first objective function, performing local bundle adjustment optimization on the three-dimensional model;
[0031] The sum of the pose errors and reprojection errors of all the registered images in the registered image set is minimized as the second objective function, and the three-dimensional model is globally optimized by bundle adjustment at a predetermined frequency.
[0032] Optionally, the posture error of the registered image subset comprises an error between a first relative posture calculated based on the prior postures of any two registered images in the registered image subset and a second relative posture calculated based on the updated postures of any two registered images in the registered image subset in the reference coordinate system;
[0033] The pose errors of all the registered images include an error between a first relative pose calculated based on the prior poses of any two registered images in the registered image set and a second relative pose calculated based on the updated poses of any two registered images in the registered image set in the reference coordinate system.
[0034] In a second aspect, the present application further proposes an incremental three-dimensional reconstruction device based on a panoramic image, comprising:
[0035] A data preprocessing module, configured to obtain at least one panoramic image from the video data of the scene to be reconstructed, perform perspective segmentation on each of the at least one panoramic image, obtain a perspective view set corresponding to the at least one panoramic image and a priori pose of each perspective view, and calculate a first matching point pair set between any two perspective views having an associated relationship in the perspective view set;
[0036] A model initialization module, configured to select at least two initialization images from the perspective image set, construct updated poses of the at least two initialization images in a reference coordinate system based on the prior poses of the at least two initialization images, then initialize and generate a three-dimensional model including a three-dimensional point set in the reference coordinate system based on a first matching point pair set between the at least two initialization images and the updated poses of the at least two initialization images, and add the at least two initialization images to the registered image set;
[0037] A model updating module, configured to select a candidate image from the remaining image sets in the perspective view set, obtain an updated position and posture of the candidate image in the reference coordinate system based on a first matching point pair set between the candidate image and the registered image set, generate incremental three-dimensional points based on the first matching point pair set between the candidate image and the registered image set and the updated position and posture of the candidate image to update the three-dimensional point set, obtain an incrementally updated three-dimensional model, and add the candidate image to the registered image set until all perspective view sets have been added to the registered image set;
[0038] A model optimization module is used to perform bundle adjustment optimization on the three-dimensional model based on the constraints of the pose error and reprojection error of the registered image set until all the perspective image sets have been added to the registered image set.
[0039] The present application can at least achieve the following beneficial effects: the panoramic image-based incremental three-dimensional reconstruction method and device of the present application, based on a single panoramic camera to collect image data and combined with an incremental three-dimensional reconstruction method, can efficiently collect data while reducing model calculation complexity and memory requirements, reducing the number of erroneous feature matches in complex scenes, especially in scenes with repeated textures, and improving the accuracy and robustness of three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only illustrate certain embodiments of the present application and should not be regarded as limiting the scope of the present application.
[0041] Figure 1 is a schematic diagram of a flow chart of an incremental 3D reconstruction method based on a panoramic image according to an embodiment of the present application;
[0042] Figure 2 is a partial flow chart of an incremental 3D reconstruction method based on a panoramic image according to another embodiment of the present application;
[0043] Figure 3 is a partial flow chart of an incremental 3D reconstruction method based on a panoramic image according to another embodiment of the present application;
[0044] Figure 4 is a correlation relationship diagram between any two perspective images in a set of three-dimensional reconstructed perspective images according to an embodiment of the present application;
[0045] Figure 5 is a partial flow chart of an incremental 3D reconstruction method based on a panoramic image according to another embodiment of the present application;
[0046] Figure 6 is a partial flow chart of an incremental 3D reconstruction method based on a panoramic image according to another embodiment of the present application;
[0047] Figure 7 is a partial flow chart of an incremental 3D reconstruction method based on a panoramic image according to another embodiment of the present application;
[0048] Figure 8 is a structural schematic diagram of an incremental three-dimensional reconstruction device based on a panoramic image according to an embodiment of the present application;
[0049] Fig. 9 is a partial structural schematic diagram of an incremental three-dimensional reconstruction device based on a panoramic image according to another embodiment of the present application;
[0050] Fig.10 is a partial structural schematic diagram of an incremental three-dimensional reconstruction device based on a panoramic image according to another embodiment of the present application;
[0051] Fig.11 is a partial structural schematic diagram of an incremental three-dimensional reconstruction device based on a panoramic image according to another embodiment of the present application;
[0052] Fig.12is a partial structural schematic diagram of an incremental three-dimensional reconstruction device based on a panoramic image according to another embodiment of the present application;
[0053] Fig.13 It is a partial structural diagram of an incremental three-dimensional reconstruction device based on a panoramic image according to another embodiment of the present application. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings of the embodiment of the present application. However, it should be understood that the described embodiments are only some exemplary embodiments of the present application, rather than all embodiments, and therefore the following detailed description of the embodiment of the present application is not intended to limit the scope of protection claimed in the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0055] It should be noted that the terms "first", "second", etc. in the specification and claims of this application are only used to distinguish and describe similar objects, rather than to describe a specific order or sequence, and cannot be understood as indicating or implying relative importance.
[0056] Figure 1 FIG. 1 is a flow chart of an incremental 3D reconstruction method based on a panoramic image according to an embodiment of the present application. Figure 1 As shown, the method may include the following steps:
[0057] Step S101, obtaining at least one panoramic image from video data of a scene to be reconstructed, performing perspective segmentation on each of the at least one panoramic image, obtaining a perspective view set corresponding to the at least one panoramic image and a priori pose of each perspective view, and calculating a first matching point pair set between any two perspective views having an associated relationship in the perspective view set;
[0058] Step S102, selecting at least two initialization images from the perspective image set, constructing updated poses of the at least two initialization images in a reference coordinate system based on the prior poses of the at least two initialization images, then initializing and generating a three-dimensional model including a three-dimensional point set in the reference coordinate system based on a first matching point pair set between the at least two initialization images and the updated poses of the at least two initialization images, and adding the at least two initialization images to the registered image set;
[0059] Step S103, selecting a candidate image from the remaining image set in the perspective image set, obtaining an updated pose of the candidate image in the reference coordinate system based on a first matching point pair set between the candidate image and the registered image set, generating incremental three-dimensional points based on the first matching point pair set between the candidate image and the registered image set and the updated pose of the candidate image to update the three-dimensional point set, obtaining an incrementally updated three-dimensional model, and adding the candidate image to the registered image set;
[0060] Step S104, performing bundle adjustment optimization on the three-dimensional model based on the constraints of the pose error and reprojection error of the registered image set, and repeating the steps of selecting candidate images to performing bundle adjustment optimization until all the perspective view sets have been added to the registered image set.
[0061] In this embodiment, in step S101, a single panoramic camera may be used to capture a panoramic video of a scene to be reconstructed. After the panoramic video is obtained, frames are extracted from the panoramic video to obtain at least one panoramic image. The at least one panoramic image may be perspectively segmented according to specified rules (such as resolution, angle, and direction, etc.) to obtain a perspective view set corresponding to the at least one panoramic image, calculate a priori pose of each perspective view, and calculate a first matching point pair set between any two perspective views that are associated with each other in the perspective view set. The first matching point pair set represents a two-dimensional matching point pair set between any two perspective views that are associated with each other. The perspective view set, the priori pose of each perspective view, and the first matching point pair set between any two perspective views are used as initial conditions for three-dimensional reconstruction optimization, thereby providing a computational basis for subsequent model initialization and reducing computational complexity and time cost.
[0062] In this embodiment, in step S102, at least two initialization images are selected from the perspective view set as initialization inputs for three-dimensional reconstruction, and a reference coordinate system is constructed based on the at least two initialization images. The reference coordinate system is a reference coordinate system used to calculate the pose relationship of the perspective view in the three-dimensional model, and is used to restore and reconstruct the three-dimensional model. Subsequently, based on the prior poses of the at least two initialization images, their updated poses in the reference coordinate system are calculated. By constructing the reference coordinate system, it is ensured that different perspective views can be integrated in a unified reference coordinate system, reducing the error accumulation caused by the inaccurate relative position relationship between the perspective views. Secondly, using the first matching point pair set and the updated pose between the at least two initialization images, a three-dimensional model containing a three-dimensional point set in the reference coordinate system can be further generated to realize the initialization process of the three-dimensional model, and at the same time, at least two initialization images are added to the registered image set. Among them, the registered image set refers to the perspective view set that has completed the pose solution in the reference coordinate system and found the corresponding three-dimensional point set, thereby successfully joining the three-dimensional reconstruction process. These perspective view sets have a clear relative position relationship in the three-dimensional space and can be effectively integrated and matched with the feature point information in other perspective views. They are important basic data in the three-dimensional reconstruction process and improve the success rate and accuracy of the three-dimensional reconstruction results.
[0063] In this embodiment, in step S103, by continuously selecting images that are associated with the registered image set from the remaining image set in the perspective view set as candidate images, and based on the first matching point pair set between the selected candidate image and the registered image and the updated pose in the reference coordinate system, incremental three-dimensional points are generated to update the three-dimensional point set, and the incrementally updated three-dimensional model is obtained, and the candidate image is added to the registered image set. With the addition of more candidate images, the three-dimensional model is gradually expanded, and the newly added perspective view provides more constraint information, which helps to improve the accuracy and precision of the three-dimensional model. By adopting the incremental three-dimensional reconstruction method, the reconstruction of the three-dimensional model from scratch is avoided, and the efficiency of three-dimensional model reconstruction in complex scenes is improved.
[0064] In this embodiment, in step S104, each time a new candidate image is added to the three-dimensional model, some three-dimensional points in the registered image set will be affected. Therefore, the three-dimensional model is optimized by bundle adjustment based on the constraints of the pose error and reprojection error of the registered image set, and the new data is processed in time to continuously reduce the three-dimensional model reconstruction error. The steps of selecting candidate images to performing bundle adjustment optimization in steps S103 and S104 are repeated for the candidate images that have not been added to the three-dimensional model until the perspective view sets have all been added to the registered image set, completing the three-dimensional reconstruction process and improving the accuracy and consistency of the three-dimensional model.
[0065] In an embodiment of the present application, a single panoramic camera is used to capture a panoramic video of a scene to be reconstructed, and at least one panoramic image in the panoramic video is perspectively segmented to obtain a corresponding perspective view set. The perspective view set based on at least one panoramic image is used as an input for three-dimensional reconstruction, and a priori pose of each perspective image in the perspective view set and a set of matching point pairs between any two perspective images that have an associated relationship are calculated to provide a computational basis for subsequent model initialization. At least two initialization images are selected from the perspective view set, and a reference coordinate system is constructed with one of the initialization images as a reference. Based on the priori poses of the at least two initialization images, an updated pose of each initialization image in the reference coordinate system is obtained, and a three-dimensional model containing a three-dimensional point set in the reference coordinate system is generated by initializing the updated poses and the matching point pair set. , and add at least two initialization images to the registered image set to implement the model initialization process; select candidate images from the remaining image sets in the perspective view set, calculate their updated pose in the reference coordinate system based on the matching point pair set of the candidate images and the registered image set, generate incremental three-dimensional points through the updated pose and matching point pair set of the two to update the three-dimensional point set, obtain the three-dimensional model after incremental update, and add the candidate images to the registered image set, thereby improving the efficiency and accuracy of three-dimensional reconstruction; as the candidate images are continuously added to the registered image set, the newly added data affects the original three-dimensional point set, and the bundle adjustment optimization is used to reduce the error impact of the registered image set on the three-dimensional model, until the perspective view set has been added to the registered image set, and the three-dimensional reconstruction process is completed. Therefore, the embodiment of the present application can complete efficient video data acquisition based on a single panoramic camera, significantly reducing the amount of data acquisition, while reducing the computational complexity and memory requirements of the three-dimensional model, reducing feature mismatching in complex scenes, especially in scenes with repeated textures, and improving the accuracy and robustness of three-dimensional reconstruction.
[0066] In one embodiment, if Figure 2 As shown, in step S101, obtaining at least one panoramic image from the video data of the scene to be reconstructed, perspective segmenting each panoramic image in the at least one panoramic image, and obtaining a perspective view set corresponding to the at least one panoramic image and a priori pose of each perspective view may include:
[0067] Step S201, extracting video frames from video data of a scene to be reconstructed to obtain at least one panoramic image, performing 360-degree perspective segmentation on the at least one panoramic image according to a set segmentation angle and direction, and obtaining a perspective image set corresponding to the at least one panoramic image;
[0068] Step S202: using a SLAM algorithm to estimate the pose of the at least one panoramic image based on feature matching between the video frames, and calculating the prior pose of each perspective image based on the pose of the at least one panoramic image.
[0069] In one embodiment, first, video frames are extracted from the video data of the scene to be reconstructed to obtain at least one panoramic image, and the at least one panoramic image is segmented into 360 degrees according to the set segmentation angle and direction to obtain a perspective view set corresponding to the at least one panoramic image. The video data of the scene to be reconstructed is collected by a panoramic camera with an ultra-large viewing angle (360 degrees in the horizontal direction and 180 degrees in the vertical direction), and after obtaining the collected original video, frames are extracted from it to obtain at least one panoramic image, and the at least one panoramic image is segmented into 360 degrees according to the set segmentation angle and direction. The camera intrinsic parameters of the plane perspective image are calibrated using a pinhole model to obtain a perspective view set corresponding to the at least one panoramic image, wherein the pinhole model is a simplified optical imaging model, and the basic principle is central perspective projection, which projects objects in three-dimensional space onto a two-dimensional imaging plane through a pinhole. The model ignores complex factors such as lens distortion and light diffraction, making calculation and analysis relatively simple.
[0070] Then, the SLAM algorithm is used to estimate the pose of the at least one panoramic image based on the feature matching between the video frames, and the prior pose of each perspective image is calculated based on the pose of the at least one panoramic image. SLAM (Simultaneous Localization and Mapping) is the full name of "Simultaneous Localization and Mapping". The algorithm can be used to pre-process the video frames, extract the feature points or feature areas in the image, and estimate the pose of the image by comparing the feature points or feature areas. For example, suppose that multiple panoramas are obtained by extracting video frames from the video data of the scene to be reconstructed, and any panorama is taken as For example, take the camera optical center to the image center of the panorama as the starting direction, and go horizontally to the right with the field of view angle Crop the panorama to a specified resolution to get a segmented perspective image , then the prior pose of this perspective image can be calculated by the following formula:
[0071]
[0072] in Indicates the rotation angle from the camera optical center to the image center of the panorama as the starting direction Assuming that a panoramic image is divided into multiple perspective images, the prior poses of these multiple perspective images can be obtained by the above calculation formula.
[0073] In one embodiment, if Figure 3 As shown, in step S101, calculating the first set of matching point pairs between any two perspective images having an associated relationship in the perspective image set may include:
[0074] Step S301, constructing an association relationship between any two perspective images in the perspective image set based on the time sequence and common field of view relationship between any two perspective images;
[0075] Step S302: using feature matching between any two perspective images in the perspective image set that are in an associated relationship, calculate and obtain a first set of matching point pairs between any two perspective images in an associated relationship.
[0076] In one embodiment, in this embodiment, the perspective view set in step S101 is used as input for reconstructing a three-dimensional model, and an association relationship between any two perspective views in the perspective view set is constructed based on the time sequence and common field of view relationship between any two perspective views in the perspective view set. The common field of view relationship indicates that the two images have the same scene content. Figure 4 The figure shows the association relationship between any two perspectives, where the nodes Represents a perspective, and the node subscripts represent the time sequence between the above perspectives, that is, the perspective Time precedes perspective , and so on, perspective The time order is the latest, and the connection lines between the nodes represent the common field of view relationship between the above perspectives, that is, and , and , and , and , and , and There is a common field of view relationship between them, that is, it can be considered and , and , and , and , and , There is a correlation between and There is no connecting line between them, indicating that they do not have a common field of view. and There is no correlation between them. On this basis, the two perspectives mentioned above that have a correlation and , and , and , and , and , and A first set of matching point pairs can be generated.
[0077] In this embodiment, by using the feature matching between any two perspective images in the perspective image set that have an associated relationship, a first matching point pair set between any two perspective images in the perspective image set that have an associated relationship is calculated, wherein the first matching point pair set represents a two-dimensional matching point pair set between any two perspective images in the perspective image set that have an associated relationship. For example, and images The set of two-dimensional matching point pairs between the two It can be expressed as:
[0078]
[0079] in, , represents the image The The two-dimensional coordinates of feature points; Representing images In and Image The The two-dimensional coordinates of the feature points matched by the feature points; Represents two images and The number of two-dimensional matching point pairs between .
[0080] Thus, this embodiment constructs an association relationship between any two perspective views in the perspective view set through the time sequence and common field of view relationship between any two perspective views in the perspective view set, and performs feature matching based on this, and calculates the first matching point pair set between any two perspective views in the association relationship, that is, the two-dimensional matching point pair set, which provides a calculation basis for the subsequent initialization construction of the three-dimensional model, reduces the amount of data processing, and simplifies the three-dimensional reconstruction process.
[0081] In one embodiment, if Figure 5 As shown, the step S102 of selecting at least two initialization images from the perspective image set, constructing updated poses of the at least two initialization images in a reference coordinate system based on the prior poses of the at least two initialization images, and then initializing and generating a three-dimensional model including a three-dimensional point set in the reference coordinate system based on a first matching point pair set between the at least two initialization images and the updated poses of the at least two initialization images may include:
[0082] Step S501, selecting a first image and a second image from the perspective image set in chronological order as the initialization images, taking the position of the first image as the origin, and constructing the reference coordinate system according to the orientation of the first image;
[0083] Step S502, calculating the updated poses of the first image and the second image in the reference coordinate system respectively through the prior poses of the first image and the second image;
[0084] Step S503, performing triangulation calculation based on the first matching point pair set between the first image and the second image and the updated poses of the first image and the second image in the reference coordinate system, reconstructing a three-dimensional point set of the two-dimensional matching points of the first image and the second image in the reference coordinate system, and initializing generation of a three-dimensional model including the three-dimensional point set.
[0085] In this embodiment, the first image and the second image are selected from the perspective image set in chronological order as the initialization image. As an example, the first two images of the perspective image set can be selected. Assume that the first image is recorded as , and its corresponding prior pose is recorded as , the second image is recorded as , and its corresponding prior pose is recorded as , during the initialization process, with the first image The position of is the origin, according to the first image The reference coordinate system is constructed in the direction , through the first image and the second image The prior pose of the second image can be calculated In the base coordinate system Updated pose of :
[0086] .
[0087] in, express The inverse operation matrix of .
[0088] Then, according to the first image calculated in step S202 and the second image A set of two-dimensional matching point pairs , for the first image and the second image In the base coordinate system Perform triangulation calculation on the updated pose under the condition to restore the reference coordinate system The three-dimensional point set under , Represents the base coordinate system The number of three-dimensional points in the three-dimensional point set under the reference coordinate system is calculated, and the three-dimensional model including the three-dimensional point set is initialized and generated. The accuracy and success rate of the initial construction of the three-dimensional model are improved by calculating the precise position and posture of the perspective view in the reference coordinate system and obtaining the three-dimensional point set by triangulation.
[0089] In one embodiment, if Figure 6 As shown, the step S103 of selecting a candidate image from the remaining image set in the perspective image set, and obtaining an updated pose of the candidate image in the reference coordinate system based on a first matching point pair set between the candidate image and the registered image set may include:
[0090] Step S601, selecting the best image from the remaining image set as the candidate image according to the common field of view relationship and the common panoramic image source relationship between the remaining image set in the perspective image set and the registered image set;
[0091] Step S602, obtaining a second matching point pair set between the candidate image and the three-dimensional point set based on a first matching point pair set between the candidate image and each registered image in the registered image set;
[0092] Step S603: Based on a second set of matching point pairs between the candidate image and the three-dimensional point set and the prior pose of the candidate image, a random sample consensus algorithm is used to optimize and obtain an updated pose of the candidate image in the reference coordinate system.
[0093] In this embodiment, the common field of view relationship and common panoramic source relationship between the remaining image set in the perspective image set and the registered image select the best image from the remaining image set as the candidate image. The common field of view relationship indicates that the two images have the same scene content, and the common panoramic source relationship indicates that the two images come from the same panoramic image. Among them, whether the remaining image set in the perspective image set has a common field of view relationship with the registered image set is used as the first condition, and whether the remaining image set in the perspective image set has a common panoramic source relationship with the registered image set is used as the second condition. The best image is selected as the candidate image from the remaining images that meet both the first and second conditions, because the perspective images obtained from the same panoramic image have overlapping field of view angles, that is, the same scene may be captured, which is more conducive to the accuracy of three-dimensional model reconstruction. If there are no remaining images that meet both the first and second conditions, the best image is selected as the candidate image from the remaining images that meet the first condition.
[0094] In one embodiment, based on step S601, a candidate image is selected. Assume that the determined candidate image is , any registered image in the 3D model is , the image is obtained according to the calculation method in step S202 and The first matching point pair set is the two-dimensional matching point pair set , for the set Any matching point pair ,if There are corresponding 3D points in the 3D model , we can get the image Two-dimensional point and 3D points in 3D models The two-dimensional and three-dimensional point matching points , repeat the operation set The remaining matching points in the image and all registered images are traversed, and the candidate image can be obtained after completion. The second set of matching point pairs with the 3D points in the 3D model ,in Representing images The number of matching point pairs with 3D points in the 3D model.
[0095] Based on the candidate images The prior pose of the 3D point and the second matching point pair set of the 3D point , the updated pose of the candidate image in the reference coordinate system is optimized using the random sample consensus algorithm, wherein in image processing, the basic principle of the random sample consensus algorithm (hereinafter referred to as the RANSAC algorithm) is to randomly select some point pairs from the data set, calculate a model (camera pose) based on these point pairs, and then evaluate the degree of adaptation of the model to all data points. By iterating this process multiple times, the models that can adapt to the most data points are selected as the optimal models. This method is particularly suitable for processing data sets containing a large number of outliers, such as mismatches in feature point matching.
[0096] Specifically in this embodiment, based on the candidate image The prior pose and the second matching point pair set are randomly selected from the second matching point pair set, and based on these random matching point pairs, the image The prior pose estimation image In the base coordinate system The updated pose under , evaluates the adaptability of the updated pose to all matching point pairs, and iterates this process multiple times to select the updated pose that can adapt to the most matching point pairs as the image In the base coordinate system The final updated pose under Thus, this embodiment can find the best pose of the candidate image when including erroneous matching points, so that the two-dimensional points in the candidate image most accurately correspond to the pose of the three-dimensional points through the camera projection model.
[0097] In this embodiment, the best image is selected from the remaining image set as the candidate image through the common field of view relationship and the common panoramic source relationship between the remaining image set in the perspective view set and the registered image, and the second matching point pair set between the candidate image and the three-dimensional point set in the three-dimensional model is calculated. The RANSAC algorithm is used to optimize the updated pose of the candidate image in the reference coordinate system based on the prior pose of the candidate image and the second matching point pair set of its three-dimensional points, so as to reduce the interference of mismatching in feature point matching, and by optimizing the updated pose of the candidate image in the reference coordinate system, the three-dimensional reconstruction accuracy is improved, the consumption of computing resources is reduced, and the foundation is laid for the subsequent acquisition of incremental updated three-dimensional models.
[0098] In one embodiment, the step S103 of generating incremental three-dimensional points based on the first matching point pair set between the candidate image and the registered image set and the updated pose of the candidate image to update the three-dimensional point set to obtain the incrementally updated three-dimensional model may specifically include:
[0099] A triangulation calculation is performed based on the first matching point pair set between the candidate image and each registered image in the registered image set and the updated posture of the candidate image in the reference coordinate system, and incremental three-dimensional points of the two-dimensional matching points of the candidate image and the registered image set in the reference coordinate system are reconstructed to update the three-dimensional point set and obtain an incrementally updated three-dimensional model.
[0100] In one embodiment, if Figure 7 As shown, the bundle adjustment optimization of the three-dimensional model based on the constraints of the pose error and reprojection error of the registered image set in step S104 may include:
[0101] Step S701, taking the sum of the pose error and the reprojection error of the registered image subset associated with the candidate image in the registered image set as the first objective function, performing a local bundle adjustment optimization on the three-dimensional model;
[0102] Step S702, taking the sum of the pose errors and reprojection errors of all the registered images in the registered image set as minimization as the second objective function, performing global bundle adjustment optimization on the three-dimensional model at a predetermined frequency.
[0103] In this embodiment, each time a new candidate image is added to the three-dimensional model, a local bundle adjustment optimization is performed on the affected three-dimensional points and image poses. Bundle Adjustment (hereinafter referred to as BA optimization) can also be called bundle adjustment or bundle adjustment. Its essence is an unconstrained nonlinear minimum optimization problem, which is achieved by minimizing the reprojection error. Specifically, it is based on the reprojection error model, combining the three-dimensional space points and the internal and external parameters of the camera as a whole for optimization, thereby improving the accuracy of three-dimensional reconstruction. In this embodiment, based on the minimization of the reprojection error of the traditional BA optimization, the calculation of the pose error of the subset of registered images in the registered image set that has an associated relationship with the candidate image is added. By adding the pose error calculation, the prior information and constraints are enriched, so that the results of the BA optimization are more accurate. The objective function of the improved local BA optimization That is, the first objective function is as follows:
[0104]
[0105] in, represents the number of panoramas involved in the registered image subset associated with the candidate image in the registered image set, that is, the number of panoramas participating in the local BA optimization; represents the number of perspectives of each panorama participating in the local BA optimization;
[0106] represents the first perspective in the set of all perspectives in the panorama participating in the local BA optimization The image The pixel coordinates of feature points, where Indicates that The image The coordinates of the 3D points matched by the feature points; Indicates the The image is in the reference coordinate system Updated pose under ; A mapping function representing the mapping relationship of the camera projection model; represents the two-dimensional point coordinates obtained by projecting the three-dimensional point coordinates based on the mapping function;
[0107] Indicates the first In the panorama images and images In the base coordinate system The relative position under and Respectively represent images and images In the coordinate system The updated pose under , is calculated as follows:
[0108] .
[0109] in, express The inverse operation matrix of .
[0110] Indicates the first In the panorama images and images The relative pose between the prior poses of and Respectively represent images and images The prior pose of is calculated as follows:
[0111] .
[0112] in, express The inverse operation matrix of .
[0113] The first objective function In Indicates the first In the panorama images and images The pose error between the updated pose and the prior pose, represents the first perspective in the set of all perspectives in the panorama participating in the local BA optimization The first objective function means that the three-dimensional model is locally optimized by bundle adjustment with the goal of minimizing the sum of the pose error and the reprojection error of the registered image subset associated with the candidate image in the registered image set.
[0114] In one embodiment, the sum of the pose errors and reprojection errors of all registered images in the registered image set in the three-dimensional model is minimized as the second objective function, and global BA optimization is performed at a preset frequency. In one embodiment, the preset frequency may include updating the number of candidate images according to a predetermined number, that is, the global BA optimization may be performed periodically according to the predetermined number of candidate image updates. The predetermined number of candidate image updates may be set according to an empirical value based on the scale of the number of perspective images in the perspective image set. In another embodiment, the preset frequency for performing global BA optimization may also be determined based on the actual number of registered images, which is not limited here.
[0115] Similar to the first objective function of the local BA optimization mentioned above, the objective function of the improved global BA optimization is That is, the second objective function is as follows:
[0116]
[0117] in, Indicates the number of panoramas involved in all registered images, that is, the number of panoramas participating in the global BA optimization. represents the number of perspectives of each panorama in the panorama participating in the global BA optimization;
[0118] represents the first perspective in the set of all perspectives in the panorama participating in the global BA optimization The image The pixel coordinates of feature points, where Indicates the The image The coordinates of the 3D points matched by the feature points; Indicates the The image is in the reference coordinate system Updated pose under ; A mapping function representing the mapping relationship of the camera projection model; represents the two-dimensional point coordinates obtained by projecting the three-dimensional point coordinates based on the mapping function;
[0119] Indicates the first In the panorama images and images In the base coordinate system The relative position under and Respectively represent images and images In the coordinate system The updated pose under , is calculated as follows:
[0120] .
[0121] in, express The inverse operation matrix of .
[0122] Indicates the first In the panorama images and images The relative pose between the prior poses of and Respectively represent images and images The prior pose of is calculated as follows:
[0123] .
[0124] in, express The inverse operation matrix of .
[0125] The second objective function In Indicates the first In the panorama images and images The error between the updated pose and the prior pose, represents the first perspective in the set of all perspectives in the panorama participating in the global BA optimization The second objective function means that the three-dimensional model is globally optimized by bundle adjustment with the goal of minimizing the sum of the pose errors and reprojection errors of all registered images in the registered image set.
[0126] In this embodiment, the first objective function is to minimize the sum of the pose errors and reprojection errors of the subset of registered images in the registered image set that are associated with the candidate image, and local BA optimization is performed on the three-dimensional model. The second objective function is to minimize the sum of the pose errors and reprojection errors of all registered images in the registered image set, and global BA optimization is performed on the three-dimensional model at a preset frequency. By introducing the calculation of the pose errors of the subset of registered images in the registered image set that are associated with the candidate image and the pose errors of all registered images, the defects of traditional three-dimensional reconstruction such as large errors and low success rate in complex scenes, especially scenes with repeated textures, are overcome, the feature mismatching in complex scenes, especially scenes with repeated textures, is reduced, and the accuracy and robustness of three-dimensional reconstruction are improved.
[0127] Figure 8 FIG. 1 is a schematic diagram of a structure of an incremental 3D reconstruction device based on a panoramic image according to an embodiment of the present application. Figure 8 As shown, the device includes the following modules:
[0128] The data preprocessing module 801 is used to obtain at least one panoramic image from the video data of the scene to be reconstructed, perform perspective segmentation on each panoramic image in the at least one panoramic image, obtain a perspective view set corresponding to the at least one panoramic image and a priori pose of each perspective view, and calculate a first matching point pair set between any two perspective views that have an associated relationship in the perspective view set;
[0129] A model initialization module 802 is used to select at least two initialization images from the perspective image set, construct updated poses of the at least two initialization images in a reference coordinate system based on the prior poses of the at least two initialization images, and then initialize and generate a three-dimensional model including a three-dimensional point set in the reference coordinate system based on a first matching point pair set between the at least two initialization images and the updated poses of the at least two initialization images, and add the at least two initialization images to the registered image set;
[0130] A model updating module 803 is used to select a candidate image from the remaining image sets in the perspective view set, obtain an updated pose of the candidate image in the reference coordinate system based on a first matching point pair set between the candidate image and the registered image set, generate incremental three-dimensional points based on the first matching point pair set between the candidate image and the registered image set and the updated pose of the candidate image to update the three-dimensional point set, obtain an incrementally updated three-dimensional model, and add the candidate image to the registered image set until all perspective view sets have been added to the registered image set;
[0131] The model optimization module 804 is used to perform bundle adjustment optimization on the three-dimensional model based on the constraints of the pose error and reprojection error of the registered image set until all the perspective image sets have been added to the registered image set.
[0132] In one embodiment, if Fig. 9 As shown, the data preprocessing module 801 may further include the following units:
[0133] A perspective image acquisition unit 901 is used to extract video frames from video data of a scene to be reconstructed to obtain at least one panoramic image, and perform 360-degree perspective segmentation on the at least one panoramic image according to a set segmentation angle and direction to obtain a perspective image set corresponding to the at least one panoramic image;
[0134] A posture processing unit 902 is used to estimate the posture of the at least one panoramic image according to feature matching between the video frames using a SLAM algorithm, and calculate the prior posture of each perspective image based on the posture of the at least one panoramic image;
[0135] In one embodiment, if Fig.10 As shown, the data preprocessing module 801 may further include the following units:
[0136] A relationship building unit 1001 is used to build an association relationship between any two perspective images in the perspective image set based on the time sequence and common field of view relationship between any two perspective images in the perspective image set;
[0137] The first matching unit 1002 is used to calculate a first matching point pair set between any two perspective images that are associated with each other in the perspective image set by using feature matching between any two perspective images that are associated with each other.
[0138] In one embodiment, if Fig.11 As shown, the model initialization module 802 may further include the following units:
[0139] A coordinate system initialization unit 1101 is used to select a first image and a second image from the perspective image set in chronological order as the initialization images, take the position of the first image as the origin, and construct the reference coordinate system according to the orientation of the first image;
[0140] A first pose calculation unit 1102, configured to calculate updated poses of the first image and the second image in the reference coordinate system according to the prior poses of the first image and the second image;
[0141] The model initialization unit 1103 is used to perform triangulation calculation based on the first matching point pair set between the first image and the second image and the updated posture of the first image and the second image in the reference coordinate system, reconstruct a three-dimensional point set of the two-dimensional matching points of the first image and the second image in the reference coordinate system, and initialize and generate a three-dimensional model including the three-dimensional point set.
[0142] In one embodiment, if Fig.12 As shown, the model updating module 803 may also include the following units:
[0143] A candidate image selection unit 1201 is used to select the best image from the remaining image set as the candidate image according to a common field of view relationship and a common panoramic image source relationship between the remaining image set in the perspective image set and the registered image set;
[0144] A second matching unit 1202 is used to obtain a second matching point pair set between the candidate image and the three-dimensional point set based on the first matching point pair set between the candidate image and each registered image in the registered image set;
[0145] The second pose calculation unit 1203 is used to optimize the updated pose of the candidate image in the reference coordinate system using a random sample consensus algorithm based on a second matching point pair set between the candidate image and the three-dimensional point set and the prior pose of the candidate image.
[0146] In one implementation, the model updating module 803 may further include:
[0147] A model incremental update unit is used to perform triangulation calculations based on a first matching point pair set between the candidate image and each registered image in the registered image set and an updated position and posture of the candidate image in the reference coordinate system, to reconstruct incremental three-dimensional points of the two-dimensional matching points between the candidate image and the registered image set in the reference coordinate system, so as to update the three-dimensional point set and obtain an incrementally updated three-dimensional model.
[0148] In one embodiment, if Fig.13 As shown, the model optimization module 804 may also include the following units:
[0149] A first optimization unit 1301 is used to perform bundle adjustment optimization on the three-dimensional model based on the constraint conditions of the pose error and reprojection error based on the registered image set, and to minimize the sum of the pose error and reprojection error of the registered image subset in the registered image set that is associated with the candidate image as a first objective function, and to perform local bundle adjustment optimization on the three-dimensional model;
[0150] The second optimization unit 1302 is used to perform global bundle adjustment optimization on the three-dimensional model at a predetermined frequency by minimizing the sum of the posture errors and reprojection errors of all the registered images in the registered image set as the second objective function.
[0151] In summary, the embodiments of the present application use a single panoramic camera to capture a panoramic video of a scene to be reconstructed, perform perspective segmentation on at least one panoramic image in the panoramic video to obtain a corresponding perspective view set, use the perspective view set based on at least one panoramic image as input for 3D reconstruction, and calculate the prior pose of each perspective view in the perspective view set and the set of matching point pairs between any two perspective views that have an associated relationship, so as to provide a computational basis for subsequent model initialization; select at least two initialization images from the perspective view set, construct a reference coordinate system with one of the initialization images as a reference, obtain an updated pose of each initialization image in the reference coordinate system based on the prior pose of the at least two initialization images, and generate a three-dimensional point set in the reference coordinate system by updating the pose and initializing the set of matching point pairs. 3D model, and add at least two initialization images to the registered image set to implement the model initialization process; select candidate images from the remaining image sets in the perspective view set, calculate their updated pose in the reference coordinate system based on the matching point pair set of the candidate images and the registered image set, generate incremental 3D points through the updated pose and matching point pair set of the two to update the 3D point set, obtain the incremental updated 3D model, and add the candidate images to the registered image set, thereby improving the efficiency and accuracy of 3D reconstruction; as the candidate images are continuously added to the registered image set, the newly added data affects the original 3D point set, and the bundle adjustment optimization is used to reduce the error impact of the registered image set on the 3D model, until the perspective view set has been added to the registered image set, and the 3D reconstruction process is completed. Thus, the embodiment of the present application can complete efficient video data acquisition based on a single panoramic camera, significantly reducing the amount of data acquisition, while reducing the computational complexity and memory requirements of the 3D model, reducing feature mismatches in complex scenes, especially in scenes with repeated textures, and improving the accuracy and robustness of 3D reconstruction.
[0152] It should be noted that those skilled in the art can understand that the different implementation methods described in the method embodiments of the present application and their explanations and technical effects achieved are also applicable to the device embodiments of the present application and will not be repeated here.
[0153] The above describes the exemplary embodiments of the present application. It should be understood that the above exemplary embodiments are not restrictive but illustrative, and the protection scope of the present application is not limited thereto. It should be understood that those skilled in the art can modify and vary the embodiments of the present application without departing from the spirit and scope of the present application, and these modifications and variations should be within the protection scope of the present application.
Claims
1. An incremental 3D reconstruction method based on panoramas, It is characterized in that The method comprises: Acquire at least one panoramic image from video data of a scene to be reconstructed, perform perspective segmentation on each of the at least one panoramic image, obtain a perspective view set corresponding to the at least one panoramic image and a priori pose of each perspective view, and calculate a first matching point pair set between any two perspective views that are associated with each other in the perspective view set; Selecting at least two initialization images from the perspective image set, constructing updated poses of the at least two initialization images in a reference coordinate system based on the prior poses of the at least two initialization images, then initializing and generating a three-dimensional model including a three-dimensional point set in the reference coordinate system based on a first matching point pair set between the at least two initialization images and the updated poses of the at least two initialization images, and adding the at least two initialization images to the registered image set; Selecting a candidate image from the remaining image set in the perspective image set, obtaining an updated pose of the candidate image in the reference coordinate system based on a first matching point pair set between the candidate image and the registered image set, generating incremental three-dimensional points based on the first matching point pair set between the candidate image and the registered image set and the updated pose of the candidate image to update the three-dimensional point set, obtaining an incrementally updated three-dimensional model, and adding the candidate image to the registered image set; The three-dimensional model is optimized by bundle adjustment based on the constraints of the pose error and reprojection error of the registered image set, and the steps from selecting candidate images to performing bundle adjustment are repeatedly executed until all the perspective view sets have been added to the registered image set.
2. The method for incremental three-dimensional reconstruction based on a panoramic image according to claim 1, It is characterized in that The method of acquiring at least one panoramic image from video data of a scene to be reconstructed, performing perspective segmentation on each panoramic image in the at least one panoramic image, and obtaining a set of perspective images corresponding to the at least one panoramic image and a priori pose of each perspective image includes: Extracting video frames from video data of the scene to be reconstructed to obtain at least one panoramic image, and performing 360-degree perspective segmentation on the at least one panoramic image according to a set segmentation angle and direction to obtain a perspective image set corresponding to the at least one panoramic image; The pose of the at least one panoramic image is estimated by using a SLAM algorithm according to feature matching between the video frames, and the prior pose of each perspective image is calculated based on the pose of the at least one panoramic image.
3. The method for incremental three-dimensional reconstruction based on a panoramic image according to claim 1, It is characterized in that The calculating of a first set of matching point pairs between any two perspective images having an associated relationship in the perspective image set comprises: Building an association relationship between any two perspective images based on a time sequence and a common field of view relationship between any two perspective images in the perspective image set; By utilizing the feature matching between any two perspective images in the perspective image set having an associated relationship, a first matching point pair set between any two perspective images in the perspective image set having an associated relationship is calculated.
4. The method for incremental three-dimensional reconstruction based on a panoramic image according to claim 1, It is characterized in that The method comprises: selecting at least two initialization images from the perspective image set, constructing updated poses of the at least two initialization images in a reference coordinate system based on the prior poses of the at least two initialization images, and then initializing and generating a three-dimensional model including a three-dimensional point set in the reference coordinate system based on a first matching point pair set between the at least two initialization images and the updated poses of the at least two initialization images, comprising: Selecting a first image and a second image from the perspective image set in chronological order as the initialization images, taking the position of the first image as the origin, and constructing the reference coordinate system according to the orientation of the first image; Obtain updated poses of the first image and the second image in the reference coordinate system respectively by calculating the prior poses of the first image and the second image; A triangulation calculation is performed based on a first matching point pair set between the first image and the second image and an updated pose of the first image and the second image in the reference coordinate system to reconstruct a three-dimensional point set of two-dimensional matching points of the first image and the second image in the reference coordinate system, and a three-dimensional model including the three-dimensional point set is initialized and generated.
5. The method for incremental three-dimensional reconstruction based on a panoramic image according to claim 1, It is characterized in that The step of selecting a candidate image from the remaining image set in the perspective image set, and obtaining an updated pose of the candidate image in the reference coordinate system based on a first matching point pair set between the candidate image and the registered image set, comprises: Selecting the best image from the remaining image set as the candidate image according to a common field of view relationship and a common panoramic image source relationship between the remaining image set in the perspective image set and the registered image set; Obtaining a second set of matching point pairs between the candidate image and the three-dimensional point set based on a first set of matching point pairs between the candidate image and each registered image in the registered image set; Based on a second set of matching point pairs between the candidate image and the three-dimensional point set and the prior pose of the candidate image, an updated pose of the candidate image in the reference coordinate system is obtained by optimizing using a random sample consensus algorithm.
6. The method for incremental three-dimensional reconstruction based on a panoramic image according to claim 5, It is characterized in that The selecting the best image from the remaining image set as the candidate image according to the common field of view relationship and the common panoramic image source relationship between the remaining image set in the perspective image set and the registered image set comprises: Whether the remaining image sets in the perspective image set have a common field of view relationship with the registered image set is used as a first condition, and whether the remaining image sets in the perspective image set have a common panoramic image source relationship with the registered image set is used as a second condition; The best image is selected as the candidate image from the remaining images that simultaneously satisfy the first condition and the second condition; if there is no remaining image that simultaneously satisfies the first condition and the second condition, the best image is selected as the candidate image from the remaining images that satisfy the first condition.
7. The method for incremental three-dimensional reconstruction based on a panoramic image according to claim 1, It is characterized in that The method of generating incremental three-dimensional points based on the first matching point pair set between the candidate image and the registered image set and the updated position and posture of the candidate image to update the three-dimensional point set to obtain the incrementally updated three-dimensional model includes: A triangulation calculation is performed based on the first matching point pair set between the candidate image and each registered image in the registered image set and the updated posture of the candidate image in the reference coordinate system, and incremental three-dimensional points of the two-dimensional matching points of the candidate image and the registered image set in the reference coordinate system are reconstructed to update the three-dimensional point set and obtain an incrementally updated three-dimensional model.
8. The method of incremental three-dimensional reconstruction based on a panoramic image according to claim 1, It is characterized in that The bundle adjustment optimization of the three-dimensional model based on the constraints of the posture error and the reprojection error of the registered image set includes: Taking minimizing the sum of the pose error and the reprojection error of the registered image subset associated with the candidate image in the registered image set as the first objective function, performing local bundle adjustment optimization on the three-dimensional model; The sum of the pose errors and reprojection errors of all the registered images in the registered image set is minimized as the second objective function, and the three-dimensional model is globally optimized by bundle adjustment at a predetermined frequency.
9. The method for incremental three-dimensional reconstruction based on a panoramic image according to claim 8, It is characterized in that The pose error of the registered image subset includes an error between a first relative pose calculated based on the prior poses of any two registered images in the registered image subset and a second relative pose calculated based on the updated poses of any two registered images in the registered image subset in the reference coordinate system; The pose errors of all the registered images include an error between a first relative pose calculated based on the prior poses of any two registered images in the registered image set and a second relative pose calculated based on the updated poses of any two registered images in the registered image set in the reference coordinate system.
10. An incremental 3D reconstruction device based on a panoramic image, It is characterized in that The device comprises: A data preprocessing module, configured to obtain at least one panoramic image from the video data of the scene to be reconstructed, perform perspective segmentation on each of the at least one panoramic image, obtain a perspective view set corresponding to the at least one panoramic image and a priori pose of each perspective view, and calculate a first matching point pair set between any two perspective views having an associated relationship in the perspective view set; A model initialization module, configured to select at least two initialization images from the perspective image set, construct updated poses of the at least two initialization images in a reference coordinate system based on the prior poses of the at least two initialization images, then initialize and generate a three-dimensional model including a three-dimensional point set in the reference coordinate system based on a first matching point pair set between the at least two initialization images and the updated poses of the at least two initialization images, and add the at least two initialization images to the registered image set; A model updating module, configured to select a candidate image from the remaining image sets in the perspective view set, obtain an updated position and posture of the candidate image in the reference coordinate system based on a first matching point pair set between the candidate image and the registered image set, generate incremental three-dimensional points based on the first matching point pair set between the candidate image and the registered image set and the updated position and posture of the candidate image to update the three-dimensional point set, obtain an incrementally updated three-dimensional model, and add the candidate image to the registered image set until all perspective view sets have been added to the registered image set; A model optimization module is used to perform bundle adjustment optimization on the three-dimensional model based on the constraints of the pose error and reprojection error of the registered image set until all the perspective image sets have been added to the registered image set.
Citation Information
Patent Citations
Point cloud model construction method and device, equipment and storage medium
CN112837419A
Robot teaching control method based on virtual vision
CN116619376A
Method for simplifying low modulus from high modulus and automatically mapping UV texture
CN117274481A
Method of pose estimation adn model refinement for video representation of a three dimensional scene
EP1890263A2
