Panoramic three-dimensional reconstruction method, device, equipment and medium

Abstract

The invention provides a panoramic three-dimensional reconstruction method, device and equipment and a medium, and relates to the field of three-dimensional reconstruction. Comprising the following steps: performing panoramic image inner mapping and self-pairing on each panoramic image in a plurality of panoramic images of a target scene to obtain a first image pair set; performing mapping and cross-image pairing among the panoramic images for the plurality of panoramic images to obtain a second image pair set; for the first image pair set and the second image pair set, communicating the self-pairing and cross-image pairing relationship to obtain a third image pair set, and further completing the image pair set; inputting the complete image pair set into a pre-trained three-dimensional reconstruction model to obtain a local three-dimensional reconstruction result of each candidate perspective image in the complete image pair set; and carrying out pose prior driven global alignment on the local three-dimensional reconstruction result to obtain a panoramic three-dimensional reconstruction result of the target scene so as to realize accurate and efficient panoramic three-dimensional reconstruction on the basis of no training.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional reconstruction, and in particular to a panoramic three-dimensional reconstruction method, device, equipment and medium. BACKGROUND

[0002] Panoramic three-dimensional reconstruction plays an important role in immersive scene understanding tasks such as robot perception, augmented reality, and autonomous driving. However, current three-dimensional reconstruction methods are mainly designed for pinhole camera images, and are difficult to directly adapt to the reconstruction needs of 360° panoramic images. The main reason is the lack of panoramic training data and the high cost of retraining. Specifically, there are the following problems: First, due to the spherical structure of panoramic images, directly inputting them into existing three-dimensional reconstruction models trained based on pinhole images will result in poor reconstruction results due to structural differences and the lack of panoramic training data, high retraining costs, and complex deployment. Second, common image pairing strategies often need to exhaustively match all image pairs when processing multi-view images generated by panoramic image projection, resulting in a quadratic increase in computational cost, affecting reconstruction accuracy and efficiency. Finally, in the global alignment stage, current methods usually treat different view images as independent inputs, which can easily cause pose drift and point cloud misalignment, affecting the stability of the overall reconstruction. SUMMARY

[0003] Based on the above technical problems, the present application provides a panoramic three-dimensional reconstruction method, device, equipment and medium, which aims to overcome the above problems or at least partially solve the above problems.

[0004] The first aspect of the present application provides a panoramic three-dimensional reconstruction method, comprising: For each panoramic image in the plurality of panoramic images of the target scene, perform intra panoramic image mapping and self-pairing to obtain a first image pair set, the first image pair set comprising image pairs with mutual co-view regions within each panoramic image; For the plurality of panoramic images, perform inter panoramic image mapping and cross-image pairing to obtain a second image pair set, the second image pair set comprising inter panoramic image pairs with common image features for each of a plurality of matching key point pairs; For a candidate perspective image pair composed of any candidate perspective image from the first image pair set and any candidate perspective image from the second image pair set, connect the self-pairing and cross-image pairing relationships to obtain a third image pair set; the first image pair set, the second image pair set and the third image pair set form a complete image pair set; Input the complete image pair set into a pre-trained three-dimensional reconstruction model to obtain a local three-dimensional reconstruction result for each candidate perspective image in the complete image pair set; A pose-prior-driven global alignment is performed on the local 3D reconstruction results to obtain the panoramic 3D reconstruction results of the target scene.

[0005] A second aspect of the present invention provides a panoramic three-dimensional reconstruction device, the device comprising: The first pairing module is used to perform panoramic image in-panoramic mapping and self-pairing for each of the multiple panoramic images of the target scene to obtain a first image pair set, which includes image pairs with mutually shared viewing areas within each panoramic image. The second pairing module is used to perform panoramic image mapping and cross-image pairing for the multiple panoramic images to obtain a second image pair set. The second image pair set includes multiple panoramic image pairs with common image features for each matching key point pair. The third pairing module is used to connect self-pairing and cross-image pairing relationships for any candidate perspective image pair composed of any candidate perspective image from the first image pair set and any candidate perspective image from the second image pair set, so as to obtain a third image pair set; the first image pair set, the second image pair set, and the third image pair set constitute a complete image pair set; The local reconstruction module is used to input the complete image pair set into a pre-trained 3D reconstruction model to obtain the local 3D reconstruction result of each candidate perspective image in the complete image pair set. The global reconstruction module is used to perform pose prior-driven global alignment on the local 3D reconstruction results to obtain the panoramic 3D reconstruction results of the target scene.

[0006] A third aspect of the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the panoramic three-dimensional reconstruction method as described in the first aspect of the present invention.

[0007] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the panoramic three-dimensional reconstruction method as described in the first aspect of the present invention.

[0008] In the panoramic 3D reconstruction method proposed in this invention, firstly, in the preprocessing stage, for each panoramic image among multiple panoramic images of the target scene, panoramic image intra-image mapping and self-pairing are performed to obtain a first image pair set. Secondly, for multiple panoramic images of the target scene, panoramic image inter-image mapping and cross-image pairing are performed to obtain a second image pair set, thereby constructing a position-aware viewpoint pairing map. Only spatially adjacent and semantically related image pairs are selected for matching, thus significantly reducing the number of image pairs while retaining sufficient common-view information, providing a foundation for stable reconstruction. Furthermore, to maintain the structural connectivity of global pose chain alignment, candidate perspective images in the first and second image pair sets are paired, connecting the self-pairing and cross-image pairing relationships to construct a third image pair set, ultimately forming a complete image pair set. Then, the complete image pair set is input into a pre-trained 3D reconstruction model to obtain the local 3D reconstruction result of each candidate perspective image in the complete image pair set. Finally, in the testing and optimization stage, pose prior-driven global alignment is performed on the local 3D reconstruction results output by the 3D reconstruction model to obtain the panoramic 3D reconstruction result of the target scene. Thus, this invention can directly adapt the 3D reconstruction model based on pinhole image pre-training to the panoramic image input scene without any retraining, thus eliminating the dependence on panoramic training data and achieving accurate and efficient panoramic 3D reconstruction. Furthermore, in the preprocessing stage, a position-aware image pairing strategy is proposed to minimize the number of projections while ensuring sufficient common viewing area, effectively balancing geometric coverage and computational efficiency, and ensuring the stability of the overall panoramic image reconstruction. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating the steps of a panoramic 3D reconstruction method according to an embodiment of the present invention; Figure 2 This is a structural diagram illustrating a training-free panoramic 3D reconstruction method according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating a training-free panoramic 3D reconstruction method according to an embodiment of the present invention. Figure 4 This is a structural block diagram of a panoramic three-dimensional reconstruction device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] In recent years, end-to-end multi-view Figure Three 360° panoramic image reconstruction methods directly regress dense point clouds and relative camera poses from image pairs, effectively simplifying the traditional process that relies on keypoint extraction, feature matching, and incremental optimization. These methods typically use pinhole images as input and exhibit good reconstruction accuracy with structured training data. However, these methods are all based on the assumption of pinhole image modeling and have poor adaptability to 360° panoramic images. A common alternative is to project the panoramic image into multiple pinhole viewpoints and then use the above model for pairwise image matching and reconstruction. A typical implementation of this approach is to directly combine a fixed number of perspective images as image pairs for inference. However, this approach has two problems: first, the large number of projected images with repetitive structures means that exhaustive matching will lead to an exponential increase in computational resources and time overhead, severely limiting inference efficiency; second, if image segmentation and matching are only performed at the projection level without considering their spatial geometric relationships, it is difficult to guarantee sufficient common viewing areas between pairs, making the reconstruction results susceptible to sparse coverage or local mismatches. In practical applications, this approach can easily lead to problems such as pose drift during the global point cloud alignment stage, especially when the number of input images is small or the geometric overlap between images is weak.

[0013] In summary, although end-to-end 3D reconstruction models have made significant progress on pinhole images in recent years, their direct application to panoramic images still faces key challenges such as model structure incompatibility, low image pair matching efficiency, and lack of spatial consistency constraints. The root causes of these problems are twofold: firstly, panoramic images inherently possess spherical structures and nonlinear projection characteristics, making it difficult to adapt traditional models through simple projection and stitching; secondly, current methods lack pairing mechanisms and pose prior modeling tailored to the structural features of panoramic images during image pairing and alignment, thus limiting their 3D reconstruction performance in real-world panoramic scenes.

[0014] Based on this, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, this invention aims to seamlessly transfer a 3D reconstruction model trained on pinhole images to a 360° panoramic image input scene, achieving accurate and stable 3D reconstruction without any retraining. To this end, this invention designs a panoramic 3D reconstruction method, which includes two key stages: a preprocessing stage and a test optimization stage. The preprocessing stage mainly performs position-aware viewpoint pairing, ensuring that the input of the pre-trained 3D reconstruction model has a sufficient common-view area while avoiding redundancy. The test optimization stage introduces a global point cloud alignment mechanism based on pose priors, effectively improving global consistency and suppressing pose drift, thereby achieving globally consistent sparse multi-view reconstruction without the need for camera intrinsic parameters, and improving the accuracy of panoramic reconstruction.

[0015] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a panoramic 3D reconstruction method according to an embodiment of the present invention. Figure 1 As shown, the panoramic 3D reconstruction method provided in this embodiment includes at least the following steps: Step S11: For each panoramic image in the multiple panoramic images of the target scene, perform panoramic image intra-image mapping and self-pairing to obtain the first set of image pairs.

[0016] In this embodiment, multiple panoramic images can be captured for the target scene. For each of the multiple panoramic images of the target scene, in-panoramic mapping and self-pairing can be performed in each panoramic image to obtain a first image pair set. Here, "self-pairing" refers to the pairing of candidate perspective images within the same panoramic image. The first image pair set is a self-paired image pair set, which includes image pairs with mutually shared viewing areas within each of the multiple panoramic images; an image pair includes two matched candidate perspective images.

[0017] Step S12: For the multiple panoramic images, perform panoramic image mapping and cross-image pairing to obtain a second set of image pairs.

[0018] In this embodiment, for multiple panoramic images of a target scene, matching key points between different panoramic images can be determined, resulting in multiple matching key point pairs. Based on these multiple matching key point pairs, panoramic image mapping and cross-image pairing are performed to obtain a second image pair set. Here, "cross-image pairing" refers to the pairing of candidate perspective images between different panoramic images. The second image pair set is a cross-image pair set, comprising panoramic image pairs with common image features from multiple matching key point pairs in the multiple panoramic images.

[0019] Step S13: For any candidate perspective image pair formed by any candidate perspective image from the first image pair set and any candidate perspective image from the second image pair set, connect the self-pairing and cross-image pairing relationships to obtain a third image pair set; the first image pair set, the second image pair set, and the third image pair set constitute a complete image pair set.

[0020] In this embodiment, to maintain the structural connectivity of the global pose chain alignment, after obtaining the first image pair set and the second image pair set, for any candidate perspective image pair composed of any candidate perspective image from the first image pair set and any candidate perspective image from the second image pair set, the self-pairing and cross-image pairing relationships are connected to construct a third image pair set. In this embodiment, the first image pair set, the second image pair set, and the third image pair set can form a complete image pair set, which is used as the input to the pre-trained 3D reconstruction model.

[0021] Step S14: Input the complete image pair set into the pre-trained 3D reconstruction model to obtain the local 3D reconstruction result of each candidate perspective image in the complete image pair set.

[0022] In this embodiment, the complete image pair set includes multiple candidate perspective images. The complete image pair set can be input into a pre-trained 3D reconstruction model to obtain the local 3D reconstruction result of each candidate perspective image in the complete image pair set, output by the pre-trained 3D reconstruction model. It is understood that in this embodiment, "local" corresponds to the perspective image in a panoramic image, and "global" corresponds to the panoramic image; the local 3D reconstruction result of the candidate perspective image is the 3D reconstruction result of the candidate perspective image. The pre-trained 3D reconstruction model in this embodiment is a 3D reconstruction model pre-trained based on pinhole images, such as DUSt3R; this embodiment does not impose any limitations on this.

[0023] Step S15: Perform pose prior-driven global alignment on the local 3D reconstruction results to obtain the panoramic 3D reconstruction results of the target scene.

[0024] In this embodiment, after obtaining the local 3D reconstruction results output by the model, global alignment driven by pose prior can be performed on the local 3D reconstruction results of each candidate perspective image, followed by panoramic image pose fusion to obtain the panoramic 3D reconstruction results of the target scene, thereby realizing the reconstruction of the panoramic image corresponding to the target scene. Thus, this embodiment can complete the 3D reconstruction of panoramic images without model structure modification or training, and still exhibits good matching and pose consistency even when the input image is sparse.

[0025] In this embodiment, the 3D reconstruction model pre-trained based on pinhole images can be directly adapted to the panoramic image input scene without any retraining, thus eliminating the dependence on panoramic training data and achieving accurate and efficient panoramic 3D reconstruction. Furthermore, a position-aware image pairing strategy is proposed in the preprocessing stage, which minimizes the number of projections while ensuring sufficient common viewing area, effectively balancing geometric coverage and computational efficiency, and ensuring the stability of the overall panoramic image reconstruction.

[0026] In conjunction with the above embodiments, in one implementation, the present invention also provides a panoramic three-dimensional reconstruction method, in which step S11 may specifically include steps S21 and S22: Step S21: Perform spherical uniform sampling on each of the multiple panoramic images of the target scene to generate a set of candidate perspective images with uniform directional coverage corresponding to the panoramic image.

[0027] In this embodiment, the purpose of panoramic image intramapping and self-pairing is to uniformly project each panoramic image into several subsequent perspective images with shared viewing areas. For each panoramic image among multiple panoramic images of the target scene, spherical uniform sampling can be performed on each panoramic image to generate a set of candidate perspective images (i.e., multiple candidate perspective images) with uniform directional coverage corresponding to that panoramic image.

[0028] Step S22: For each panoramic image, a set of candidate perspective images is used to retain only the candidate perspective image pairs whose viewing angle does not exceed the angle threshold, so as to realize the panoramic image intra-mapping and self-pairing, and obtain the first image pair set.

[0029] In this embodiment, for a set of candidate perspective images corresponding to each panoramic image, only candidate perspective image pairs with a viewing angle not exceeding the angle threshold are retained to obtain the first image pair corresponding to the panoramic image, so as to realize the intra-panoramic mapping and self-pairing, thereby obtaining a first image pair set composed of the first image pairs corresponding to multiple panoramic images.

[0030] In an optional example, for each panoramic image, the field of view (FoV) projected onto the pinhole image can be set as follows: Calculate the minimum number of projections (i.e., the number of required viewing angles) N, ensuring that each spherical point is covered by at least two viewing angles (i.e., each spherical point is observed by at least two different pinhole projections). Then, use Fibonacci spherical uniform sampling to generate N projection direction vectors, each direction defined by a rotation matrix. Mapping to the corresponding viewpoint generates candidate perspective images (i.e., pinhole images). To reduce the number of candidate perspective images to be paired, this embodiment adopts a pairing strategy based on angular similarity, effectively reducing redundancy. A preset included angle threshold (which can be freely set according to requirements without specific limitations, such as the included angle threshold being the field of view angle) is used. Construct candidate perspective image pairs and retain only image pairs whose viewpoint angle does not exceed the angle threshold.

[0031] Specifically, in an optional example, Example 1 is shown: Let the input panoramic image be The field of view is Then the area of ​​the spherical cap covered by each projection on the sphere is: ; To ensure that every point on a unit sphere is covered by at least two viewpoints, the required number of projected viewpoints is:

[0032] Next, to uniformly distribute these N projection directions on the sphere, Fibonacci spherical sampling is performed as follows to generate... Projection direction vectors :

[0033]

[0034]

[0035] in, Using the golden angle, we ensure that the sampling direction has an approximately uniform angular interval on the sphere, that is, to ensure that the viewing angle is distributed as evenly as possible on a unit sphere. Then, a rotation matrix is ​​constructed for each projected viewing angle. ,Should To align the sphere with the projection direction The rotation matrix.

[0036] Each panoramic image Projecting as candidate perspective images A set of candidate perspective images with uniform directional coverage is obtained, and the projection relationship is as follows:

[0037] in: This is the projection function from a spherical plot to an isometric rectangular plot. Let be the back projection function of the pinhole camera model. The intrinsic parameter matrix of the pinhole camera can be defined as follows:

[0038] in, These are the width and height (in pixels) of the output image (i.e., the generated candidate perspective image).

[0039] After obtaining a set of candidate perspective images corresponding to each panoramic image, considering that pairing all candidate perspective images together will result in a high... The number of combinations is high, and the computational cost is high. This embodiment adopts a pairing strategy based on angular similarity to effectively reduce redundancy: for any two candidate perspective images, the viewpoint direction... , If the following conditions are met:

[0040] This implies the existence of a shared viewing area, meaning the viewing direction is preserved. The angle of view does not exceed the field of view. Candidate perspective image pairs constitute a valid first image pair. Thus, the first image pair set is obtained. .

[0041] Thus, this embodiment decomposes each panoramic image into a set of overlapping perspective views through intra-panorama mapping and pairing. This embodiment employs a uniform viewpoint sampling and angle filtering mechanism to ensure sufficient common viewing areas with minimal redundancy, forming a position-aware pairing strategy: to reduce computational and memory overhead, it aims to achieve complete and uniform coverage of the sphere with the minimum number of pinhole views, and through a pairing strategy based on angular similarity, it not only ensures common viewing between candidate perspective images but also avoids expensive pairwise feature matching or explicit overlapping region detection.

[0042] In conjunction with any of the above embodiments, the present invention also provides a panoramic three-dimensional reconstruction method, in which step S12 may specifically include steps S31 to S34: Step S31: For each pair of panoramic images in the multiple panoramic images, including the first panoramic image and the second panoramic image, determine M initial matching key point pairs with common image features between the first panoramic image and the second panoramic image.

[0043] While intra-image mapping and pairing within a single panoramic image can effectively guarantee shared viewing areas among pinhole images (i.e., candidate perspective images), establishing a globally consistent 3D structure across multiple panoramic images requires robust cross-image pairing relationships. Therefore, this embodiment also performs inter-panoramic image mapping and cross-image pairing during the preprocessing stage. This process aims to automatically identify key regions with significant visual similarities in different panoramic images and generate high-quality cross-image pairings centered on these regions.

[0044] Specifically, multiple panoramic images can be randomly divided into multiple pairs of panoramic images. Each pair of panoramic images includes a first panoramic image and a second panoramic image. The first panoramic image and the second panoramic image are either panoramic images from the multiple panoramic images of the target scene, and the first panoramic image and the second panoramic image are different.

[0045] This embodiment can determine M initial matching keypoint pairs with common image features between the first panoramic image and the second panoramic image contained in each pair of panoramic images. Each initial matching keypoint pair includes two keypoints with common image features from the first panoramic image and the second panoramic image, respectively.

[0046] In an optional example, for two different panoramic images SuperPoint and SuperGlue can be used to extract matching key points. And its corresponding matching relationship (such as matching confidence), thus obtaining M initial matching keypoint pairs. .in: For the coordinates of the key points, for The corresponding matching confidence, As the confidence threshold, only retain The matching keypoint pairs are used to obtain M initial matching keypoint pairs to improve matching reliability.

[0047] Step S32: Based on the panoramic image height of the multiple panoramic images and the ordinate of each matching key point in the M initial matching key point pairs, a first screening is performed with the goal of suppressing polar distortion interference to obtain multiple intermediate matching key point pairs.

[0048] This embodiment considers the significant distortion in the upper and lower polar regions of the equirectangular projection image. Therefore, after obtaining M initial matching keypoint pairs corresponding to each pair of panoramic images, a first screening of these M initial matching keypoints is performed based on the panoramic image height of the panoramic image. Specifically, this embodiment can perform the first screening based on the panoramic image height of multiple panoramic images and the ordinate of each matching keypoint in the corresponding M initial matching keypoint pairs, with the goal of suppressing polar distortion interference, to obtain multiple intermediate matching keypoint pairs corresponding to each pair of panoramic images.

[0049] In one alternative embodiment, key points on the top and bottom quarter edges of the panoramic image can be excluded. Let the height of the panoramic image be... , To match the y-coordinates of key points, the key points retained during the first filtering must meet the following conditions: This results in multiple intermediate matching keypoint pairs (used to suppress polar distortion interference).

[0050] Step S33: Divide the first panoramic image and the second panoramic image into multiple grids of target size. Based on the multiple intermediate matching key point pairs, select the point with the highest matching confidence in each grid to form the intermediate matching key point set of the first panoramic image and the second panoramic image respectively. Use non-maximum suppression to retain intermediate matching key point pairs with a spacing not less than the target spacing to obtain multiple target matching key point pairs.

[0051] In this embodiment, the first panoramic image and the second panoramic image can be divided into multiple grids of the target size (e.g., divided into...). The keypoints in the multiple intermediate matching keypoint pairs are distributed across the divided grids. Then, based on these intermediate matching keypoint pairs, the keypoint with the highest matching confidence is selected from each grid to form the intermediate matching keypoint sets for the first and second panoramic images, respectively. The target spacing is obtained based on the panoramic image width of the multiple panoramic images.

[0052] For example: grid point 1 in the first panoramic image includes key point 1 in intermediate matching key point pair 1 and key point 2 in intermediate matching key point pair 2; grid point 2 in the second panoramic image includes key point 1' in intermediate matching key point pair 1 and key point 2' in intermediate matching key point pair 2; wherein, the matching confidence of intermediate matching key point pair 1 is higher than the matching confidence of intermediate matching key point pair 2, then key point 1 is selected from grid point 1 in the first panoramic image and key point 1' is selected from grid point 2 in the second panoramic image to form intermediate matching key points in the intermediate matching key point sets of the first and second panoramic images respectively.

[0053] After obtaining the intermediate matching keypoint sets for the first and second panoramic images, to further filter out spatially balanced keypoints, non-maximum suppression (NMS) can be used to retain intermediate matching keypoint pairs with a spacing not less than the target spacing, performing a second filtering to obtain multiple target matching keypoint pairs corresponding to the first and second panoramic images. For example, for the intermediate matching keypoint sets of the first and second panoramic images, the spacing between matching keypoints can be retained to be not less than the target spacing. The intermediate matching keypoint pairs yield multiple target matching keypoint pairs, among which... A scaling factor set for experience. The width of the panoramic image.

[0054] Step S34: Based on each target matching key point pair, generate candidate perspective images corresponding to the first panoramic image and the second panoramic image respectively, so as to realize the mapping between panoramic images and cross-image pairing, and obtain the second image pair set.

[0055] In this embodiment, a robust cross-panoramic projection direction can be constructed based on each target matching keypoint pair. Candidate perspective images corresponding to the first and second panoramic images are generated using these directions as centers, thereby achieving inter-panoramic image mapping and cross-image pairing to obtain a second image pair set. The second image pair set includes panoramic image pairs for each target matching keypoint pair; that is, the candidate perspective images corresponding to the first and second panoramic images generated based on each target matching keypoint pair constitute a second image pair within the second image pair set.

[0056] In an optional example, for each of the multiple target matching keypoint pairs, the same method of projecting a panoramic image into a candidate perspective image as described in Embodiment 1 above can be used to generate candidate perspective images corresponding to the first panoramic image and the second panoramic image, respectively, to obtain a second image pair, thereby forming a cross-image pairing set, i.e., the second image pair set. .

[0057] In this embodiment, to ensure global consistency, key point pairs are extracted between different panoramic images. Confidence filtering, polar cropping, grid sampling, and NMS are used to construct spatially uniform and reliable cross-panorama key point pairs. Then, based on the directions pointed to by these key point pairs, corresponding cross-panorama projection images are generated, and a set of cross-image pairs is constructed. Thus, through inter-panorama mapping and cross-panorama pairing in this embodiment, reliable matching pairs can be found between different panoramic images, guiding the generation of additional perspective projections, enhancing global connectivity, and mitigating pose drift.

[0058] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a panoramic 3D reconstruction method. In this method, and specifically, step S13 above, "for a candidate perspective image pair composed of any candidate perspective image from the first image pair set and any candidate perspective image from the second image pair set, connect the self-pairing and cross-image pairing relationships to obtain a third image pair set," may include the following steps S41 to S42: Step S41: Traverse any candidate perspective image from the first image pair set and traverse any candidate perspective image from the second image pair set to obtain multiple candidate perspective image pairs.

[0059] In this embodiment, after obtaining the first image pair set and the second image pair set, any candidate perspective image from the first image pair set and any candidate perspective image from the second image pair set can be traversed to obtain multiple candidate perspective image pairs. Each candidate perspective image pair includes: a candidate perspective image from the first image pair set and a candidate perspective image from the second image pair set.

[0060] Step S42: With connected self-pairing and cross-graph pairing as the target, only candidate perspective image pairs with a viewpoint angle not exceeding the angle threshold are retained to obtain the third image pair set.

[0061] In this embodiment, for the multiple candidate perspective image pairs obtained in the previous step, with the goal of connecting self-pairing and cross-image pairing relationships, only candidate perspective image pairs in the multiple candidate perspective image pairs whose viewpoint angle does not exceed the angle threshold are retained, thus obtaining the connection set, namely the third image pair set.

[0062] In an alternative instance, to maintain the structural connectivity of the global pose chain alignment, the second image pair set is... and the first image pair set The images in the image set are paired according to the viewing direction using the same image pairing method as in Example 1 above, thus constructing a connection set. That is, a third set of image pairs is constructed, connecting self-pairing and cross-graph pairing relationships to finally form a complete set of image pairs: .

[0063] In this embodiment, after obtaining the first set of image pairs and the second set of image pairs, in order to maintain the connectivity of the graph structure, auxiliary connections are further established with the goal of connecting self-pairing and cross-graph pairing relationships, that is, a third set of image pairs is constructed, thereby maintaining the structural connectivity of global pose chain alignment.

[0064] In summary, this invention constructs position-aware viewpoint pairings during the preprocessing stage, aiming to transform each input panoramic image into a set of pinhole images with uniform viewpoints, minimal redundancy, and spatial structural connectivity, along with their pairing structures. .

[0065] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a panoramic 3D reconstruction method. In this method, after obtaining a complete set of image pairs, steps S51 to S53 may be further included, and step S14 may specifically include step S54: Step S51: For each candidate perspective image in the complete image pair set, determine the semantic vector of the candidate perspective image.

[0066] In this embodiment, a semantic filtering mechanism is provided for single panoramic image reconstruction or sparse image input scenarios: First, for each candidate perspective image in the complete image pair set, the semantic vector of the candidate perspective image can be determined. For example, a CLIP encoder (such as ViT-B / 32) can be used to generate a global feature vector (i.e., semantic vector) for each candidate perspective image in the complete image pair set.

[0067] Step S52: Determine the semantic similarity based on the semantic vector of each candidate perspective image pair in the complete image pair set.

[0068] In this embodiment, the semantic similarity of each candidate perspective image pair is determined based on its semantic vector in the complete image pair set, such as by determining the cosine similarity.

[0069] Step S53: Retain the top Q candidate perspective image pairs with the highest semantic similarity to obtain the final complete image pair set after semantic filtering.

[0070] In this embodiment, based on the semantic similarity of each candidate perspective image pair, the top Q (an integer greater than 1, which can be freely set without restriction) candidate perspective image pairs with the highest semantic similarity are retained to complete the semantic filtering and obtain the final complete image pair set after semantic filtering.

[0071] In an optional example, for candidate perspective images semantic vectors For any candidate perspective image Calculate semantic similarity: ; Before retaining each image The most similar image pairs are selected as valid pairing candidates, and there is a 50% probability that the number of pairs to be retained will be randomly perturbed. The final set of complete image pairs after semantic filtering is denoted as . .

[0072] Step S54: Input the final complete image pair set into the pre-trained 3D reconstruction model to obtain the local 3D reconstruction result of each candidate perspective image in the final complete image pair set.

[0073] In this embodiment, after obtaining the final complete image pair set, the final complete image pair set is input into a pre-trained 3D reconstruction model to obtain the local 3D reconstruction result of each candidate perspective image in the final complete image pair set.

[0074] In this embodiment, to further optimize the number of image matching pairs, a semantic feature-based semantic awareness image filtering strategy is introduced. This strategy eliminates redundant matching pairs, further reducing the number of matching pairs and retaining the most semantically consistent image pairs. In one embodiment, the semantic feature-based semantic awareness image filtering strategy (i.e., semantic awareness match filtering) is only enabled in the scenario of single panoramic image reconstruction (i.e., only one panoramic image is provided for each target scene) to filter redundant or low-quality image pairings. In the scenario of multi-panoramic image reconstruction (multiple panoramic images are uniformly sampled from each target scene), to maintain cross-view geometric connectivity, this strategy is not executed and the complete set of unfiltered image pairs is used directly. .

[0075] In the scenario of single panoramic image reconstruction, only intra-panoramic image mapping and self-pairing are performed to obtain the first image pair set. The first image pair set is then input into the pre-trained 3D reconstruction model to obtain the local 3D reconstruction result of each candidate perspective image in the first image pair set.

[0076] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a panoramic 3D reconstruction method. In this method, step S15 may at least include steps S61 to S64: Step S61: Treat the candidate perspective images originating from the same panoramic image in the complete image pair set as a group of homologous perspective images, select the candidate perspective image with the highest local pose confidence in the group of homologous perspective images as the reference image, and determine the global pose of the reference image.

[0077] In this embodiment, the local 3D reconstruction result includes at least local point cloud, local pose, and local pose confidence. To minimize registration error across panoramic images, this embodiment processes multiple candidate perspective images from the same panoramic image as a group. Candidate perspective images originating from the same panoramic image in the complete image pair set can be considered as a group of homologous perspective images. For each group of homologous perspective images... The candidate perspective image with the highest local pose confidence among the group of source perspective images was selected as the reference image. And determine the global pose of the reference image. .

[0078] This embodiment uses each group of perspective images from the same source as the processing object for global alignment. It utilizes the spatial structural consistency of these images, which originate from the same panoramic image, to avoid pose drift and point cloud misalignment when the viewpoint is sparse or the shared viewing area is small, thereby improving the stability of the overall reconstruction.

[0079] Step S62: Based on the global pose and local pose of the reference image, and the local poses of other candidate perspective images in the group, the global pose of the reference image is propagated to the other candidate perspective images in the group using pose prior, thereby realizing the conversion from local pose to global pose and obtaining the globally aligned poses of the other candidate perspective images in the group.

[0080] In this embodiment, other candidate perspective images within the group are candidate perspective images other than the reference image among the same source perspective images in the group. To reduce pose drift of images originating from the same panoramic view, the global pose of the reference image can be propagated to the other candidate perspective images within the group based on the global and local poses of the reference image, as well as the local poses of the other candidate perspective images within the group, using pose priors. This achieves the transformation from local poses to global poses within the group, resulting in the globally aligned poses of the other candidate perspective images within the group. In this embodiment, the pose priors include the local poses of the reference image obtained during panoramic image projection and the local poses of the other candidate perspective images within the group.

[0081] In an optional example, pose propagation can be performed on other candidate perspective images within the group using the following formula: , ; in, For the global pose of the reference image, For the local pose of the reference image, For the local pose of other candidate perspective images within the group, The pose after global alignment of other candidate perspective images within the group.

[0082] Step S63: Based on the scaling factor and globally aligned pose of the other candidate perspective images in the group, and the local point cloud of the other candidate perspective images in the group in the local 3D reconstruction result, obtain the globally aligned point cloud of the other candidate perspective images in the group.

[0083] In this embodiment, the global point cloud of the other candidate perspective images in the group can be obtained based on the scaling factor of the other candidate perspective images in the group, the pose of the other candidate perspective images in the group after global alignment, and the local point cloud of the other candidate perspective images in the group in the local 3D reconstruction result.

[0084] In an optional example, the scaling factor of other candidate perspective images within the group can be estimated using the Procrustes alignment method. By combining the globally aligned poses of other candidate perspective images within the group, the local point clouds of the other candidate perspective images within the group are transformed to the world coordinate system, resulting in the globally aligned point clouds of the other candidate perspective images within the group: ; in, This is the point cloud after global alignment with other candidate perspective images within the group. The pose is the globally aligned version of the other candidate perspective images within the group. This represents the local pose of other candidate perspective images within the group. The scaling factor is the local point cloud. The scale relative to the point cloud of the image that has been globally registered.

[0085] Step S64: Based on the scaling factor and global pose of the reference image, and the local point cloud of the reference image in the local 3D reconstruction result, obtain the global point cloud of the reference image.

[0086] In this embodiment, for the reference image, the global point cloud of the reference image can be obtained based on the scaling factor and global pose of the reference image, as well as the local point cloud of the reference image in the local 3D reconstruction result.

[0087] In an optional example, the scaling factor of the reference image can be estimated using the Procrustes alignment method. Combined with the global pose of the reference image The local point cloud of the reference image Transform to the world coordinate system to obtain the global point cloud of the reference image. : .

[0088] In this embodiment, the panoramic 3D reconstruction result of the panoramic image corresponding to the group of common perspective images includes: the global pose of the reference image in the group of common perspective images, the global point cloud of the reference image, and the globally aligned point cloud and pose of other candidate perspective images within the group of common perspective images. In this way, the panoramic 3D reconstruction result of the panoramic image corresponding to each group of common perspective images can be obtained, thereby obtaining the panoramic 3D reconstruction result of multiple panoramic images and realizing the reconstruction of multiple panoramic images.

[0089] In this embodiment, to reduce pose drift, candidate perspective images from the same panoramic image are grouped together. Only the candidate perspective image with the highest confidence is selected as the reference image to participate in cross-image Procrustes alignment, and its global pose is recorded. Then, the global pose of the reference image is combined with the local relative pose within the group and propagated to other viewpoint images within the group to realize the conversion from local pose within the group to global pose, and complete the pose prior-guided global alignment to reduce the propagation of reconstruction error and achieve a unified panoramic-level output.

[0090] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a panoramic 3D reconstruction method. In this method, step S15 may further include steps S71 to S74: Step S71: Determine the weight of the candidate perspective image based on the confidence level of the local point cloud of each candidate perspective image within a set of source perspective images.

[0091] In this embodiment, the local 3D reconstruction result also includes the confidence level of the local point cloud. The weight of the candidate perspective image can be determined based on the confidence level of the local point cloud of each candidate perspective image (i.e., including the reference image and other candidate perspective images in the group) within a set of source perspective images.

[0092] In an optional example, the weight of a candidate perspective image can be determined based on the product of the confidence level of the local point cloud of each candidate perspective image and the temperature adjustment coefficient, and the sum of the products of the confidence level of the local point cloud of each candidate perspective image within the same group of source perspective images and the temperature adjustment coefficient.

[0093] For example, the softmax weights of candidate perspective images can be defined based on the following formula: ; Among them, a set of perspective images from the same origin is recorded. for ,against Any candidate perspective image , Candidate perspective images The confidence level of the local point cloud. Candidate perspective images The weight, This is the temperature regulation coefficient, with a value of 3.

[0094] Step S72: Based on the translation vector and weight of each candidate perspective image within a set of source perspective images, obtain the displacement in the pose of the panoramic image corresponding to the set of source perspective images.

[0095] In this embodiment, the displacement in the pose of the panoramic image corresponding to the set of source perspective images can be obtained based on the translation vector of each candidate perspective image within a set of source perspective images and the calculated weight of each candidate perspective image. The translation vector of each candidate perspective image is the output of a pre-trained 3D reconstruction model.

[0096] In an optional example, a set of originating perspective images can be calculated using the following formula. The displacement in the pose of the corresponding panoramic image :

[0097] Among them, it is known Internal candidate perspective image The translation vector is , Candidate perspective images The weight.

[0098] Step S73: Based on the rotation matrix prior of each candidate perspective image in a set of source perspective images and the rotation matrix corresponding to the pose after global alignment, obtain the frontal pose of the panoramic image from which the candidate perspective image originates.

[0099] In this embodiment, the frontal pose of the panoramic image from which the candidate perspective image originates can be obtained based on the prior rotation matrix of each candidate perspective image within a set of source perspective images, and the rotation matrix corresponding to the globally aligned pose of each candidate perspective image within a set of source perspective images.

[0100] The prior rotation matrices of candidate perspective images within a set of source perspective images include: the rotation matrix in the local pose of the reference image obtained during the projection of the panoramic image (i.e., the process of mapping the panoramic image to candidate perspective images), and the rotation matrices in the local poses of other candidate perspective images within the group obtained during the projection of the panoramic image. The rotation matrices corresponding to the globally aligned poses of the candidate perspective images within a set of source perspective images include: the rotation matrix in the global pose of the reference image, and the rotation matrices in the globally aligned poses of other candidate perspective images within the group.

[0101] For example, candidate perspective images can be determined using the following formula. Direct frontal pose of the panoramic image. : ; in, Candidate perspective images Prior rotation matrix; Candidate perspective images The rotation matrix corresponding to the globally aligned pose is based on the candidate perspective image. Globally aligned pose The extracted result.

[0102] Step S74: Based on the displacement in the pose of the panoramic image corresponding to a set of source perspective images and the frontal pose of the panoramic image from which each candidate perspective image in the set of source perspective images originates, obtain the world camera pose of the panoramic image.

[0103] In this embodiment, the world camera pose of the panoramic image can be obtained based on the displacement in the pose of the panoramic image corresponding to a set of common perspective images, and the frontal pose of the panoramic image from which each candidate perspective image in the set of common perspective images originates. That is, the global pose of the panoramic image corresponding to the set of common perspective images can be obtained, and then the global pose of each of the multiple panoramic images can be obtained. Finally, the panoramic 3D reconstruction result of each of the multiple panoramic images can be obtained, thereby realizing the 3D reconstruction of multiple panoramic images.

[0104] In an optional example, after obtaining a set of homologous perspective images... Internal candidate perspective image Direct frontal pose of the panoramic image. after, Will Convert to unit quaternion And perform a weighted average: ; in, Group of perspective images from the same source The corresponding quaternion of the panoramic image, Candidate perspective images The weights. Finally, Convert back to homologous perspective image group Rotation matrix of the corresponding panoramic image Based on rotation matrix and homologous perspective image group The displacement in the pose of the corresponding panoramic image A group of perspective images from the same origin was obtained. The global pose of the corresponding panoramic image : .

[0105] In this embodiment, the image positions within a group are weighted and averaged using the local point cloud confidence level as the weight to obtain the translation of the panoramic image. The rotation matrix of the panoramic image is then recovered using the unit quaternion averaging method, ultimately yielding the complete camera pose of the panoramic image. .

[0106] Understandably, the panoramic 3D reconstruction result for each panoramic image includes: the complete camera pose for each panoramic image. , and the set of all aligned global point clouds (i.e., the globally aligned point clouds of other candidate perspective images in the group within this panoramic image, and the global point cloud of the reference image).

[0107] In other words, this embodiment introduces relative position information between viewpoints as pose priors to mitigate error accumulation during point cloud alignment. To this end, a global alignment strategy based on pose priors is proposed, utilizing known directional relationships between projections within the same panoramic image to enhance spatial consistency. This prior information is integrated into the global alignment process, effectively suppressing pose drift. Ultimately, this strategy achieves accurate alignment even in sparse viewpoints and low-overlap scenes.

[0108] In this embodiment, the point cloud of the reference image is scaled and combined with the obtained global pose to complete the transformation of the point cloud from local to global coordinates. Then, the image positions within the group are weighted and averaged using confidence as the weight to obtain the translation of the panoramic image. The rotation matrix of the panoramic image is then recovered using the unit quaternion averaging method, finally obtaining the complete camera pose (i.e., global pose) of the panoramic image. Thus, this embodiment further introduces the relative rotation information between viewpoints derived from projection geometry as a pose prior during the testing phase, integrating it into the global point cloud alignment process to avoid error accumulation caused by unstable matching paths during alignment. This achieves globally consistent sparse multi-view 3D structure reconstruction without relying on known camera intrinsic parameters, effectively improving the reconstruction accuracy in 360° image scenes.

[0109] In one embodiment, such as Figure 2 As shown, Figure 2 This is a structural diagram illustrating a training-free panoramic 3D reconstruction method according to an embodiment of the present invention. Figure 2 The preprocessing stage includes three main steps: the first step is panoramic image intra-image mapping and pairing, which uniformly projects each panoramic image into several perspective images with shared viewing areas. The second step is the mapping and pairing between panoramic images. This step extracts keypoint pairs between different panoramic images and uses confidence filtering, polar cropping, grid sampling, and NMS to construct spatially uniform and reliable cross-image keypoint pairs. Then, based on the directions pointed to by these keypoint pairs, corresponding cross-panoramic projection images are generated, and a set of cross-image pairs is constructed. To maintain the connectivity of the graph structure, auxiliary connections are further established. To form a complete set of image pairs The final step is optional—semantic-aware matching filtering, which provides an optional semantic filtering mechanism in single-image reconstruction or sparse image input scenarios.

[0110] After obtaining the mapped image and matching pairs in the preprocessing stage, image pair inference is performed using DUST3R to obtain the output local point cloud and pose. Then, in the test-time optimization stage, global alignment guided by pose prior is first performed based on the local point cloud and pose, followed by panoramic image pose fusion. This transforms the local point cloud and local pose output after the preprocessing stage into global point cloud and global pose, and restores the camera pose of the panoramic image.

[0111] In one embodiment, such as Figure 3 As shown, Figure 3 This is a flowchart illustrating a training-free panoramic 3D reconstruction method according to an embodiment of the present invention. Figure 3 The workflow is as follows: Step 1: Perform the preprocessing stage to construct location-aware viewpoint pairings: This step aims to transform each input panoramic image into a set of pinhole images with uniform viewpoints, minimal redundancy, and spatial structural connectivity, along with their paired structures.

[0112] Step 2: Perform image pair inference: such as using DUSt3R for image pair inference. Set the image pair set... (or matching without semantic awareness) The input is fed into the pre-trained 3D reconstruction model DUSt3R, and for each image pair... Output local point cloud Point cloud confidence and local camera pose ,in The number of points.

[0113] Step 3: Optimize during testing by performing global alignment driven by pose priors. This stage uses pose priors for global alignment, reducing reconstruction error propagation and achieving a unified panoramic output.

[0114] In one embodiment, given an ordered set of panoramic images (such as Image I) i Image I i+1 …Image I nFirst, in the preprocessing stage, pinhole projection and viewpoint selection are performed based on the projection direction and feature center to generate and pair perspective views, ensuring that each image pair has a stable geometric overlap area, generating a compact set of pinhole image pairs with shared viewing areas. This embodiment designs a position-aware image pairing strategy in the preprocessing stage. This strategy filters perspective image pairs based on angular proximity and feature similarity, minimizing the number of projections while ensuring sufficient shared viewing areas, effectively balancing geometric coverage and computational efficiency. In other words, this embodiment performs spherical uniform sampling on each panoramic image in the preprocessing stage, generating a set of candidate perspective views with uniform directional coverage. A position-aware viewpoint pairing map is constructed using the viewing angle and image feature similarity, selecting only spatially adjacent and semantically related image pairs for matching. This significantly reduces the number of image pairs while retaining sufficient shared viewing information, providing a foundation for stable reconstruction.

[0115] Subsequently, these image pairs are input into a pre-trained DUSt3R model to obtain local 3D reconstruction results (including local point clouds and local poses). During the test-time optimization phase, a coarse pose initialization is performed based on the local point clouds, and then the camera pose is corrected based on the local pose as a pose prior. The global point clouds are then aligned to obtain the global point cloud and global pose, thereby achieving accurate 360° 3D reconstruction without training. This embodiment introduces a pose prior-based global alignment strategy during the test-time optimization phase. This strategy utilizes known spatial orientation information between projected views within the same panoramic image to suppress pose drift during global alignment, thereby improving global pose consistency and reducing error accumulation.

[0116] To evaluate the reconstruction accuracy, pose estimation capability, and operational efficiency of the panoramic 3D reconstruction method proposed in this invention in real-world scenarios, this invention underwent systematic validation on the publicly available Matterport3D 360°RGBD dataset. This dataset provides high-resolution isometric panoramic images, corresponding depth maps, and real camera poses, making it representative and comparable to current representative methods DUSt3R and MASt3R. The camera pose evaluation criteria used in this invention include: relative rotation accuracy (RRA), relative translation accuracy (RTA), and average accuracy over a 30-degree angle range (mAA@30); the reconstruction quality evaluation criteria include: reconstruction accuracy (Acc), integrity (Comp), and normal consistency (NC); efficiency metrics include: number of image pairs, average inference time, and peak memory usage. The testing environment for this invention was an Ubuntu 20.04 operating system, equipped with an Intel Xeon Gold 6330 CPU (2.00GHz) and an NVIDIA RTX 4090 GPU (2235MHz core frequency, 24GB memory).

[0117] As shown in Table 1, the present invention achieves 100% pose estimation accuracy in single-view scenarios, which is significantly better than existing methods, demonstrating its strong robustness and high pose estimation capability under sparse input.

[0118] Table 1 Comparison of Camera Pose Estimation Accuracy

[0119] As shown in Table 2, under single-frame input conditions, the method of this invention significantly improves surface accuracy and integrity through position-aware viewpoint pairing and attitude-prior-guided global alignment. In contrast, MASt3R suffers from reconstruction fragmentation and deformation issues.

[0120] Table 2 Comparison of 3D Reconstruction Quality

[0121] As shown in Table 3, while maintaining leading reconstruction quality, this invention effectively reduces redundant image pairs through a position-aware viewpoint pairing strategy, thereby reducing overall memory consumption and inference time in tandem.

[0122] Table 3 Comparison of Reasoning Efficiency and Resource Consumption

[0123] In summary, this invention proposes a 3D reconstruction method that can adapt to 360° panoramic image input without training. First, in one embodiment of this invention, position-aware image pairing is performed during the preprocessing stage. A spherical Fibonacci uniform sampling and angle constraint strategy are used to control the generation of perspective views with uniform orientation. Further, image feature similarity is used for filtering, retaining only image pairs with geometric co-viewing or semantic consistency, effectively reducing the number of redundant pairs and significantly reducing redundant image pairs while maintaining uniform view coverage. Second, to improve the stability and consistency of global pose estimation, one embodiment of this invention designs a pose prior-guided global point cloud alignment mechanism during the design optimization stage. Utilizing the known relative directional relationships between multiple views from the same panoramic image, a reference image-guided pose propagation path is established for local pose propagation. Combined with Procrustes alignment, unified fusion of point clouds and camera poses from different views is achieved, thereby alleviating the error accumulation problem during multi-image fusion and improving global consistency. Furthermore, one embodiment of this invention introduces a semantically aware matching module, further reducing the number of image matching pairs through CLIP feature similarity filtering, thereby further reducing computational overhead. Ultimately, the overall method of this invention requires no modification to the model structure or participation in the training phase, possessing high versatility and deployment flexibility. While maintaining the end-to-end training-free characteristic, it significantly improves the geometric accuracy, pose stability, and computational cost of 3D reconstruction. Experimental results show that this invention can achieve high-quality sparse 3D reconstruction and high-precision pose estimation under panoramic image input conditions.

[0124] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0125] Based on the same inventive concept, one embodiment of the present invention provides a panoramic three-dimensional reconstruction device. (Reference) Figure 4 , Figure 4 This is a structural block diagram of a panoramic three-dimensional reconstruction device provided in an embodiment of the present invention. Figure 4 As shown, the panoramic 3D reconstruction device includes: The first pairing module is used to perform panoramic image in-panoramic mapping and self-pairing for each of the multiple panoramic images of the target scene to obtain a first image pair set, which includes image pairs with mutually shared viewing areas within each panoramic image. The second pairing module is used to perform panoramic image mapping and cross-image pairing for the multiple panoramic images to obtain a second image pair set. The second image pair set includes multiple panoramic image pairs with common image features for each matching key point pair. The third pairing module is used to connect self-pairing and cross-image pairing relationships for any candidate perspective image pair composed of any candidate perspective image from the first image pair set and any candidate perspective image from the second image pair set, so as to obtain a third image pair set; the first image pair set, the second image pair set, and the third image pair set constitute a complete image pair set; The local reconstruction module is used to input the complete image pair set into a pre-trained 3D reconstruction model to obtain the local 3D reconstruction result of each candidate perspective image in the complete image pair set. The global reconstruction module is used to perform pose prior-driven global alignment on the local 3D reconstruction results to obtain the panoramic 3D reconstruction results of the target scene.

[0126] Optionally, the first pairing module includes: The sampling module is used to perform spherical uniform sampling on each of the multiple panoramic images of the target scene to generate a set of candidate perspective images with uniform directional coverage corresponding to the panoramic image. The self-pairing module is used to retain only candidate perspective image pairs whose viewing angle does not exceed the angle threshold for a set of candidate perspective images corresponding to each panoramic image, so as to realize panoramic image intra-mapping and self-pairing and obtain the first image pair set.

[0127] Optionally, the second pairing module includes: The point matching module is used to determine M initial matching key point pairs with common image features between the first panoramic image and the second panoramic image for each pair of panoramic images in the multiple panoramic images. The first filtering module is used to perform a first filtering based on the panoramic image height of the multiple panoramic images and the ordinate of each matching key point in the M initial matching key point pairs, with the goal of suppressing polar distortion interference, to obtain multiple intermediate matching key point pairs. The second filtering module is used to divide the first panoramic image and the second panoramic image into multiple grids of target size, and based on the multiple intermediate matching key point pairs, select the point with the highest matching confidence in each grid to form the intermediate matching key point set of the first panoramic image and the second panoramic image respectively, and use non-maximum suppression to retain intermediate matching key point pairs with a spacing not less than the target spacing to obtain multiple target matching key point pairs, wherein the target spacing is obtained based on the panoramic image width of the multiple panoramic images; The cross-image pairing module is used to generate candidate perspective images corresponding to the first panoramic image and the second panoramic image based on each target matching key point pair, so as to realize the mapping between panoramic images and cross-image pairing, and obtain the second image pair set, which includes panoramic image pairs of each target matching key point pair.

[0128] Optionally, the third pairing module includes: The traversal module is used to traverse any candidate perspective image from the first image pair set and any candidate perspective image from the second image pair set to obtain multiple candidate perspective image pairs. The connectivity module is used to retain only candidate perspective image pairs whose viewpoint angle does not exceed a threshold, based on connectivity self-pairing and cross-graph pairing relationships, to obtain the third image pair set.

[0129] Optionally, the device further includes: The semantic determination module is used to determine the semantic vector of each candidate perspective image in the complete image pair set after obtaining the complete image pair set. The similarity determination module is used to determine the semantic similarity based on the semantic vector of each candidate perspective image pair in the complete image pair set; The semantic filtering module is used to retain the top Q candidate perspective image pairs with the highest semantic similarity, thus obtaining the final complete set of image pairs after semantic filtering. The local reconstruction module includes: The local reconstruction submodule is used to input the final complete image pair set into the pre-trained 3D reconstruction model to obtain the local 3D reconstruction result of each candidate perspective image in the final complete image pair set.

[0130] Optionally, the global reconstruction module includes at least: The first determining module is used to regard candidate perspective images originating from the same panoramic image in the complete image pair set as a group of homologous perspective images, select the candidate perspective image with the highest local pose confidence in the group of homologous perspective images as a reference image, and determine the global pose of the reference image. The second determining module is used to propagate the global pose of the reference image to the other candidate perspective images in the group based on the global pose and local pose of the reference image, and the local pose of other candidate perspective images in the group, using pose priors to achieve the conversion from local pose to global pose, and obtain the globally aligned pose of the other candidate perspective images in the group; wherein, the pose priors include: the local pose of the reference image obtained when projecting the panoramic image and the local poses of the other candidate perspective images in the group. The third determining module is used to obtain the globally aligned point cloud of the other candidate perspective images in the group based on the scaling factor and globally aligned pose of the other candidate perspective images in the group, as well as the local point cloud of the other candidate perspective images in the local 3D reconstruction result. The fourth determining module is used to obtain the global point cloud of the reference image based on the scaling factor and global pose of the reference image, and the local point cloud of the reference image in the local 3D reconstruction result. The panoramic 3D reconstruction results of the panoramic images corresponding to the group of source perspective images include: the global pose of the reference image, the global point cloud of the reference image, and the globally aligned point cloud and pose of other candidate perspective images in the group.

[0131] Optionally, the global reconstruction module also includes: The weight determination module is used to determine the weight of a candidate perspective image based on the confidence level of the local point cloud of each candidate perspective image within a set of source perspective images. The displacement determination module is used to obtain the displacement in the pose of the panoramic image corresponding to the set of same-source perspective images based on the translation vector and weight of each candidate perspective image in the set of same-source perspective images. The fifth determining module is used to obtain the frontal pose of the panoramic image from which the candidate perspective image originates, based on the rotation matrix prior of each candidate perspective image in a set of source perspective images and the rotation matrix corresponding to the pose after global alignment; the rotation matrix prior includes: the rotation matrix in the local pose of the reference image obtained when projecting the panoramic image and the rotation matrix in the local pose of other candidate perspective images in the set; The sixth determining module is used to obtain the world camera pose of the panoramic image based on the displacement in the pose of the panoramic image corresponding to a set of source perspective images and the frontal pose of the panoramic image from which each candidate perspective image in the set of source perspective images originates.

[0132] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the panoramic three-dimensional reconstruction method as described in any of the above embodiments of the present invention.

[0133] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the program implements the steps of the panoramic three-dimensional reconstruction method described in any of the above embodiments of the present invention.

[0134] As the apparatus embodiment is basically similar to the method embodiment, it is described in a relatively simple manner. For relevant details, please refer to the description of the method embodiment.

[0135] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0136] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure One One or more processes and / or boxes Figure One A device that provides the functions specified in one or more boxes.

[0138] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure One One or more processes and / or boxes Figure One The function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure One One or more processes and / or boxes Figure One Figure OneThe steps of the function specified in one or more boxes.

[0140] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the present invention.

[0141] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0142] The above provides a detailed description of the panoramic three-dimensional reconstruction method, apparatus, device, and medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A panoramic 3D reconstruction method, characterized in that, The method includes: For each panoramic image in multiple panoramic images of the target scene, perform panoramic image in-image mapping and self-pairing to obtain a first image pair set, which includes image pairs with mutually shared viewing areas within each panoramic image; For the multiple panoramic images, panoramic image mapping and cross-image pairing are performed to obtain a second image pair set. The second image pair set includes multiple panoramic image pairs with common image features for each matching key point pair. For any candidate perspective image pair formed by any candidate perspective image from the first image pair set and any candidate perspective image from the second image pair set, connect the self-pairing and cross-image pairing relationships to obtain a third image pair set; the first image pair set, the second image pair set, and the third image pair set constitute a complete image pair set; The complete image set is input into a pre-trained 3D reconstruction model to obtain the local 3D reconstruction result of each candidate perspective image in the complete image set; A pose-prior-driven global alignment is performed on the local 3D reconstruction results to obtain the panoramic 3D reconstruction results of the target scene.

2. The panoramic three-dimensional reconstruction method according to claim 1, characterized in that, For each panoramic image in a set of multiple panoramic images of the target scene, perform intra-panoramic mapping and self-pairing to obtain a first set of image pairs, including: For each panoramic image in the target scene, spherical uniform sampling is performed to generate a set of candidate perspective images with uniform directional coverage corresponding to the panoramic image. For each panoramic image, a set of candidate perspective images is provided. Only candidate perspective image pairs with a viewing angle not exceeding the angle threshold are retained to achieve intra-panoramic mapping and self-pairing, thus obtaining the first set of image pairs.

3. The panoramic three-dimensional reconstruction method according to claim 1, characterized in that, For the multiple panoramic images, perform panoramic image mapping and cross-image pairing to obtain a second set of image pairs, including: For each pair of panoramic images in the multiple panoramic images, which includes a first panoramic image and a second panoramic image, M initial matching key point pairs with common image features are determined between the first panoramic image and the second panoramic image. Based on the panoramic image height of the multiple panoramic images and the ordinate of each matching key point in the M initial matching key point pairs, a first screening is performed with the goal of suppressing polar distortion interference to obtain multiple intermediate matching key point pairs. The first panoramic image and the second panoramic image are respectively divided into multiple grids of target size. Based on the multiple intermediate matching key point pairs, the point with the highest matching confidence is selected in each grid to form the intermediate matching key point set of the first panoramic image and the second panoramic image respectively. Non-maximum suppression is used to retain intermediate matching key point pairs with a spacing not less than the target spacing, thereby obtaining multiple target matching key point pairs. The target spacing is obtained based on the panoramic image width of the multiple panoramic images. Based on each target matching key point pair, candidate perspective images corresponding to the first panoramic image and the second panoramic image are generated respectively to realize the mapping between panoramic images and cross-image pairing, thereby obtaining the second image pair set, which includes panoramic image pairs of each target matching key point pair.

4. The panoramic three-dimensional reconstruction method according to claim 1, characterized in that, For any candidate perspective image pair consisting of any candidate perspective image from the first image pair set and any candidate perspective image from the second image pair set, connect the self-pairing and cross-image pairing relationships to obtain a third image pair set, including: Traverse any candidate perspective image from the first image pair set and traverse any candidate perspective image from the second image pair set to obtain multiple candidate perspective image pairs. With connectivity self-pairing and cross-graph pairing as the objective, only candidate perspective image pairs with a viewpoint angle not exceeding the angle threshold are retained to obtain the third image pair set.

5. The panoramic three-dimensional reconstruction method according to any one of claims 1 to 4, characterized in that, After obtaining the complete set of image pairs, the method further includes: For each candidate perspective image in the complete image pair set, determine the semantic vector of that candidate perspective image; Based on the semantic vector of each candidate perspective image pair in the complete image pair set, determine the semantic similarity; The top Q candidate perspective image pairs with the highest semantic similarity are retained to obtain the final complete set of image pairs after semantic filtering; The complete image set is input into a pre-trained 3D reconstruction model to obtain the local 3D reconstruction results of each candidate perspective image in the complete image set, including: The final complete image pair set is input into the pre-trained 3D reconstruction model to obtain the local 3D reconstruction result of each candidate perspective image in the final complete image pair set.

6. The panoramic three-dimensional reconstruction method according to claim 1, characterized in that, A pose-prior-driven global alignment is performed on the local 3D reconstruction results to obtain a panoramic 3D reconstruction result of the target scene, including at least: Candidate perspective images originating from the same panoramic image in the complete image pair set are regarded as a group of homologous perspective images. The candidate perspective image with the highest local pose confidence in this group of homologous perspective images is selected as the reference image, and the global pose of the reference image is determined. Based on the global and local poses of the reference image, and the local poses of other candidate perspective images within the group, the global pose of the reference image is propagated to the other candidate perspective images within the group using pose priors, thereby realizing the transformation from local poses to global poses and obtaining the globally aligned poses of the other candidate perspective images within the group; wherein, the pose priors include: the local poses of the reference image obtained when projecting the panoramic image and the local poses of the other candidate perspective images within the group. Based on the scaling factor and globally aligned pose of other candidate perspective images within the group, and the local point cloud of other candidate perspective images in the local 3D reconstruction result, the globally aligned point cloud of other candidate perspective images within the group is obtained. The global point cloud of the reference image is obtained based on the scaling factor and global pose of the reference image, and the local point cloud of the reference image in the local 3D reconstruction result. The panoramic 3D reconstruction results of the panoramic images corresponding to the group of source perspective images include: the global pose of the reference image, the global point cloud of the reference image, and the globally aligned point cloud and pose of other candidate perspective images in the group.

7. The panoramic three-dimensional reconstruction method according to claim 6, characterized in that, Perform pose-prior-driven global alignment on the local 3D reconstruction results to obtain the panoramic 3D reconstruction results of the target scene, and further include: The weight of a candidate perspective image is determined based on the confidence level of the local point cloud of each candidate perspective image within a set of originating perspective images. Based on the translation vector and weight of each candidate perspective image within a set of source perspective images, the displacement in the pose of the panoramic image corresponding to the set of source perspective images is obtained. Based on the rotation matrix prior of each candidate perspective image in a set of source perspective images and the rotation matrix corresponding to the pose after global alignment, the frontal pose of the panoramic image from which the candidate perspective image originates is obtained; the rotation matrix prior includes: the rotation matrix in the local pose of the reference image obtained when projecting the panoramic image and the rotation matrix in the local pose of other candidate perspective images in the set. The world camera pose of the panoramic image is obtained by considering the displacement in the pose of the panoramic image corresponding to a set of common perspective images and the frontal pose of the panoramic image from which each candidate perspective image in the set of common perspective images originates.

8. A panoramic three-dimensional reconstruction device, characterized in that, The device includes: The first pairing module is used to perform panoramic image in-panoramic mapping and self-pairing for each panoramic image in multiple panoramic images of the target scene to obtain a first image pair set, which includes image pairs with mutually shared viewing areas in each panoramic image. The second pairing module is used to perform panoramic image mapping and cross-image pairing for the multiple panoramic images to obtain a second image pair set. The second image pair set includes multiple panoramic image pairs with common image features for each matching key point pair. The third pairing module is used to connect self-pairing and cross-image pairing relationships for any candidate perspective image pair composed of any candidate perspective image from the first image pair set and any candidate perspective image from the second image pair set, so as to obtain a third image pair set; the first image pair set, the second image pair set, and the third image pair set constitute a complete image pair set; The local reconstruction module is used to input the complete image pair set into a pre-trained 3D reconstruction model to obtain the local 3D reconstruction result of each candidate perspective image in the complete image pair set. The global reconstruction module is used to perform pose prior-driven global alignment on the local 3D reconstruction results to obtain the panoramic 3D reconstruction results of the target scene.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the panoramic three-dimensional reconstruction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the panoramic three-dimensional reconstruction method as described in any one of claims 1 to 7.

Images