Three-dimensional reconstruction method, electronic device, storage medium and computer program product

Abstract

This application provides a 3D reconstruction method, electronic device, storage medium, and computer program product. The method includes: determining a dense point cloud corresponding to a first scene based on multi-view images corresponding to the first scene; determining a first region contained in each view image of the multi-view images; the first region being the region observed by fewer than a first number of viewpoints; constructing a background model corresponding to the first scene based on the first region and a skybox model; training a 3DGS model based on the dense point cloud and the background model to obtain a trained 3DGS model; and using the trained 3DGS model to perform 3D reconstruction of the first scene. This method can improve the quality of 3D reconstruction.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a three-dimensional reconstruction method, electronic device, storage medium, and computer program product. Background Technology

[0002] When performing large-scale scene reconstruction, image data is first preprocessed to remove noise, correct color and geometric distortion, and then scale-invariant feature transform (SIFT) feature points are extracted for initial camera pose estimation and sparse point cloud generation. The sparse point cloud is then used to train 3D Gaussian sputtering (3DGS) for reconstruction. However, for sports scenes, which include athletes in motion, large areas of weakly textured fields or lawns, large areas of spectator seating and stadium buildings, sparse data acquisition cannot fully capture all the information in the sports scene, resulting in low quality of 3D reconstruction. Summary of the Invention

[0003] This application provides a three-dimensional reconstruction method, electronic device, storage medium, and computer program product that can improve the quality of three-dimensional reconstruction.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a three-dimensional reconstruction method, the method comprising: Based on the multi-view images corresponding to the first scene, determine the dense point cloud corresponding to the first scene; Determine a first region contained in each of the multi-view images; the first region is the region observed by fewer than a first number of viewpoints. Construct a background model corresponding to the first scene based on the first region and the skybox model; The 3D Gaussian sputtering 3DGS model is trained based on the dense point cloud and the background model to obtain the trained 3DGS model. The trained 3DGS model is used to reconstruct the first scene in three dimensions.

[0005] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the three-dimensional reconstruction method provided in the embodiments of this application.

[0006] This application provides a storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the three-dimensional reconstruction method provided in this application.

[0007] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the three-dimensional reconstruction method provided in this application.

[0008] The embodiments of this application have the following beneficial effects: By analyzing multi-view images corresponding to the first scene, a dense point cloud corresponding to the first scene is determined. Compared to sparse point clouds, dense point clouds can express more scene details. That is, even with sparse acquisition of multi-view images corresponding to the first scene, determining the dense point cloud corresponding to the first scene can express more scene details. A first region is determined for each view image in the multi-view images. The first region is the region observed by fewer than a first number of views. A background model corresponding to the first scene is constructed based on the first region and the skybox model. Taking the background corresponding to the first scene into account during 3D reconstruction can improve the quality of background reconstruction. Furthermore, the 3DGS model is trained using the dense point cloud and the background model to perform 3D reconstruction. Training the 3DGS with the dense point cloud allows the obtained 3DGS model to retain more scene details. Training the 3DGS with the background model takes into account background details during the 3D reconstruction process, thereby improving the quality of 3D reconstruction. Attached Figure Description

[0009] Figure 1 A flowchart illustrating a three-dimensional reconstruction method provided in an embodiment of this application; Figure 2 A flowchart illustrating an exemplary sparse reconstruction method for a large sports scene provided in this application embodiment; Figure 3 A schematic diagram illustrating an exemplary effect of dense point cloud enhancement provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0012] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. It should also be noted that the terms "first," "second," etc., used in the embodiments of this application are merely for distinguishing similar objects and do not represent a specific ordering of objects. It is understood that "first," "second," etc., may be interchanged in a specific order or sequence where permissible, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0013] In related technologies, when reconstructing large scenes, image data is first preprocessed to remove noise, correct color, and correct geometric distortion. Then, SIFT feature points are extracted for initial camera pose estimation and sparse point cloud generation. A three-dimensional (3D) Gaussian model is introduced, and parameters are iteratively optimized to fit the scene. Its innovation lies in dividing the large scene into multiple local regions for independent optimization, then using a merging strategy to fuse the local models, adjusting the Gaussian parameters of overlapping adjacent regions to ensure a smooth model transition. This approach primarily focuses on reconstructing general large scenes and is not suitable for sparsely acquired sports scenes. Sports scenes are more complex, containing athletes, large areas of weakly textured fields or lawns, large spectator stands, and stadium buildings, while also considering the issue of sparse acquisition, making this method difficult to handle. Regarding the utilization of multi-view data, although multi-view geometric constraints are used, they are ineffective for sparsely acquired sports scenes. Multi-view data in sports scenes often suffers from unbalanced perspectives and severe occlusion in some areas. Furthermore, the texture processing for background areas lacks specificity and cannot effectively solve the problems of texture loss and color deviation.

[0014] This application provides a three-dimensional reconstruction method. Figure 1 This is a flowchart illustrating a three-dimensional reconstruction method provided in an embodiment of this application; as shown below. Figure 1 As shown, the method includes: S101. Based on the multi-view images corresponding to the first scene, determine the dense point cloud corresponding to the first scene.

[0015] It should be noted that the first scene can be any scene that needs to be reconstructed in 3D. The specific scene can be determined according to the actual situation and is not limited here. As an example, the first scene can be a sports scene, such as a stadium or a football field.

[0016] In this embodiment of the application, before determining the dense point cloud corresponding to the first scene based on the multi-view images corresponding to the first scene, the method further includes: acquiring the multi-view images corresponding to the first scene. Specifically, acquiring the multi-view images corresponding to the first scene includes: deploying a multi-view camera acquisition system in the first scene, selecting a key moment, and acquiring images captured by all cameras at the same moment from the multi-view camera acquisition system, maintaining these images as multi-view images. It should be noted that the shooting angles of the cameras included in the multi-view camera acquisition system need to cover a 360-degree (°) field of view of the first scene. The number of cameras can be determined according to the actual situation and is not limited here. As an example, the number of cameras can be 36.

[0017] It should be noted that the number of viewpoints in the multi-view images corresponding to the first scene is the same as the number of cameras. The specific number can be determined according to the actual situation, and no limit is made here. As an example, when there are 36 cameras, there are 36 viewpoints, and the multi-view images are 3 images from different viewpoints.

[0018] In this embodiment of the application, the process of determining the dense point cloud corresponding to the first scene specifically includes: using a dense feature extraction operator to extract features from the multi-view image to obtain dense feature points corresponding to the first scene; and converting the dense feature points into a dense point cloud.

[0019] It should be noted that the dense feature extraction operator can also be understood as a feature extraction model for sports scenes. The dense feature extraction operator includes a dynamically perceptive convolutional module. The feature extraction network uses the Swin Transformer as its backbone architecture, replacing the original hybrid structure, thus ensuring robust and dense feature extraction even in sports scenes with weak textures. A multi-level feature extraction module is constructed: the shallow layer uses 3×3 convolutional layers to extract basic features; the middle layer captures and fuses local features through local window attention using the Swin Transformer; and the deep layer uses a cross-window attention mechanism to aggregate global features. Its core idea is to allow the convolutional kernel to "perceive" the contextual information of the input features (such as target pose, texture distribution, and motion state) and generate a dedicated convolutional kernel that matches the input in real time, thereby improving the specificity and robustness of feature extraction. During training, a dataset containing 100 real sports events and 1 million multi-view images was used. An adaptive moment estimation optimizer with weight decay (Adam with Weight Decay, AdamW) was employed, with an initial learning rate of 0.0001, decaying by 10% every 10 epochs. The loss function combined contrastive learning loss and L1 loss to enhance feature robustness. A dense feature extraction operator was used to extract features from the multi-view images, obtaining dense feature points corresponding to the first scene. This can be understood as inputting the multi-view images into the dense feature extraction operator for feature extraction, resulting in dense feature points corresponding to the first scene. These dense feature points can be understood as two-dimensional (2D) feature points, and their number can be one or more, the specific number determined based on the actual situation.

[0020] In this embodiment of the application, the process of converting dense feature points into dense point clouds specifically includes: subjecting dense feature points to spatial consistency constraints and feature consistency constraints to obtain dense point clouds.

[0021] It should be noted that obtaining a dense point cloud by applying spatial consistency constraints and feature consistency constraints to dense feature points can be understood as first applying spatial consistency constraints to dense feature points, and then applying feature consistency constraints to obtain the dense point cloud. The process of first applying spatial consistency constraints to dense feature points and then applying feature consistency constraints to obtain a dense point cloud can be illustrated as follows: Using the camera parameters of each camera, the dense feature points (2D feature points) corresponding to each viewpoint are projected into 3D space. Layered bundle adjustment is used to constrain the spatial consistency of the feature points, meaning that the same feature point under multiple viewpoints has unique and stable coordinates in 3D space, avoiding geometric misalignment caused by viewpoint differences. Then, feature consistency is used to check whether the 2D feature attributes of points in the same 3D space are consistent under different viewpoints, requiring that the feature descriptors under multiple viewpoints have similarity, avoiding feature ambiguity caused by viewpoint changes, and finally obtaining the dense point cloud.

[0022] To facilitate understanding, an example is provided here. In the case of a football scene, considering the characteristics of football scenes such as large scale, weak texture in the grass area, and high precision requirements for key structures (such as goals and sidelines), dense feature points are filtered and transformed in three-dimensional space. First, spatial consistency constraints are implemented. 2D feature points are projected into 3D space using camera parameters. A layered bundle adjustment method is then used to constrain the spatial consistency of feature points. This involves classifying features based on the characteristics of typical elements in the feature descriptor. For static key structures such as goalposts, corner flags, and penalty spots, a weight of 3 is assigned in the bundle adjustment. For static background areas such as the grass and stands, outliers are filtered out using adjacent view depth continuity constraints (e.g., depth difference between adjacent point clouds ≤ 5 cm) to avoid cumulative errors at large scene scales. For dynamic targets such as players and the football, kinematic constraints are introduced (e.g., the physical relationship between player step frequency and displacement, and the parabolic motion model of the football). The weights are dynamically adjusted in the bundle adjustment—relaxing the spatial error to 8 cm during high-speed movement (e.g., at the moment of shooting) and tightening it to 5 cm during low-speed movement, ensuring that the 3D coordinates conform to physical laws while also adapting to projection deviations caused by motion blur. Next, feature consistency constraints are implemented. A layered verification mechanism is designed for the specific features of the football scene (e.g., high repetition of grass textures, varied player jersey patterns, and uniform football surface texture). For static areas such as lawns, borders, and billboards, a "multi-scale feature fusion verification" is adopted. In the lawn area, the cosine similarity of the feature descriptors from different perspectives is required to be ≥0.75 to avoid mismatches caused by changes in lighting. At the edge of the border, the contour features extracted by the edge detection operator are combined to force the edge pixel gradient direction to be consistent under multiple perspectives (deviation ≤5°) to ensure that the edge remains straight in three-dimensional space.

[0023] It should be noted that before converting dense feature points into a dense point cloud, this method also includes feature matching of the dense feature points. The specific process includes: in the feature matching stage, the feature points output by the network contain high-dimensional descriptor vectors, and the degree of matching is measured by calculating cosine similarity—particularly enhancing the feature discrimination of typical elements in a football scene such as grass texture, jersey color blocks, and football edges. During initial matching, the cosine similarity of feature point descriptors between images is calculated, and a threshold of 0.7 is used to filter initial matching point pairs.

[0024] The solution in this application utilizes the feature learning and association capabilities of deep learning networks to deeply explore complex relationships between data, accurately estimating feature points when processing occluded and unbalanced viewpoint data. It directly learns and estimates dense feature points from multi-view data, transforming them into a dense point cloud. This skips the initial feature point extraction stage, which is prone to information loss in traditional methods, preserves detailed information about the sports scene, improves the quality and realism of scene reconstruction, and generates a more complete and accurate dense point cloud.

[0025] S102. Determine the first region contained in each viewpoint image in the multi-view image.

[0026] The first region is the region observed by fewer than the first number of viewpoints.

[0027] It should be noted that the first region can be understood as the weakly observed region in each viewpoint image. The specific number of the first number of viewpoints can be determined according to the actual situation. As an example, when the total number of viewpoints is 36, the first number can be 8. The first region is the region observed by fewer than the first number of viewpoints. For example, the first region is the region in each viewpoint image that is observed by fewer than 8 viewpoints.

[0028] In this embodiment of the application, the process of determining the first region contained in each view image in the multi-view image specifically includes: dividing each view image in the multi-view image to obtain one or more image blocks corresponding to each view image; determining one or more observation information corresponding to one or more image blocks; and determining the first region from one or more image blocks based on one or more observation information.

[0029] It should be noted that the method for dividing the multi-view image into its constituent images can be determined based on the actual situation and is not limited here. As an example, it can be divided according to the resolution of each view image. For ease of understanding, an example is provided here, where a certain view image (or each view image) has a resolution of 3840. A 2160 resolution image, which is then divided into 128... The size of 72 is divided into multiple image blocks.

[0030] In this embodiment, observation information can be understood as information about whether each image patch is observed from other viewpoints. Determining one or more observation information corresponding to one or more image patches can be understood as determining the observation information corresponding to each image patch. The process of determining one or more observation information corresponding to one or more image patches specifically includes: projecting the first image patch corresponding to the first viewpoint image into three-dimensional space and then backprojecting it into the second viewpoint image to obtain the observation information corresponding to the first image patch; the first viewpoint image is any viewpoint image in the multi-viewpoint image; the second viewpoint image is all other viewpoint images in the multi-viewpoint image except for the first viewpoint image; the first image patch is any image patch among the one or more image patches corresponding to the first viewpoint image.

[0031] It should be noted that the first-view image can be any view image in the multi-view image, the first image block is any image block among one or more image blocks obtained after dividing the first-view image, and the second-view image is all the other view images in the multi-view image except for the first-view image.

[0032] It should be noted that projecting the first image patch corresponding to the first viewpoint image onto 3D space and then backprojecting it onto the second viewpoint image to obtain the observation information corresponding to the first image patch can be understood as projecting each image patch onto 3D space and then backprojecting it onto the second viewpoint image (or all other viewpoint images except the first viewpoint image) to obtain the observation information of the first image patch. The method for determining the observation information of other image patches in the first viewpoint image besides the first image patch is the same as the method for determining the observation information of the first image patch, and will not be repeated here. The method for determining the observation information of each image patch corresponding to each viewpoint image in the second viewpoint image is the same as the method for determining the observation information of the first image patch in the first viewpoint image, and will not be repeated here.

[0033] In this embodiment of the application, the process of determining a first region from one or more image blocks based on one or more observation information specifically includes: when the observation information corresponding to the first image block indicates that the first image block is observed by less than a first number of viewpoints, the first image block is taken as the first region.

[0034] It should be noted that when the observation information corresponding to the first image patch indicates that the first image patch is observed by fewer than a first number of viewpoints, the first image patch is considered the first region. This can be understood as, when the observation information corresponding to the first image patch indicates that the first image patch is observed by fewer than 8 viewpoints, the first image patch is considered the first region (weak observation region) in the first-view image. After determining the observation information corresponding to the first image patch, it is also necessary to determine the observation information of all other image patches besides the first image patch within one or more image patches corresponding to the first-view image, in order to obtain the complete first region (weak observation region) contained in the first-view image.

[0035] S103. Construct the background model corresponding to the first scene based on the first region and the skybox model.

[0036] It should be noted that the skybox model can be understood as a pre-built model. The construction process of the skybox model specifically includes: extracting three-dimensional structural data from the stadium's Building Information Modeling (BIM), generating the stadium's exterior and spectator seating basic textures using 3D modeling software, collecting historical image data to synthesize background textures such as the sky and stadium exterior, generating an initial texture library according to the skybox hexahedral facet mapping rules, and obtaining the skybox model.

[0037] In this embodiment of the application, the process of constructing a background model corresponding to the first scene based on the first region and the skybox model specifically includes: stitching the first region contained in each view image to obtain a stitched image; performing semantic segmentation on the stitched image to obtain background region mask information; and updating the skybox model using the background region mask information to obtain the background model corresponding to the first scene.

[0038] It should be noted that the method for stitching the first region contained in each viewpoint image can be any image stitching method, and the specific method can be determined according to the actual situation. No limitation is made here; as an example, the stitching method can be an image stitching algorithm. The stitched image can be understood as a panoramic image. Stitching the first region contained in each viewpoint image to obtain the stitched image can be understood as stitching the first region (weakly observed region) contained in each viewpoint image into a panoramic image using an image stitching algorithm. Semantic segmentation is performed on the stitched image to obtain background region mask information. This can be understood as generating a fine-grained background region mask (mask) on the stitched image using a semantic segmentation method (fine-tuning a pre-trained U-Net model). The background region mask information is then used to update the skybox model to obtain the background model corresponding to the first scene. This can be understood as applying the background region mask information to the panoramic image to extract background texture, updating it in the skybox background model, and using a Poisson fusion algorithm to ensure smooth texture transitions to obtain the background model.

[0039] The solution in this application constructs and maps a skybox texture. It generates the skybox texture using panoramic images, historical event images from the same camera position, or BIM models, corrects the distortion of the background image of the edge camera, divides the strong and weak observation areas, maps the corrected texture to the skybox model, and constructs a global prior for the background to avoid problems such as blurred background texture and geometric distortion.

[0040] S104. The 3DGS model is trained based on the dense point cloud and background model to obtain the trained 3DGS model.

[0041] It should be noted that training a 3DGS model based on dense point clouds and a background model to obtain a trained 3DGS model can be understood as initializing the training of the 3DGS model using dense point clouds, while simultaneously performing background supervision based on the background model during the training process to obtain the trained 3DGS model.

[0042] To facilitate understanding, an example is provided here. In 3DGS model training, the model is initialized with dense point clouds. Based on the weakly observed region background mask and the skybox background model, the following operations are performed: using the mapping relationship between the image and 3D space coordinates, the association between the background region mask pixels and the Gaussian distribution of the 3DGS model background is established. The texture of the skybox background model is accurately mapped to the back of the 3DGS model background region. The color difference loss between the background region and the skybox texture is calculated. The stochastic gradient descent algorithm is used to adjust the Gaussian distribution parameters of the background region in the 3DGS model through backpropagation, so that the texture quality of the background region is consistent with the skybox background model, thus obtaining the trained 3DGS model.

[0043] It should be noted that when calculating the color difference loss between the background region and the skybox texture, only the masked portion of the background region is considered. After converting both to the CIELAB color space (CIE Lab), the mean squared error (MSE) is used to calculate the color difference, and the structural similarity index (SSIM) is introduced to calculate the texture structure difference as a supplementary loss term. The texture quality loss function is incorporated into the overall loss function during 3DGS model training. Specifically, this can be achieved through... Let represent the first two terms, which are the photometric loss function and structural similarity error inherent in 3DGS, respectively, and the third term is the background supervision enhancement error introduced in the embodiments of this application. Set to 0.2, Set it to 0.1.

[0044] S105. Use the trained 3DGS model to perform 3D reconstruction of the first scene.

[0045] It should be noted that using the trained 3DGS model to reconstruct the first scene in 3D can be understood as using the trained 3DGS model to perform free-viewpoint trajectory roaming and rendering of the first scene, then performing Video Super-Resolution (VSR) enhancement on the rendered video, finally obtaining the 3D reconstructed first scene. Specifically, the VSR enhancement method involves using existing multi-view video of a stadium to fine-tune and train the VSR model, making it suitable for large-scale sports scenes; then using this model to process the rendered video, resulting in an enhanced video with doubled resolution and reduced rendering artifacts.

[0046] The solution in this application embodiment determines a dense point cloud corresponding to a first scene based on multi-view images of the first scene. Dense point clouds can express more scene details than sparse point clouds. That is, when sparsely acquiring multi-view images of the first scene, determining the dense point cloud corresponding to the first scene can express more scene details. A first region is determined in each view image of the multi-view images. The first region is the region observed by less than a first number of views. A background model corresponding to the first scene is constructed based on the first region and the skybox model. Taking the background corresponding to the first scene into account during 3D reconstruction can improve the quality of background reconstruction. Furthermore, the 3DGS model is trained using the dense point cloud and the background model to perform 3D reconstruction. Training the 3DGS with the dense point cloud can make the obtained 3DGS model retain more scene details. Training the 3DGS with the background model can take into account the background details in the 3D reconstruction process, thereby improving the quality of 3D reconstruction.

[0047] To facilitate understanding, an example is provided here. In practical applications, the above 3D reconstruction method can be understood as a sparse reconstruction method for large-scale sports scenes. This method constructs a multi-view dense point cloud estimation model suitable for sports scenes using feature extraction operators, combines bundle adjustment to constrain feature points, and transforms the estimated dense feature points into a dense point cloud. Simultaneously, it determines strong and weak observation regions by identifying these regions through multi-view images, establishes a mask for these regions, and supervises the training of a 3D Gaussian sputtering model based on a pre-built skybox model. Finally, it trains a sports scene video super-resolution enhancement model based on massive amounts of sports scene data and applies it to the rendered video. This achieves high-quality reconstruction based on sparsely acquired large-scale sports scene data, avoids loss of detail and texture, improves background reconstruction quality, and results in a visually appealing free-roaming rendered video.

[0048] Construction of a Multi-View Dense Point Cloud Estimation Model for Stadiums: Compared with related technologies, this application introduces a multi-view dense point cloud estimation model, adding a step from sparse point cloud to dense point cloud. This enables high-quality reconstruction of large-scale sports scenes while preserving more texture details, even with sparse data acquisition. The multi-view dense point cloud estimation model includes a feature extraction operator specifically designed for sports scene design. This operator includes a dynamic-aware convolution module, which can keenly capture the fast and complex non-rigid movements of athletes. For large-area weak-texture regions commonly found in sports scenes, such as football fields and synthetic running tracks, a special convolution kernel design and multi-layer feature fusion are used to efficiently extract subtle features from these regions. Then, combined with a multi-view consistency constraint module, the estimated dense 2D feature points are transformed into a high-confidence 3D dense point cloud. Compared with traditional sparse point clouds, this dense point cloud can express more scene detail information. Initializing 3DGS training with this dense point cloud enables the model to converge faster, resulting in a model that retains more texture details and has higher model quality.

[0049] Background enhancement scheme based on skybox technology: Related technologies do not have specific processing for background region reconstruction. This application introduces a background enhancement scheme based on skybox technology, which effectively solves the problems of poor background reconstruction quality and defects and holes caused by sparse camera acquisition. By constructing a background skybox model, it is introduced into the background supervision of 3DGS training. The Gaussian parameters of the background region are optimized by the multi-view consistency loss function, thereby improving the reconstruction quality of the background region and filling the background holes.

[0050] Application of a dedicated video super-resolution enhancement model for sports scenes: Related technologies inevitably have flaws in the reconstruction of large scenes with sparse acquisition. Noise and flaws will appear in videos rendered by 3D Gaussian models. This application introduces a video super-resolution enhancement model for sports scenes to address this problem. The video super-resolution enhancement model for sports scenes, which is specially trained using massive sports scene video data, can enhance videos rendered by 3DGS, reduce flaws in the rendered video, and improve the quality of the rendered video.

[0051] This application embodiment describes the high-quality 3DGS reconstruction and rendering of specific exciting moments captured by a multi-camera acquisition system deployed in a large-scale sports scene. Figure 2 A flowchart illustrating an exemplary sparse reconstruction method for large-scale sports scenes provided in this application embodiment; as shown... Figure 2 As shown, the specific steps are as follows: 1. Multi-camera acquisition system.

[0052] 2. Multi-view image sequence of exciting moments.

[0053] It should be noted that by selecting the highlights from the multi-camera acquisition system, a multi-view image sequence of the highlights is obtained.

[0054] 3. Camera parameter estimation.

[0055] Camera parameter estimation is performed by traditional feature extraction and matching on multi-view image sequences of exciting moments.

[0056] 4. Camera parameter model.

[0057] 5. Sparse dot clouds.

[0058] It should be noted that the 3DGS in step 18 is initialized according to steps 1-5, that is, the sparse point cloud is initialized using the conventional method.

[0059] 6. Sports Scene Feature Extraction Model.

[0060] It should be noted that the sequence of multi-view images of exciting moments is input into the sports scene feature extraction model through the sports scene feature extraction operator.

[0061] 7. Dense feature points.

[0062] 8. Spatial consistency constraints.

[0063] It should be noted that spatial consistency constraints are applied through a camera parameter model.

[0064] 9. Feature consistency constraint.

[0065] 10. Dense point clouds.

[0066] 11. Strong and weak observation area division model.

[0067] 12. Strong observation area mask (the area in the public view captured by the camera, usually the grass area of ​​the sports field).

[0068] 13. Weak observation area mask (the area around the camera, usually the audience area and other venue areas).

[0069] 14. Map the background texture to the skybox.

[0070] 15. Panoramic images and prior acquired background images.

[0071] 16. Poisson blending eliminates seams in skybox textures.

[0072] 17. Background skybox model.

[0073] 18. 3DGS initialization.

[0074] It should be noted that 3DGS is initialized based on the dense point cloud from step 10.

[0075] 19. Background monitoring to optimize background voids and imperfections.

[0076] 20. Large Gaussian kernels are split into small Gaussian kernels, and pixel alignment is optimized for details.

[0077] 21. 3DGS model.

[0078] 22. Immersive experience with free-viewpoint roaming videos.

[0079] It should be noted that interactive rendering of 3DGS model parts and preset trajectory rendering are used to create an immersive experience of free-viewpoint roaming videos.

[0080] 23. Super-resolution model for sports scene videos to eliminate flaws and enhance image quality.

[0081] 24. Enhanced video with immersive free-viewpoint roaming experience.

[0082] The above plan will be explained in detail below.

[0083] Step 1: Deploy a multi-view camera acquisition system in the stadium and select exciting moments to obtain multi-view image sequences of those moments as the raw data for 3DGS reconstruction.

[0084] Typically, 36 cameras are deployed in a ring around the perimeter of a football field, covering a 360° field of view. After selecting a highlight moment, images captured by all cameras at the same time are obtained from the 36 camera acquisition streams and saved as an image sequence, resulting in a total of 36 images.

[0085] Step 2: Based on the multi-view image sequence of the highlights, perform feature extraction and feature matching to obtain the camera parameters for each camera and a sparse feature point cloud. This process is typically accomplished using the Colmap algorithm, which extracts SIFT corner points from the image as features.

[0086] Step 3: Construction of a dense feature extraction operator and extraction of dense feature points from multi-view sports scene images based on exciting moments. The previous step used SIFT corner features for feature matching to obtain a sparse point cloud. Conventional methods directly use this sparse point cloud for subsequent 3DGS training initialization. However, due to the large scale, weak texture, and sparse number of viewpoints in a football field, and the lack of consistent supervision across multiple viewpoints during training, the 3DGS model trained based on the sparse point cloud has poor detail representation capabilities. To address the texture characteristics of sports scenes, a robust and dense feature extraction operator suitable for sports scenes was constructed. This operator includes a dynamically perceptive convolution module, aiming to solve the problem of insufficient adaptability of SIFT to sports scenes (including large areas of weakly textured grass, repetitive textures (such as spectator seats), athletes lacking clearly distinguishable textures, and multi-view texture differences). Its core idea is to allow the convolution kernel to "perceive" the contextual information of the input features (such as target pose, texture distribution, and motion state) and generate a dedicated convolution kernel that matches the input in real time, thereby improving the specificity and robustness of feature extraction.

[0087] Step 4: Design a multi-view consistency constraint module to filter the dense feature points estimated in the previous step and transform them into a dense point cloud in 3D space. Specifically, this involves projecting 2D feature points into 3D space using camera parameters and using bundle adjustment to constrain the spatial consistency of the feature points. This ensures that the same feature point in multiple views has unique and stable coordinates in 3D space, avoiding geometric misalignment caused by viewpoint differences. Then, feature consistency is used to check whether the 2D feature attributes of points in the same 3D space are consistent under different viewpoints. This requires that the feature descriptors under multiple viewpoints have similarity, avoiding feature ambiguity caused by viewpoint changes.

[0088] Step 5: Calculate the strong and weak observation regions of the multi-view image sequence of the highlights. Specifically, this involves dividing each 3840-pixel image into its corresponding region. 2160 resolution image according to 128 The size of 72 is divided into multiple image blocks. The image blocks are projected into three-dimensional space and then back-projected to other viewpoints. It is determined whether the image block is within the coordinate range of other viewpoints. If it is, it is recorded that the image block has been observed once by other viewpoints. If an image block is observed by no less than 8 viewpoints, it is recorded as a strong observation area. Otherwise, it is recorded as a weak observation area. Thus, the strong and weak observation area mask of each image is obtained.

[0089] Step 6: Based on the strong and weak observation area mask generated in the previous step, and combined with prior venue information, generate a background skybox model. Specifically, the prior information texture is extracted from the stadium's BIM using 3D structural data. 3D modeling software is used to generate the stadium exterior and seating basic textures. Historical image data is collected to synthesize background textures for the sky, stadium exterior, etc. An initial texture library is generated according to the skybox hexahedral facet mapping rules. Then, weak observation areas from multi-view images are stitched together into a panoramic image using an image stitching algorithm. A refined background region mask is then generated using a semantic segmentation method (pre-trained U-Net model fine-tuning). This mask is applied to the panoramic image to extract background textures, which are then updated in the skybox background model. A Poisson fusion algorithm is used to ensure smooth texture transitions.

[0090] Step 7: Initialize the 3DGS training using the dense point cloud obtained in Step 4. Simultaneously, add background supervision for the weakly observed regions generated in the previous step during training. This enhances the model's expressive power while improving the reconstruction quality of the background region, addressing the issue of voids in the background reconstruction. Specifically, based on the detailed background mask of the weakly observed regions, add a background enhancement loss function with the mask as weight, and perform Gaussian kernel size suppression based on the number of Gaussian-covered pixels in each view as weight, thereby improving the reconstruction detail and the quality of the background region reconstruction.

[0091] Step 8: Based on the 3DGS model trained in the previous step, perform free viewpoint trajectory roaming and rendering, and then enhance the rendered video using VSR. Specifically, use existing multi-view videos of stadiums to fine-tune and train the VSR, making the model suitable for large-scale sports scenes. Use this model to process the rendered video, resulting in an enhanced video with doubled resolution and reduced rendering artifacts.

[0092] The advantages of this application's embodiments mainly include four key aspects: dense point cloud model generation, skybox reconstruction, background supervision during 3DGS training, and video enhancement (VSR model) for rendered videos. These aspects are closely interconnected. The dense point cloud model provides a high-quality point cloud foundation for subsequent 3DGS training; skybox reconstruction establishes a global prior for the background; 3DGS training optimizes the model under background supervision; and the VSR model enhances the 3DGS rendered video, improving video quality. The overall process achieves efficient and high-quality sparse reconstruction of large-scale sports scenes.

[0093] Implementation of Dense Point Cloud Estimation Model. The multi-view dense point cloud estimation model in this application's embodiments draws on the core idea of ​​Robust Dense Feature Matching, and incorporates targeted designs in key stages such as feature extraction, matching strategies, and point cloud generation. The specific implementation method is as follows: 1. Feature extraction and enhancement.

[0094] To address the texture characteristics of sports scenes, a robust and dense feature extraction operator suitable for sports scenarios was constructed. This operator includes a dynamically perceptive convolutional module, and the feature extraction network uses the Swing Transformer as its backbone architecture, replacing the original hybrid structure, thus ensuring robust and dense feature extraction even in sports scenes with weak textures. A multi-level feature extraction module is constructed: the shallow layer uses 3×3 convolutional layers to extract basic features; the middle layer captures and fuses local features through Swing Transformer local window attention; and the deep layer uses a cross-window attention mechanism to aggregate global features. The core idea is to allow the convolutional kernel to "perceive" the contextual information of the input features (such as target pose, texture distribution, and motion state) and generate a dedicated convolutional kernel that matches the input in real time, thereby improving the specificity and robustness of feature extraction. During training, a dataset containing 100 real sports events and 1 million multi-view images was used, employing the AdamW optimizer with an initial learning rate of 0.0001, decaying by 10% every 10 epochs. The loss function combines contrastive learning loss and L1 loss to enhance feature robustness.

[0095] 2. Robust and dense feature matching.

[0096] To address the unique characteristics of football scenes, a dedicated matching scheme based on learnable descriptors is employed. During the feature matching phase, the network outputs feature points containing high-dimensional descriptor vectors. The degree of matching is measured by calculating cosine similarity—particularly enhancing the feature discrimination of typical elements in football scenes such as grass texture, jersey color blocks, and the edge of the football. In the initial matching phase, the cosine similarity of feature point descriptors between images is calculated, and a threshold of 0.7 is used to filter initial matching point pairs.

[0097] 3. Multi-perspective consistency constraint model.

[0098] To address the characteristics of football scenes—large scale, weak texture in grass areas, and high precision requirements for key structures (such as goals and sidelines)—targeted filtering and 3D spatial transformation of dense feature points are implemented. First, spatial consistency constraints are applied. 2D feature points are projected into 3D space using camera parameters. Layered bundle adjustment is then used to constrain the spatial consistency of feature points. This involves classifying features based on the characteristics of typical elements in the feature descriptor. Static key structures such as goalposts, corner flags, and penalty spots are assigned a 3x weight in bundle adjustment. Static background areas such as grass and stands are filtered for outliers by adjacency depth continuity constraints (e.g., depth difference between adjacent point clouds ≤ 5cm) to avoid cumulative errors at large scene scales. For dynamic targets such as players and the football, kinematic constraints are introduced (e.g., the physical relationship between player step frequency and displacement, and the parabolic motion model of the football). Weights are dynamically adjusted in bundle adjustment—the spatial error is widened to 8cm during high-speed motion (e.g., at the moment of shooting) and tightened to 5cm during low-speed motion, ensuring that the 3D coordinates conform to physical laws while adapting to projection deviations caused by motion blur. Then, feature consistency constraints are applied, and a layered verification mechanism is designed for the specific features of the football scene (such as high repetition of grass texture, varied player jersey patterns, and uniform surface texture of the football). For static areas such as grass, sidelines, and advertising boards, a "multi-scale feature fusion verification" is adopted. In the grass area, the cosine similarity of feature descriptors from different perspectives is required to be ≥0.75 to avoid mismatches caused by changes in lighting. At the edge of the sideline, the contour features extracted by the edge detection operator are combined to force the edge pixel gradient direction to be consistent across multiple perspectives (deviation ≤5°) to ensure that the edgeline remains straight in three-dimensional space.

[0099] Through the above optimizations, a dense, high-quality point cloud model can be obtained, which can significantly reduce point cloud drift and sparsity caused by large-scale errors and weak textures in football scenes, especially improving the consistency of three-dimensional coordinates of core elements such as goals and grass by more than 30%.

[0100] Figure 3 This application provides an exemplary schematic diagram illustrating the effect of dense point cloud enhancement; as shown in the embodiments. Figure 3 As shown, Figure 3 (a) is the original 3D reconstruction result; Figure 3 (b) is a rendering of the enhanced 3D reconstruction using dense point clouds.

[0101] The specific implementation method of the skybox background enhancement scheme is as follows.

[0102] 1. Skybox texture generation and region division.

[0103] Skybox textures are constructed by fusing prior information with real-time acquired data. Prior information textures are extracted from the stadium's BIM (Building Information Modeling) of 3D structural data. 3D modeling software is used to generate basic textures for the stadium exterior and seating area. Historical image data is collected to synthesize background textures for the sky, stadium exterior, etc., and an initial texture library is generated according to the skybox hexahedral facet mapping rules. Then, weakly observed areas from multi-view images are stitched together using an image stitching algorithm to create panoramic images. A refined background region mask is then generated using a semantic segmentation method (fine-tuned from a pre-trained U-Net model). This mask is applied to the panoramic images to extract background textures, which are then updated in the skybox background model. A Poisson fusion algorithm is used to ensure smooth texture transitions. Specifically, the strong and weak observation regions are divided into three parts: each 3840... 2160 resolution image according to 128 The size of 72 is divided into multiple image blocks. The image blocks are projected into three-dimensional space and then back-projected to other viewpoints. It is determined whether the image block is within the coordinate range of other viewpoints. If it is, it is recorded that the image block has been observed once by other viewpoints. If an image block is observed by no less than 8 viewpoints, it is recorded as a strong observation area. Otherwise, it is recorded as a weak observation area. Thus, the strong and weak observation area mask of each image is obtained.

[0104] 2. Collaborative optimization of skybox and 3DGS model.

[0105] In 3DGS model training, the model is initialized with dense point clouds. Based on the weak observation region background mask and the skybox background model, the following operations are performed: using the mapping relationship between the image and 3D spatial coordinates, the association between the background region mask pixels and the Gaussian distribution of the 3D Gaussian model background is established, and the texture of the skybox background model is accurately mapped to the surface of the 3D Gaussian model background region; When calculating the loss due to the color difference between the background region and the skybox texture, only the masked portion of the background region is considered. Both are converted to the CIE Lab color space, and the color difference is calculated using MSE. SSIM is introduced to calculate the texture structure difference as a supplementary loss term. The texture quality loss function is incorporated into the overall loss function during 3DGS model training. Specifically, this can be achieved through... Let represent the first two terms, which are the photometric loss function and structural similarity error inherent in 3DGS, respectively, and the third term is the background supervision enhancement error introduced in the embodiments of this application. Set to 0.2, The learning rate was set to 0.1. A stochastic gradient descent algorithm was used with a learning rate of 0.0001. Backpropagation was used to adjust the Gaussian distribution parameters of the background region in the 3D Gaussian model, ensuring the texture quality of the background region matched that of the skybox background model. Training was terminated and the 3DGS model was saved when the training loss function was less than 0.015 or when 30,000 iterations were reached. This model is the 3DGS model trained on the image sequence of this highlight moment.

[0106] Specific implementation of 3DGS rendering video enhancement based on VSR model.

[0107] To further improve the quality of 3D Gaussian model (3DGS) rendered videos and reduce the GPU memory consumption for training large-scale sports scenes, a video super-resolution (VSR) model was fine-tuned using existing multi-view videos of stadiums. This made the model suitable for large-scale sports scenes, and also doubled the resolution of the rendered videos and optimized rendering artifacts. The specific algorithm implementation details are as follows: 1. Data preprocessing.

[0108] The process involves cleaning multi-view videos of real sports events, removing blurry, jittery, or occluded segments. The videos are then downsampled by half as input for model training, with the unsampled data corresponding to the ground truth of the respective video segment as the output.

[0109] 2. Model selection and initialization.

[0110] A pre-trained Real-world Video Super-Resolution (RealVSR) model was selected as the base model. This model is based on a Transformer architecture and includes a spatiotemporal feature extraction module, a multi-scale feature fusion module, and a reconstruction module. Official pre-trained weights trained on a large-scale general video dataset were directly loaded to initialize the model parameters, retaining its basic feature extraction capabilities learned in general scenarios.

[0111] 3. Fine-tune the training process.

[0112] A comprehensive loss function is defined, including reconstruction loss (using MSE to measure pixel differences), perceptual loss (calculating the mean squared error by extracting feature maps through a Visual Geometry Group 19-layer (VGG19) network), and adversarial loss (training a Generative Adversarial Network (GAN) to make the generated super-resolution video frames more realistic). The preprocessed video frames are input into a fine-tuned VSR model, trained using the Adam optimizer with an initial learning rate of 0.0001, decaying by 10% every 10 epochs. Data is randomly shuffled in batches (36 epochs of 512) each epoch. The generator and discriminator parameters are updated simultaneously using 512 video frames as input for training, while minimizing the overall loss function.

[0113] 4. Resolution enhancement and defect optimization.

[0114] The finely tuned VSR model upscales low-resolution video frames rendered by 3DGS, such as increasing the resolution from 1080p to 2160p. It optimizes rendering artifacts through perceptual loss and adversarial loss; perceptual loss reduces blur and jagged edges, while adversarial loss makes video frames more realistic. In a football match test, the 3DGS-rendered video resolution increased by 50%, with significantly improved clarity and realism.

[0115] The solution presented in this application enables high-quality reconstruction of large-scale sports scenes with sparse data acquisition. Addressing the challenges of complex, weakly textured, and sparsely acquired sports scenes, this application constructs a multi-view dense point cloud estimation model. This model, combined with deep learning algorithms, processes multi-view data to achieve accurate reconstruction of sports scenes while preserving more details. The 3D Gaussian model trained on this model better represents the scene, resulting in a high-quality scene reconstruction model that facilitates the synthesis of high-quality free-view roaming videos.

[0116] The solution proposed in this application can efficiently process the unique multi-view data of sports scenes and avoid information loss during point cloud generation. To address the problems of unbalanced perspectives and severe occlusion in multi-view data of sports scenes, the multi-view dense point cloud estimation model of this application utilizes the feature learning and association capabilities of deep learning networks to deeply explore the complex relationships between data, accurately estimating feature points when processing occluded and unbalanced perspective data. It directly learns and estimates dense feature points from multi-view data, transforming it into a dense point cloud. This skips the initial feature point extraction stage, which is prone to information loss in traditional methods, preserves detailed information of the sports scene, improves the quality and realism of scene reconstruction, and generates a more complete and accurate dense point cloud.

[0117] The solution proposed in this application improves the background reconstruction quality of large-scale sports scenes. Addressing the challenge of insufficient background reconstruction due to sparse camera capture in large-scale sports scenes, this application proposes a background enhancement scheme based on Skybox technology. By constructing and mapping Skybox textures, using panoramic images, historical event images from the same camera position, or BIM models to generate Skybox textures, distortion of the background image from edge cameras is corrected, and strong and weak observation areas are defined. The corrected texture is then mapped to the Skybox model, constructing a global background prior. In the training of the 3D Gaussian sputtering model, the Skybox texture is introduced as a background prior. By constructing a multi-view consistency loss function, the Gaussian parameters of the background region are supervised and constrained, and the objective function is optimized to ensure that the Gaussian distribution parameters of the background region are consistent with the Skybox texture, avoiding problems such as background texture blurring and geometric distortion.

[0118] The sports scene video enhancement model provided in this application is designed for the characteristics of sports scenes. It is a dedicated video super-resolution enhancement model trained on massive amounts of data and applied to the rendering video of the 3DGS model. This model can effectively improve the output resolution and remove defects such as noise and blur, thereby optimizing the visual quality of the reconstructed scene.

[0119] This application provides a three-dimensional reconstruction device. Figure 4 This is a schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of this application; as shown below. Figure 4 As shown, the three-dimensional reconstruction device 400 includes: The first determining unit 401 is used to determine the dense point cloud corresponding to the first scene based on the multi-view image corresponding to the first scene. The second determining unit 402 is used to determine a first region contained in each viewpoint image of the multi-view image; the first region is a region observed by less than a first number of viewpoints. Construction unit 403 is used to construct a background model corresponding to the first scene based on the first region and the skybox model; Training unit 404 is used to train the 3DGS model based on the dense point cloud and the background model to obtain the trained 3DGS model. The 3D reconstruction unit 405 is used to perform 3D reconstruction of the first scene using the trained 3DGS model.

[0120] In some embodiments, the second determining unit 402 is further configured to divide each view image in the multi-view image to obtain one or more image blocks corresponding to each view image; determine one or more observation information corresponding to the one or more image blocks; and determine the first region from the one or more image blocks based on the one or more observation information.

[0121] In some embodiments, the second determining unit 402 is further configured to project the first image block corresponding to the first view image onto a three-dimensional space and then back-project it onto the second view image to obtain the observation information corresponding to the first image block; the first view image is any view image in the multi-view image; the second view image is all other view images in the multi-view image except for the first view image; the first image block is any image block among one or more image blocks corresponding to the first view image.

[0122] In some embodiments, the second determining unit 402 is further configured to, when the observation information corresponding to the first image block indicates that the first image block is observed by less than a first number of viewpoints, identify the first image block as the first region.

[0123] In some embodiments, the first determining unit 401 is further configured to perform feature extraction on the multi-view image using a dense feature extraction operator to obtain dense feature points corresponding to the first scene; and convert the dense feature points into the dense point cloud.

[0124] In some embodiments, the first determining unit 401 is further configured to perform spatial consistency constraints and feature consistency constraints on the dense feature points to obtain the dense point cloud.

[0125] In some embodiments, the construction unit 403 is further configured to stitch together the first region contained in each viewpoint image to obtain a stitched image; perform semantic segmentation on the stitched image to obtain background region mask information; and update the skybox model using the background region mask information to obtain a background model corresponding to the first scene.

[0126] This application also provides an electronic device. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; as shown below. Figure 5 As shown, the electronic device 500 includes a processor 501 and a memory 503. Optionally, the electronic device 500 may also include a communication bus 502.

[0127] In specific embodiments, the processor 501 described above can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the above processor function can also be other types, and this embodiment does not specifically limit it.

[0128] In this embodiment, the communication bus 502 is used to establish a connection between the processor 501 and the memory 503; when the processor 501 executes the running program stored in the memory 503, it implements the following three-dimensional reconstruction method: Based on the multi-view images corresponding to the first scene, a dense point cloud corresponding to the first scene is determined; a first region is determined in each view image of the multi-view images; the first region is the region observed by less than a first number of views; a background model corresponding to the first scene is constructed based on the first region and the skybox model; a 3DGS model is trained based on the dense point cloud and the background model to obtain a trained 3DGS model; and the first scene is reconstructed in three dimensions using the trained 3DGS model.

[0129] Furthermore, the processor 501 is also configured to divide each view image in the multi-view image to obtain one or more image blocks corresponding to each view image; determine one or more observation information corresponding to the one or more image blocks; and determine the first region from the one or more image blocks based on the one or more observation information.

[0130] Furthermore, the processor 501 is also configured to project the first image block corresponding to the first view image onto a three-dimensional space and then back-project it onto the second view image to obtain the observation information corresponding to the first image block; the first view image is any view image in the multi-view image; the second view image is all other view images in the multi-view image except for the first view image; the first image block is any image block among one or more image blocks corresponding to the first view image.

[0131] Furthermore, the processor 501 is also configured to, when the observation information corresponding to the first image block characterizes the first image block as the first region when the first image block is observed by less than a first number of viewpoints, use the first image block as the first region.

[0132] Furthermore, the processor 501 is also used to extract features from the multi-view image using a dense feature extraction operator to obtain dense feature points corresponding to the first scene; and to convert the dense feature points into the dense point cloud.

[0133] Furthermore, the processor 501 is also used to perform spatial consistency constraints and feature consistency constraints on the dense feature points to obtain the dense point cloud.

[0134] Furthermore, the processor 501 is also configured to stitch together the first region contained in each viewpoint image to obtain a stitched image; perform semantic segmentation on the stitched image to obtain background region mask information; and update the skybox model using the background region mask information to obtain a background model corresponding to the first scene.

[0135] This application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors. The computer program implements the three-dimensional reconstruction method described above.

[0136] Based on the above embodiments, this application provides a computer program product, including a computer program that can be executed by one or more processors, and the computer program implements the three-dimensional reconstruction method as described above.

[0137] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0139] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A three-dimensional reconstruction method, characterized in that, The method includes: Based on the multi-view images corresponding to the first scene, determine the dense point cloud corresponding to the first scene; Determine a first region contained in each of the multi-view images; the first region is the region observed by fewer than a first number of viewpoints. Construct a background model corresponding to the first scene based on the first region and the skybox model; The 3D Gaussian sputtering 3DGS model is trained based on the dense point cloud and the background model to obtain the trained 3DGS model. The trained 3DGS model is used to reconstruct the first scene in three dimensions.

2. The method according to claim 1, characterized in that, Determining the first region contained in each viewpoint image of the multi-view images includes: Each view image in the multi-view image is divided to obtain one or more image blocks corresponding to each view image; Determine one or more observation information corresponding to the one or more image patches; The first region is determined from the one or more image patches based on the one or more observation information.

3. The method according to claim 2, characterized in that, Determining one or more observation information corresponding to the one or more image patches includes: The first image block corresponding to the first viewpoint image is projected into three-dimensional space and then back-projected into the second viewpoint image to obtain the observation information corresponding to the first image block; the first viewpoint image is any viewpoint image in the multi-viewpoint image; the second viewpoint image is all other viewpoint images in the multi-viewpoint image except for the first viewpoint image; the first image block is any image block among one or more image blocks corresponding to the first viewpoint image.

4. The method according to claim 2, characterized in that, Determining the first region from the one or more image patches based on the one or more observation information includes: If the observation information corresponding to the first image patch indicates that the first image patch is observed by less than a first number of viewpoints, then the first image patch is taken as the first region.

5. The method according to claim 1, characterized in that, The step of determining the dense point cloud corresponding to the first scene based on the multi-view images corresponding to the first scene includes: The dense feature extraction operator is used to extract features from the multi-view image to obtain dense feature points corresponding to the first scene; The dense feature points are converted into the dense point cloud.

6. The method according to claim 5, characterized in that, The step of converting the dense feature points into the dense point cloud includes: The dense point cloud is obtained by applying spatial consistency constraints and feature consistency constraints to the dense feature points.

7. The method according to claim 1, characterized in that, The step of constructing the background model corresponding to the first scene based on the first region and the skybox model includes: The first region contained in each viewpoint image is stitched together to obtain the stitched image; Semantic segmentation is performed on the stitched image to obtain background region mask information; The skybox model is updated using the background region mask information to obtain the background model corresponding to the first scene.

8. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 7.

9. A storage medium having a computer program or computer-executable instructions stored thereon, characterized in that, When the computer program or the computer-executable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program or computer-executable instructions, characterized in that, When the computer program or the computer-executable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images