3D GS reconstruction method and system based on scene environment selection

Abstract

The invention provides a 3D GS reconstruction method and system based on scene environment selection, and belongs to the technical field of 3D reconstruction. Comprising the steps of obtaining a multi-angle shot video of a scene environment to be reconstructed, segmenting the obtained multi-angle shot video into a plurality of pictures and storing the pictures in the same folder; traversing the folder, sequentially carrying out scene judgment under various shooting scenes on each picture in the traversing process, and determining the scene to which the picture belongs; if the current picture belongs to more than one of multiple shooting scenes at the same time, determining the scene according to the scene priority; and carrying out statistics on scene proportions of all the pictures in the folder, and selecting the 3DGS variant matched with the scene with the highest proportion to carry out three-dimensional reconstruction. According to the invention, high-quality three-dimensional reconstruction of images shot at any angle and in any environment can be realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of three-dimensional reconstruction, and particularly relates to a 3DGS reconstruction method and system based on scene environment selection. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] With the progress of science and technology, the development of the visual field, especially three-dimensional technology, has been increasingly concerned. The reconstruction method based on neural representation has rapidly become a research hotspot. Three-dimensional reconstruction is an important problem in computer vision and graphics, aiming to restore the three-dimensional geometric information and appearance information of the real world through multi-view information. As a new three-dimensional representation method, 3DGS (Gaussian Splatting) has outstanding performance in static scenes in terms of efficient rendering speed and excellent reconstruction quality. 3DGS uses a three-dimensional Gaussian distribution to model the scene, and realizes high-quality modeling of the scene by optimizing the shape, position, color and transparency of the Gaussian sphere.

[0004] 3DGS introduces point cloud representation and differentiable rendering, making the reconstruction process more real-time, suitable for more extensive scene modeling, and faster rendering. However, in actual scene modeling, three-dimensional modeling based on 3DGS has difficult-to-solve technical problems, such as: (1) Poor generalization ability. Existing 3DGS methods rely on static, uniform illumination standardized scenes (such as laboratory environments), and when faced with complex scenes such as low light, strong exposure, dynamic blur, high dynamic range (HDR), or human / moving objects in actual applications, the reconstruction quality decreases dramatically, mainly manifested in severe detail loss in low light scenes, artifacts in strong exposure areas, ghosting or structural misplacement in dynamic object reconstruction, and inaccurate restoration of brightness levels in HDR scenes.

[0005] (2) High dependence on human intervention. Existing solutions require manual selection of reconstruction algorithms that adapt to the scene, resulting in a process that requires users to analyze scene characteristics in advance and manually match algorithms, which is time-consuming and has high professional threshold. In addition, human judgment is limited by experience and is at risk of misjudgment, such as difficulty in quantitatively evaluating image blur, inability to accurately detect dynamic displacement, and easy neglect of multiple scene intersections.

[0006] (3) Limited freedom of shooting. To match fixed algorithms, users often need to constrain shooting conditions (such as specific time periods and fixed lighting environments), which limits the flexibility of reconstruction requirements such as "arbitrary angle and arbitrary environment". The practicality of outdoor exploration, emergency scenes and other scenes is limited. SUMMARY

[0007] In order to overcome the above-mentioned deficiencies of the prior art, the present application provides a 3DGS reconstruction method and system based on scene environment selection, which can realize high-quality three-dimensional reconstruction for images taken in any angle and any environment.

[0008] To achieve the above object, one or more embodiments of the present application provide the following technical solutions: The present application provides a 3DGS reconstruction method based on scene environment selection.

[0009] A 3DGS reconstruction method based on scene environment selection, comprising: Obtaining a multi-angle shooting video of a scene environment to be reconstructed, and dividing the obtained multi-angle shooting video into multiple pictures and storing them in the same folder; Traversing the multiple pictures in the folder, and sequentially performing scene judgment under multiple shooting scenes for each picture in the traversal process to determine the scene to which the current picture belongs; if the current picture is judged to belong to more than one of the multiple shooting scenes, the scene to which the current picture belongs is determined according to the scene priority. Statistically analyzing the proportion of the scenes to which all the pictures in the folder belong, and selecting a 3DGS variant suitable for the scene with the highest proportion for three-dimensional reconstruction.

[0010] Further, determining the scene to which the current picture belongs according to the scene priority comprises: sequentially judging whether the current picture meets the proportion conditions of human body, blur, low-light image, HDR image, dynamic image, and high-light image, and the first proportion condition met by the current picture corresponds to the scene to which the current picture belongs.

[0011] Further, the scene judgment under multiple shooting scenes for each picture in the traversal process comprises: low-light scene, high-light scene, HDR scene, blur scene, human body scene, and dynamic scene.

[0012] Further, the high-light scene judgment for the current picture comprises: calculating the number of pixel values with high brightness in the current picture based on the np.sum function, comparing the number of pixel values with high brightness with the total number of pixels in the current picture, and taking the size of the ratio after comparison as the judgment basis for whether the current picture belongs to the high-light scene.

[0013] Further, the blur scene judgment for the current picture comprises: performing grayscale conversion on the current picture, and using the Laplacian operator to perform edge detection on the current picture after grayscale conversion to obtain the second derivative of the current picture; and calculating the variance of the Laplacian response based on the obtained second derivative to determine whether the current picture is a blur scene.

[0014] Further, the current picture is subjected to human body scene judgment, including: first, the current picture is converted into an RGB format suitable for the MediaPiple function by using the MediaPiple function; subsequently, the current picture after format conversion is subjected to human body detection, and if key points of a human body can be detected in the current picture, a True result is returned.

[0015] Further, the current picture is subjected to dynamic scene judgment, including: first, a feature matching function is defined, the current picture and any picture adjacent to the current picture are converted into grayscale pictures, and key points and corresponding descriptors of the two pictures are extracted; subsequently, a brute force matcher is used to perform knn matching based on a Hamming distance to screen high-quality matching pairs and judge whether the two pictures belong to the same scene; finally, whether the current picture is a dynamic scene is determined based on average displacement and matching point estimation.

[0016] The second aspect of the present application provides a 3DGS reconstruction system based on scene environment selection.

[0017] A 3DGS reconstruction system based on scene environment selection, comprising: An image segmentation module configured to: acquire multi-angle shooting videos of a scene environment to be reconstructed, and segment the obtained multi-angle shooting videos into multiple pictures and store the multiple pictures in a same folder; A scene judgment module configured to: traverse the multiple pictures in the folder, and sequentially perform scene judgment under multiple shooting scenes on each picture in the traversal process to determine a scene to which the current picture belongs; if the current picture is judged to belong to more than one of the multiple shooting scenes, the scene to which the current picture belongs is determined according to a scene priority. A three-dimensional reconstruction module configured to: count a proportion of a scene to which all the pictures in the folder belong, and select a 3DGS variant suitable for the scene with the highest proportion to perform three-dimensional reconstruction. The third aspect of the present application provides a computer readable storage medium having a program stored thereon, the program being executed by a processor to implement the steps in the 3DGS reconstruction method based on scene environment selection according to the first aspect of the present application.

[0018] The fourth aspect of the present application provides an electronic device comprising a memory, a processor, and a program stored on the memory and executable on the processor, wherein the processor implements the steps in the 3DGS reconstruction method based on scene environment selection according to the first aspect of the present application when executing the program.

[0019] The above one or more technical solutions have the following beneficial effects: (1) The present application sequentially performs scene judgment under multiple shooting scenes for each picture traversed, that is, through quantitative detection algorithms of six types of scenes (such as low-light intensity threshold judgment, double threshold verification of HDR scene, displacement vector analysis of dynamic scene), the optimal 3DGS variant can be automatically matched for three-dimensional modeling, for example: for low-light scenes, normal 3DGS can be used in combination with image enhancement to restore dark details; for HDR scenes, HDR splatting can be enabled to accurately restore high-contrast levels; for dynamic scenes, DynGS can be called to eliminate motion smearing. Therefore, compared with the prior art, the present application can break through the limitation of standardized scenes and realize accurate adaptive reconstruction of complex scenes.

[0020] (2) When the current picture is judged to belong to more than one of multiple shooting scenes, the present application provides that the scene to which the current picture belongs is determined according to the scene priority, that is, when multiple conditions are triggered at the same time (such as night dynamic human body), the dominant scene is selected according to the priority, without manual intervention, and end-to-end automation from image input to algorithm output can be realized. Compared with the prior art, the present application can eliminate subjective misjudgment based on such automatic decision-making throughout the process.

[0021] (3) The present application breaks through the mandatory requirements for shooting environments and realizes "random shooting and building" under unconstrained conditions in terms of time, environment and operation, for example: in terms of time freedom, whether it is low light at night or strong exposure at noon, direct reconstruction is possible; in terms of environmental freedom, handheld shaking (blurred scenes), fast moving targets (dynamic scenes) and other non-ideal conditions are supported; in terms of operational freedom, the user only needs to provide a video stream taken at any angle without pre-adjusting parameters.

[0022] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0023] The drawings accompanying the specification of the present application serve to provide a further understanding of the present application, the illustrative embodiments of the present application and the description thereof serve to explain the present application and do not constitute an improper limitation of the present application.

[0024] Figure 1 A flowchart of a 3DGS reconstruction method based on scene environment selection in the embodiment one of the present application.

[0025] Figure 2 A flowchart of 3DGS reconstruction in the prior art.

[0026] Figure 3 A flowchart of determining the scene to which the current picture belongs according to the scene priority in the embodiment one of the present application. DETAILED DESCRIPTION

[0027] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0028] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit the exemplary embodiments according to the present application.

[0029] In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0030] In order to facilitate the understanding of the technical solutions of the present application, the following terms are explained: 1) Strategy_map: used to describe the parameters of different environments.

[0031] 2) np: Numpy is a python library for efficient matrix calculation.

[0032] 3) Ssim: used to calculate the structural similarity between two pictures, to determine whether the scene has changed.

[0033] 4) Lower's ratio test: a ratio test used in feature matching to eliminate false matches.

[0034] 5) Orb: a computer vision algorithm used to extract and describe key points in an image.

[0035] 6) BFMatcher: a brute force matcher in OpenCV, used to compare the distance between two feature descriptors one by one to find the nearest neighbor match.

[0036] 7) Laplancian Variance: Laplacian variance, a commonly used index for image blur detection.

[0037] 8) HDR: high dynamic range image, with a larger range of brightness and contrast than ordinary images.

[0038] 9) ffmpeg: a multimedia processing tool that can be used to record, convert and stream audio and video.

[0039] 10) MediaPiple: a data stream processing machine learning application development framework developed and open sourced by Google.

[0040] Embodiment one The present embodiment discloses a 3DGS reconstruction method based on scene environment selection.

[0041] As shown in Figure 1 A 3DGS reconstruction method based on scene environment selection, comprising: Step S1, obtaining a multi-angle shooting video of a scene environment to be reconstructed, and dividing the obtained multi-angle shooting video into multiple pictures and storing them in the same folder; Step S2, traversing the multiple pictures in the folder, and sequentially performing scene judgment under multiple shooting scenes for each picture in the traversal process to determine the scene to which the current picture belongs; if the current picture is judged to belong to more than one of the multiple shooting scenes, the scene to which the current picture belongs is determined according to the scene priority; Step S3, counting the proportion of the scene to which all pictures in the folder belong, and selecting a 3DGS variant suitable for the scene with the highest proportion for three-dimensional reconstruction.

[0042] As shown in Figure 2 The existing 3DGS reconstruction method often relies on manual experience to select appropriate shooting environments to adapt to the reconstruction strategy when dealing with different types of scenes. However, due to the significant differences in scene complexity such as texture complexity, lighting conditions, and blur degree, directly using a unified 3DGS modeling scheme often leads to problems such as insufficient reconstruction accuracy, artifact generation, or low rendering efficiency. At the same time, relying solely on the human eye to judge photos also has certain limitations. In view of this, the present application provides the above-mentioned 3DGS reconstruction method based on scene environment selection. Based on this method, there is no need to use manual judgment, and the user's cumbersome steps of finding a suitable 3DGS are reduced, which not only simplifies the process, but also allows the user to shoot scenes in different environments. At the same time, it increases the degree of freedom of the user's shooting, that is, for images shot at any angle and in any environment, the present application can achieve high-quality three-dimensional reconstruction. In order to facilitate the understanding of the technical scheme of the present application, the specific implementation method in the technical scheme of the present application will be further explained and described below.

[0043] In step S1, a multi-angle shooting video of a scene environment to be reconstructed is obtained, and the obtained multi-angle shooting video is divided into multiple pictures and stored in the same folder.

[0044] Use a camera to shoot a scene or an object building that the user needs to reconstruct at multiple angles (if the pixel quality of the shooting can be improved, it is better under the condition of allowing), cover the entire scene; then, process the shooting video, that is, use ffmpeg to divide the video to obtain pictures; and put the pictures into a folder as the input for subsequent scene judgment. Among them, ffmpeg is a multimedia processing tool for processing audio and video, which can be used to record, convert and stream audio and video.

[0045] In step S2, a plurality of pictures in the folder are traversed, and scene judgment under a plurality of shooting scenes is sequentially performed on each picture in the traversal process to determine the scene to which the current picture belongs; if the current picture is judged to belong to more than one of a plurality of shooting scenes, the scene to which the current picture belongs is determined according to the scene priority.

[0046] The scene judgment under a plurality of shooting scenes is sequentially performed on each picture in the traversal process, and the plurality of shooting scenes include a low-lightness scene, a high-lightness scene, an HDR scene, a blur scene, a human body scene, and a dynamic scene; wherein the low-lightness scene is a picture shot under low light (night), the high-lightness scene is a picture shot under high light (complex light), and the HDR scene is a picture shot under high dynamic (large color difference); the blur scene is a picture showing different blur degrees of different regions, and the human body scene is a picture shot on a person; the dynamic scene is a picture showing the change process of objects, light, motion, and other elements in the scene within a certain time.

[0047] The scene judgment under six shooting scenes is sequentially performed on each picture in the traversal process, and the specific steps are as follows: Firstly, the low-lightness scene of the current picture is judged, including: The input color picture (current picture) is converted into a gray picture, then np.mean (gray) is used to calculate the average brightness of the picture, and the threshold value threshold = 90 (in a common 8-bit gray picture, the pixel value range is [0, 255], and 80~100 is usually set as the threshold value of low light in industrial detection), when it is lower than this threshold value, it is considered to be low light. Then, the calculation result is returned and recorded for subsequent processing. Wherein, np.mean (gray) is a function function in python library for calling mean function to calculate the average, and gray represents the parameter required to be averaged.

[0048] Secondly, the high-lightness scene of the current picture is judged, including: pixel_ratio_threshold is used as the threshold value of the pixel high-lightness ratio, and brightness_threshold = 240 (255 is used as the maximum, and 240 is very close to the maximum value, which means that the pixel is in a high-light area) is used as the threshold value of the high-lightness pixel.

[0049] On this basis, first, the current picture is converted into a gray image based on the function body; wherein the function body, i.e. the function cv2.cvtColor(img, cv2.COLOR_BGR2GRAY), is a functional function in openCV that can convert an image from one color space to another color space. Subsequently, the number of pixel value highlights in the current picture after gray conversion is calculated based on the np.sum function, and the number of pixel value highlights obtained is compared with the total number of pixels of the current picture, and the size of the ratio after comparison is used as the basis for judging whether the current picture belongs to a high-light scene, i.e. if the ratio after comparison is greater than the defined threshold, it is considered to belong to a high-light scene. The calculation result is then returned for recording and subsequent processing. Wherein the np.sum function is a functional function in the python library for calling the sum function to calculate the sum.

[0050] The third step is to judge the HDR scene of the current picture, including: Define std_threshold=50 (in the range of gray pixels, the maximum standard deviation is about 76, so set a high value to represent that the brightness distribution difference is in the maximum difference, and if it is set too low, it will be misjudged as normal bright or dark but high contrast) to represent the threshold of whether the overall brightness standard deviation is large enough, and local_contrast_threshold=30 (too low is easy to misjudge noise or rich image, too high will miss the real HDR scene) to represent whether the local contrast is large enough.

[0051] On this basis, first, the picture is converted into a gray image, and the pixel brightness standard deviation of the entire current image is calculated using np.std; then, the size of the picture frame is set (which can be adjusted according to the size of the input picture later), and a list block is initialized to store the local brightness standard deviation; then, the picture is traversed, and the results are added to the list, the average value of the local brightness standard deviation is calculated, the overall local level is obtained, and then the defined threshold is used for judgment, and the result is returned and recorded for subsequent processing. Wherein the np.std function is a functional function in the python library for calling the standard deviation function to calculate the standard deviation.

[0052] HDR images usually contain multiple exposure levels, resulting in uneven brightness distribution, and the use of standard deviation as a judgment method; at the same time, local contrast is also an important information of the image, which is used to judge the local area light change.

[0053] The fourth step is to judge the blur scene of the current picture, including: Set threshold as the threshold of fuzzy judgment, convert the current picture to grayscale, and use Laplace operator to detect the edge of the current picture after grayscale conversion to get the second derivative of the current picture. The response ability of Laplace operator represents the definition of edge image. By calculating the variance of Laplace response, if the variance is large, it means that the edge information is rich and the image is clear; if the variance is small, it is the opposite. Return and record the result for subsequent processing.

[0054] Laplace operator is a commonly used second-order edge detection operator, which is sensitive to edge and attack changes of image. Variance of Laplace operator is often used as a quantitative index of image definition.

[0055] Step 5, human scene judgment of current picture, including: First, convert the current picture to RGB format suitable for MediaPiple function by using MediaPiple function; MediaPipe is an open source machine learning framework for multimedia tasks provided by Google, which judges human body by detecting whether there are human key points in the image. Then, use the defined human pose detector pose_detector to detect the human body in the current picture after format conversion. If the key points of human body can be detected in the current picture, return True result, save and return the result for subsequent processing. Further, the human pose detector can be represented as: pose_detector= mp_pose.Pose(static_image_mode=True, min_detection_confidence=0.5); Wherein, Mp_pose is a class provided by MeidaPipe to detect the key points of human body; min_detection_confidence represents the minimum confidence threshold that the pose detection model considers the detection result valid; static_image_mode indicates whether the input is considered as a static image. Thus, the personal body pose detector pose_detector is finally created.

[0056] Step 6, dynamic scene judgment of current picture, including: 1) Define feature matching function, that is, image feature matching function based on ORB feature extraction and BFMatcher. Convert the current picture and any picture adjacent to the current picture to grayscale image; then, use orb algorithm (based on average Euclidean distance of matching point displacement, which can well reflect the strength of relative motion between images) to extract key points and corresponding descriptors of the two pictures; according to the descriptor, judge whether to perform subsequent operation, if the key points are too few, exit directly.

[0057] The definition process of the feature matching function can be summarized as follows: First, grayscale processing is performed, i.e., the input picture is converted to a grayscale picture. Then, ORB is initialized and key points and descriptors are extracted. The key points represent the positions of the image with significant features, and the descriptors represent the feature vectors corresponding to each key point. Subsequently, exception handling is performed, BFMatcher is created, and KNN matching is performed. Finally, based on Lowe's Ratio Test, bad matches are filtered, and the result is returned.

[0058] Using the orb algorithm to extract the key points and corresponding descriptors of the two pictures can be summarized as follows: First, an instance of orb needs to be created (orb = cv2.ORB_create(nfeatures=1000)); then, the detectAndCompute() function is called, i.e.: kp1, des1 = orb.detectAndCompute(gray1, None); where gray1 represents the input grayscale picture, and None represents the mask parameter; Kp1 and des1 are the return values, representing the key point list and the descriptor matrix, respectively.

[0059] 2) Using the brute force matcher to perform knn matching based on hamming distance to filter good matching pairs and determine whether the two pictures belong to the same scene. Specifically: using the brute force matcher to perform knn matching based on hamming distance, returning the two nearest neighbors of each descriptor. Through lower's ratio test, good matching pairs are selected; where Lowe's Ratio Test is a strategy for filtering good matching pairs, by comparing the nearest neighbor and this nearest neighbor distance to eliminate false matches, it is widely used in feature matching. Then determine whether the two pictures are the same scene, i.e. using the number of good matches obtained by feature matching, when greater than or equal to the default threshold, it is considered as the same scene.

[0060] Finally, based on the average displacement and matching point estimation, determine whether the current picture is a dynamic scene, specifically: through the displacement of the feature matching points, estimate the intensity of the motion of the two pictures, continue to use the feature matching method to obtain the matching points, calculate the Euclidean distance, calculate the average value of the displacement of each matching point as the motion amplitude, calculate the proportion of good matching points in the total number of key points of the first image. Use the average displacement and matching point estimation to determine whether there is significant motion and not the same scene, and then determine whether it is a dynamic motion.

[0061] The SSIM calculates the structural similarity between two pictures, which can be used to determine whether the scene has changed, and thus can better reflect the changes of the image. The scene is determined to be dynamic by combining the number of feature matches, motion displacement, etc., which belongs to a multi-modal fusion decision method and has wide application in the field of action recognition.

[0062] After the above six steps, if the current picture is determined to belong to more than one of multiple shooting scenes, the scene to which the current picture belongs is determined according to the scene priority. The scene to which the current picture belongs is determined according to the scene priority, including: sequentially determining whether the current picture meets the proportion condition of human body, total number of blur, low-light image, HDR image, dynamic image, and high-light image. In the sequential determination process, the current picture first meets the proportion condition, and the current picture belongs to the corresponding scene.

[0063] As shown in Figure 3 , the scene to which the current picture belongs is determined according to the scene priority, specifically: 1) Determine whether there is a human body scene in the scene to which the current picture belongs. If there is, calculate the human body proportion; if not, go directly to the next step of judgment and do not calculate the human body proportion. When calculating the human body proportion, if the human body proportion is greater than 0.5, it is directly determined that the scene to which the current picture belongs is the human body scene, and the judgment process is exited.

[0064] 2) Determine whether there is a blur scene in the scene to which the current picture belongs. If there is, calculate the total number of blur proportions; if not, go directly to the next step of judgment and do not calculate the total number of blur proportions. When calculating the total number of blur proportions, if the total number of blur proportions is greater than 0.3, it is directly determined that the scene to which the current picture belongs is the blur scene, and the judgment process is exited.

[0065] 3) Determine whether there is a low-brightness (low-light) scene in the scene to which the current picture belongs. If there is, calculate the low-light image proportion; if not, go directly to the next step of judgment and do not calculate the low-light image proportion. When calculating the low-light image proportion, if the low-light image proportion is greater than 0.3, it is directly determined that the scene to which the current picture belongs is the low-brightness scene, and the judgment process is exited.

[0066] 4) Determine whether there is an HDR scene in the scene to which the current picture belongs. If there is, calculate the HDR image proportion; if not, go directly to the next step of judgment and do not calculate the HDR image proportion. When calculating the HDR image proportion, if the HDR image proportion is greater than 0.3, it is directly determined that the scene to which the current picture belongs is the HDR scene, and the judgment process is exited.

[0067] 5) judging whether there is a dynamic scene in the scene to which the current picture belongs, if there is, then calculating the dynamic image proportion, if not, then directly entering the next step of judgment, and no longer calculating the dynamic image proportion. When calculating the dynamic image proportion, if the dynamic image proportion is greater than 0.3, then it is directly determined that the scene to which the current picture belongs is a dynamic scene, and the judgment process is exited.

[0068] 6) judging whether there is a high brightness (high light) scene in the scene to which the current picture belongs, if there is, then calculating the high light image proportion, if not, then directly entering the next step of judgment, and no longer calculating the high light image proportion. When calculating the high light image proportion, if the high light image proportion is greater than 0.3, then it is directly determined that the scene to which the current picture belongs is a high brightness scene, and the judgment process is exited.

[0069] In step S3, the proportion of the scene to which all pictures in the folder belong is counted, and the 3DGS variant suitable for the scene with the highest proportion is selected for three-dimensional reconstruction.

[0070] After determining the scene based on the above process, the corresponding 3DGS variant is selected for three-dimensional reconstruction. Specifically, image enhancement processing and ordinary 3DGS are used in low light scenes, MS-GS is used in complex light scenes, SRGS is used in blurred scenes, HDRsplatting is used in high dynamic range scenes, GaussianAvatars is used in human shooting scenes, and DynGS is used in dynamic scenes.

[0071] Based on the 3DGS reconstruction method based on scene environment selection provided by the application, it is no longer necessary to distinguish by artificial or must force the scene to be shot, and it can be done "randomly". Not only does it speed up the reconstruction, but it also further improves the quality of the reconstruction; at the same time, it avoids the problem of being unable to judge or misjudge in time by artificial, and also reduces the cumbersome steps of users finding the corresponding 3DGS algorithm.

[0072] Embodiment Two The embodiment discloses a 3DGS reconstruction system based on scene environment selection.

[0073] A 3DGS reconstruction system based on scene environment selection, comprising: An image segmentation module configured to: multi-angle shoot the scene environment to be reconstructed, and segment the shot video into multiple pictures and store them in the same folder; The scene judgment module is configured to: traverse the plurality of pictures in the folder, and perform scene judgment in a plurality of shooting scenes on each picture in the traversal process in sequence to determine a scene to which the current picture belongs; and if the current picture is determined to belong to more than one of the plurality of shooting scenes, determine the scene to which the current picture belongs according to a scene priority. The three-dimensional reconstruction module is configured to: count a proportion of the scene to which all the pictures in the folder belong, and select a 3DGS variant adapted to the scene with the highest proportion to perform three-dimensional reconstruction. Embodiment three An object of this embodiment is to provide a computer-readable storage medium.

[0074] A computer-readable storage medium has a computer program stored thereon, which, when executed by a processor, implements the steps in the 3DGS reconstruction method based on scene environment selection according to Embodiment One of the present disclosure.

[0075] Embodiment four An object of this embodiment is to provide an electronic device.

[0076] An electronic device includes a memory, a processor, and a program stored on the memory and executable on the processor, and the processor implements the steps in the 3DGS reconstruction method based on scene environment selection according to Embodiment One of the present disclosure when executing the program.

[0077] The steps involved in the devices of Embodiments Two, Three, and Four above correspond to the method of Embodiment One, and the specific implementation can be seen in the relevant description of Embodiment One. The term “computer-readable storage medium” should be understood to include a single medium or multiple media of one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry instruction sets for execution by a processor and cause the processor to perform any of the methods in the present disclosure.

[0078] Those skilled in the art should understand that the above-mentioned modules or steps of the present disclosure can be implemented by a general computer device, and alternatively, they can be implemented by program code executable by a computing device, so that they can be stored in a storage device and executed by a computing device, or they can be made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module. The present disclosure is not limited to any specific combination of hardware and software.

[0079] The above describes the specific embodiments of the present disclosure in conjunction with the accompanying drawings, but is not a limitation on the scope of protection of the present disclosure. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present disclosure without creative labor are still within the scope of protection of the present disclosure.

Claims

1. A 3DGS reconstruction method based on scene environment selection, characterized in that, include: Obtain multi-angle video footage of the scene environment to be reconstructed, split the obtained multi-angle video footage into multiple images and save them into the same folder; The system iterates through multiple images in the folder, performing scene judgments under various shooting scenarios on each image during the iteration process to determine the scene to which the current image belongs. If the current image is determined to belong to more than one of the aforementioned shooting scenarios, the scene to which the current image belongs is determined according to the scene priority. The system calculates the percentage of each scene represented by all images in the folder and selects the 3DGS variant that matches the scene with the highest percentage for 3D reconstruction.

2. The 3DGS reconstruction method based on scene environment selection as described in claim 1, characterized in that, The scene to which the current image belongs is determined according to scene priority, including: judging whether the current image meets the proportion conditions of human body, total number of blurs, low light image, HDR image, dynamic image, and high light image in turn. In the process of judging in turn, the proportion condition that the current image meets first corresponds to the scene to which the current image belongs.

3. The 3DGS reconstruction method based on scene environment selection as described in claim 1, characterized in that, For each image in the traversal process, scene judgments are performed under various shooting scenarios, including: low brightness scene, high brightness scene, HDR scene, blurred scene, human body scene, and dynamic scene.

4. The 3DGS reconstruction method based on scene environment selection as described in claim 3, characterized in that, The process involves determining whether the current image is a high-brightness scene, including: calculating the number of highlighted pixels in the current image using the np.sum function, comparing the number of highlighted pixels with the total number of pixels in the current image, and using the ratio as the criterion for determining whether the current image belongs to a high-brightness scene.

5. The 3DGS reconstruction method based on scene environment selection as described in claim 3, characterized in that, The process for determining whether the current image is a blurred scene includes: performing grayscale conversion on the current image, and using the Laplacian operator to perform edge detection on the grayscale-converted current image to obtain the second derivative of the current image; and calculating the variance of the Laplacian response based on the obtained second derivative to determine whether the current image is a blurred scene.

6. The 3DGS reconstruction method based on scene environment selection as described in claim 3, characterized in that, The process involves determining the presence of a human in the current image, including: first, using the MediaPiple function to convert the current image to an RGB format compatible with the MediaPiple function; then, performing human detection on the converted image, and returning True if key points of a human are detected in the current image.

7. The 3DGS reconstruction method based on scene environment selection as described in claim 3, characterized in that, The dynamic scene determination of the current image includes: First, defining a feature matching function to convert the current image and any adjacent image into grayscale images, and extracting key points and corresponding descriptors from the two images; then, using a brute-force matcher based on Hamming distance to perform k-nearest neighbor matching to filter high-quality matching pairs and determine whether the two images belong to the same scene; finally, determining whether the current image is a dynamic scene based on average displacement and matching point estimation.

8. A 3DGS reconstruction system based on scene environment selection, characterized in that, include: The image segmentation module is configured to: acquire multi-angle video of the scene environment to be reconstructed, segment the acquired multi-angle video into multiple images and save them into the same folder; The scene determination module is configured to: traverse multiple images in the folder, perform scene determination under multiple shooting scenarios for each image during the traversal process to determine the scene to which the current image belongs; if the current image is determined to belong to more than one of the multiple shooting scenarios, then determine the scene to which the current image belongs according to the scene priority. The 3D reconstruction module is configured to: count the percentage of each scene to which all images in the folder belong, and select the 3DGS variant that matches the scene with the highest percentage for 3D reconstruction.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in a 3DGS reconstruction method based on scene environment selection as described in any one of claims 1-7.

10. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the 3DGS reconstruction method based on scene environment selection as described in any one of claims 1-7.

Images