A gaussian point cloud training system and method based on background separation

Abstract

This invention discloses a Gaussian point cloud training system and method based on background separation, relating to the field of Gaussian point cloud technology. The method includes: acquiring historical construction records, extracting corresponding feature points, obtaining the scene change degree of an image frame set based on the coordinate positions of the feature points in a spatial coordinate system, obtaining the image change degree of the image frame set based on the grayscale histogram of the images in the image frame set, and drawing a scatter plot; acquiring the current target video, obtaining the target frame interval based on the target video and the scatter plot, extracting keyframes and non-keyframes, and processing them to obtain a rendered static background, thereby achieving fusion optimization of the foreground and background of the scene within the target video. This invention effectively solves the problem of efficient training for scenes with mixed dynamic foreground and static background by analyzing keyframes and non-keyframes, avoiding redundant calculations, reducing computational complexity, and significantly improving efficiency.

Description

Technical Field

[0001] This invention relates to the field of Gaussian point cloud technology, specifically a Gaussian point cloud training system and method based on background separation. Background Technology

[0002] Using Gaussian point cloud technology to process videos for high-quality 3D scene rendering is a current research hotspot and important development direction in the field of 3D reconstruction and rendering. Compared with traditional point cloud reconstruction methods, Gaussian point cloud technology models the 3D structure in the scene as a large number of point clouds with various feature attributes, which can capture the detailed texture, light and shadow changes, and even complex optical properties such as translucency and reflection of the object surface more delicately. However, when processing complex videos containing dynamic foregrounds and static backgrounds, existing Gaussian point cloud technology requires complete training of all Gaussian point clouds for each frame. Since the training time increases linearly with the number of frames, the static background is repeatedly calculated in each frame, which causes a huge waste of computing resources and leads to low training efficiency. Summary of the Invention

[0003] The purpose of this invention is to provide a Gaussian point cloud training system and method based on background separation, so as to solve the problems raised in the prior art.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A Gaussian point cloud training method based on background separation includes the following steps:

[0006] Step S1: Obtain historical construction records of 3D models built based on Gaussian point clouds. Generate corresponding image frame sets based on the image sequences in the videos corresponding to the construction records, and extract corresponding feature points from the generated 3D models. Based on the coordinate positions of the feature points in the spatial coordinate system, obtain the scene change degree of the image frame set. Based on the grayscale histogram of each image in the image frame set, obtain the image change degree of the image frame set, and draw a scatter plot between the scene change degree and the image change degree.

[0007] Step S2: Obtain the target video currently used for 3D model reconstruction. Based on the grayscale histogram and scatter plot of each frame in the target video, obtain the target frame interval of the target video. Then, extract the key frames and non-key frames in the target video according to the target frame interval.

[0008] Step S3: Initially train the keyframes, establish and update the Gaussian set, and determine the foreground and background point clouds in each keyframe by analyzing the changes in the point clouds in the keyframes; extract the camera shooting angle of the keyframes, render the background point cloud, and obtain the final rendered static background.

[0009] Step S4: Initialize the point cloud in non-keyframes. Based on the rendered static background, extract and optimize the foreground point cloud in non-keyframes, thereby achieving the fusion optimization of the foreground and background of the scene in the target video.

[0010] Preferably, step S1 includes:

[0011] Step S1-1: Obtain historical construction records. Construction records are image sequences generated from videos containing only the background area as input. A 3D model is constructed using Gaussian point cloud technology, and the construction process of the model is recorded. Extract all image sequences from the video V corresponding to a certain construction record R. Take several images in the image sequence at preset frame intervals as a set of image frames to obtain several image frame sets. Obtain the model formed by each image in a set of image frames S, as well as the position, RGB value, and covariance matrix of each point cloud in each model. Obtain the brightness of each point cloud based on the RGB value, and perform eigenvalue decomposition on the covariance matrix to obtain the corresponding eigenvalues ​​and eigenvectors.

[0012] 3D Gaussian Splatting is an emerging 3D scene reconstruction technique. It achieves high-quality 3D scene rendering by training a large number of point clouds with features such as position, covariance, color, and transparency. Each frame in the image sequence formed by the video can form a corresponding 3D model. The process of obtaining the brightness of the point cloud from the RGB values ​​and performing feature decomposition on the covariance matrix are existing technologies and will not be elaborated here.

[0013] Step S1-2: Obtain a point cloud set formed by several adjacent point clouds. If the variance between the feature values ​​of each point cloud in the point cloud set is less than a preset variance threshold in any feature vector direction, then the point cloud set is marked. If the brightness of a certain point cloud P in the marked point cloud set is the largest compared with the brightness of several adjacent point clouds, then point cloud P is taken as a feature point, and thus all feature points in the model formed based on each frame image are obtained.

[0014] Preferably, step S1 further includes the following steps:

[0015] Steps S1-3: Establish the spatial coordinate system corresponding to the scene in the video, obtain all feature points in the model formed by each frame image in the image frame set S, and obtain the coordinate points corresponding to each feature point in the spatial coordinate system through coordinate alignment technology.

[0016] Extract the coordinates of a feature point F1 within the model formed by a certain frame T1 in the image frame set S. Take the next frame of image T1 in the image frame set S as T2, and take the feature point within the model formed by image T2 that is closest to feature point F1 as F2. If the coordinate distance between feature point F1 and feature point F2 is L... 12 If the distance is less than the preset distance threshold, then the coordinate distance L will be... 12 The target distance of feature point F1 is used as the target distance; then all target distances corresponding to image T1 are obtained, and the average distance of image T1 is obtained by averaging. Then the average distance of each image in the image frame set S is obtained, and the variance between the average distances is used as the scene change degree of image frame set S.

[0017] Steps S1-4: Convert each image in the image frame set S into a grayscale image and establish a grayscale histogram corresponding to each grayscale image. Then, obtain the histogram similarity between any two adjacent grayscale images in the image frame set S, and calculate the average of all histogram similarities to obtain the degree of image change in the image frame set S. Based on the several image frame sets obtained from the video V, draw a scatter plot of the degree of scene change as a function of the degree of image change.

[0018] Preferably, step S2 includes the following steps:

[0019] Step S2-1: Obtain the interval range [W1, W2] for extracting frames from the target video, where W1 and W2 are the preset minimum and maximum frame intervals, respectively; obtain the set of image frames extracted from the target video with a frame interval of W0, convert each image in the set of image frames into a grayscale image, and then obtain the degree of image change Y0 of the set of image frames.

[0020] Step S2-2: Extract the scene change degree of N scatter points that are closest to the scene change degree Y0 from the scatter plot, and calculate the average value to obtain the scene change degree X0 of the image frame set, and then obtain the target value W0 / X0 with a frame number interval of W0; obtain the target value corresponding to each frame number interval within the interval range, and take the frame number interval corresponding to the maximum value as the target frame number interval; then extract key frames from the image sequence of the target video according to the target frame number interval, and the rest are non-key frames.

[0021] To simplify the calculations for static background rendering, the frame interval W0 should be relatively large. However, to ensure more reliable static background rendering and adaptability to lighting conditions, the scene change level X0 should be relatively small. The target value W0 / X0 satisfies the requirement that a larger frame interval W0 and a smaller scene change level X0 result in a more reasonable frame interval W0. Therefore, the frame interval corresponding to the maximum target value is taken as the target frame interval, and key frames and non-key frames are extracted based on the target frame interval, which is a balanced solution that takes into account both computation and reliability.

[0022] Preferably, step S3 includes: establishing a Gaussian meta-set G, obtaining a 3D point cloud model corresponding to each keyframe, and randomly extracting several point clouds from the 3D point cloud model and inputting them into the Gaussian meta-set G; extracting the camera shooting perspective corresponding to each keyframe, rendering a prediction image based on the Gaussian meta-set G, then calculating the rendering loss based on the prediction image and the real image of the keyframe, and finally updating the feature attributes of all point clouds in the Gaussian meta-set G; determining the foreground and background point clouds of each keyframe in the target video by analyzing the degree of change of the point clouds in the time series; pre-rendering and caching the background image based on the camera shooting perspective, establishing a background image cache library, and obtaining the final rendered static background.

[0023] Preferably, step S4 includes: using the foreground point cloud of the keyframe as an initialization template, performing Gaussian primitive initialization on the foreground point cloud of the non-keyframe; training the foreground point cloud of the non-keyframe based on the obtained rendered static background, specifically including rendering the foreground, compositing it with the rendered static background to obtain a composite image, and calculating the loss between the composite image and the real image, thereby updating the feature attributes of the foreground point cloud of the non-keyframe; and simultaneously rendering the foreground point cloud and background point cloud according to the obtained foreground point cloud and background point cloud in the keyframe and non-keyframe to obtain a complete scene, thereby accurately synthesizing a complete scene model.

[0024] A Gaussian point cloud training system based on background separation includes a scatter plot building module, a target frame interval calculation module, a static background rendering module, and a scene fusion optimization module.

[0025] Scatter plot generation module: Used to acquire historical construction records of 3D models built based on Gaussian point clouds, generate corresponding image frame sets based on the image sequences in the corresponding videos of the construction records, and extract corresponding feature points from the generated 3D models; based on the coordinate positions of the feature points in the spatial coordinate system, obtain the scene change degree of the image frame set; based on the grayscale histogram of each image in the image frame set, obtain the image change degree of the image frame set, and draw a scatter plot between the scene change degree and the image change degree.

[0026] Target Frame Interval Calculation Module: This module is used to obtain the target video currently used for 3D model reconstruction. Based on the grayscale histogram and scatter plot of each frame in the target video, it obtains the target frame interval of the target video and then extracts key frames and non-key frames from the target video based on the target frame interval.

[0027] Static background rendering module: used for initial training of keyframes, establishing and updating Gaussian meta-sets, determining foreground and background point clouds in each keyframe by analyzing the changes in point clouds in the keyframes, extracting the camera shooting angle of the keyframes, rendering the background point cloud, and obtaining the final rendered static background.

[0028] Scene fusion optimization module: It is used to initialize the point cloud in non-key frames, extract and optimize the foreground point cloud in non-key frames based on the rendered static background, and then realize the fusion optimization of the foreground and background of the scene in the target video.

[0029] Preferably, the target frame interval calculation module includes an image change degree calculation unit and a target frame interval calculation unit;

[0030] Image change degree calculation unit: used to obtain the interval range for extracting frames of images in the target video; obtain the set of image frames extracted from the target video at a certain frame interval, convert each image in the image frame set into a grayscale image, and then obtain the image change degree of the image frame set;

[0031] Target frame number interval calculation unit: It is used to obtain the scene change degree of the image frame set based on the scatter plot and the degree of image change, and then obtain the target frame number interval; and extract key frames from the image sequence of the target video according to the target frame number interval, and the rest are non-key frames.

[0032] Preferably, the scene fusion optimization module includes a scene fusion optimization unit;

[0033] Scene fusion optimization unit: Used to initialize the foreground point cloud of keyframes as an initialization template and perform Gaussian primitive initialization on the foreground point cloud of non-keyframes; Based on the obtained rendered static background, it trains the foreground point cloud of non-keyframes, specifically including rendering the foreground, compositing it with the rendered static background to obtain a composite image, and calculating the loss between the composite image and the ground truth image, thereby updating the feature attributes of the foreground point cloud of non-keyframes; Based on the obtained foreground point cloud and background point cloud in keyframes and non-keyframes, it simultaneously renders the foreground point cloud and background point cloud to obtain a complete scene, and then accurately synthesizes a complete scene model.

[0034] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention provides a Gaussian point cloud training system and method based on background separation, comprising: acquiring historical construction records, extracting corresponding feature points, obtaining the scene change degree of the image frame set based on the coordinate positions of the feature points in the spatial coordinate system, obtaining the image change degree of the image frame set based on the grayscale histogram of the images in the image frame set, and drawing a scatter plot; acquiring the current target video, and obtaining the target frame interval based on the target video and the scatter plot, extracting key frames and non-key frames, and processing them to obtain the rendered static background, thereby achieving the fusion optimization of the foreground and background of the scene in the target video. This invention effectively solves the problem of efficient training for scenes with mixed dynamic foreground and static background by analyzing key frames and non-key frames, avoiding redundant calculations, reducing computational complexity, and significantly improving efficiency. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating a Gaussian point cloud training method based on background separation according to the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Example: Figure 1 As shown, this invention provides a Gaussian point cloud training method based on background separation, comprising the following steps:

[0038] Step S1: Obtain historical construction records of 3D models built based on Gaussian point clouds. Generate corresponding image frame sets based on the image sequences in the videos corresponding to the construction records, and extract corresponding feature points from the generated 3D models. Based on the coordinate positions of the feature points in the spatial coordinate system, obtain the scene change degree of the image frame set. Based on the grayscale histogram of each image in the image frame set, obtain the image change degree of the image frame set, and draw a scatter plot between the scene change degree and the image change degree.

[0039] Step S1-1: Obtain historical construction records. Construction records are image sequences generated from videos containing only the background area as input. A 3D model is constructed using Gaussian point cloud technology, and the construction process of the model is recorded. Extract all image sequences from the video V corresponding to a certain construction record R. Take several images in the image sequence at preset frame intervals as a set of image frames to obtain several image frame sets. Obtain the model formed by each image in a set of image frames S, as well as the position, RGB value, and covariance matrix of each point cloud in each model. Obtain the brightness of each point cloud based on the RGB value, and perform eigenvalue decomposition on the covariance matrix to obtain the corresponding eigenvalues ​​and eigenvectors.

[0040] Step S1-2: Obtain a point cloud set formed by several adjacent point clouds. If the variance between the feature values ​​of each point cloud in the point cloud set is less than a preset variance threshold in any feature vector direction, then the point cloud set is marked. If the brightness of a certain point cloud P in the marked point cloud set is the largest compared with the brightness of several adjacent point clouds, then point cloud P is taken as a feature point, and thus all feature points in the model formed based on each frame image are obtained.

[0041] Since this scheme analyzes three-dimensional space, the number of eigenvectors and eigenvalues ​​after eigenvalue decomposition is three. If the variance among all eigenvalues ​​is small in any eigenvector direction, it indicates that the point clouds within the point cloud set belong to the same location category. For example, for a wall, when the covariance matrix of each point cloud in the point cloud set is decomposed, the eigenvalues ​​corresponding to the three eigenvector directions are not significantly different from each other, that is, the fluctuation of the eigenvalues ​​in each direction is small. It can be determined that these point clouds belong to the same wall. This is because the point clouds of the same wall have a consistent spatial distribution, and the dominant direction (eigenvector) and corresponding intensity (eigenvalue) of its dispersion will not change drastically, so the variance among the eigenvalues ​​is naturally small. However, if the point clouds belong to different location categories (such as some from the wall and some from the edge), the eigenvalues ​​will fluctuate significantly due to differences in geometric features, and the variance will also be large. This scheme extracts feature points from the marked point cloud set precisely to eliminate the influence of different location categories on the calculation of the degree of change in the following scene, thereby making the data in the scatter plot more reliable.

[0042] Steps S1-3: Establish the spatial coordinate system corresponding to the scene in the video, obtain all feature points in the model formed by each frame image in the image frame set S, and obtain the coordinate points corresponding to each feature point in the spatial coordinate system through coordinate alignment technology.

[0043] Extract the coordinates of a feature point F1 within the model formed by a certain frame T1 in the image frame set S. Take the next frame of image T1 in the image frame set S as T2, and take the feature point within the model formed by image T2 that is closest to feature point F1 as F2. If the coordinate distance between feature point F1 and feature point F2 is L... 12 If the distance is less than the preset distance threshold, then the coordinate distance L will be... 12 The target distance of feature point F1 is used as the target distance; then all target distances corresponding to image T1 are obtained, and the average distance of image T1 is obtained by averaging. Then the average distance of each image in the image frame set S is obtained, and the variance between the average distances is used as the scene change degree of image frame set S.

[0044] Since the models formed by each image are different, it is unreasonable to analyze each model directly. Therefore, in this step, coordinate alignment technology is needed to obtain the coordinates of each feature point in the spatial coordinate system. Coordinate alignment technology is an existing technology and will not be elaborated here. Variance represents the degree of brightness change of the scene between different images. The larger the variance, the greater the degree of scene change within the image frame set.

[0045] Steps S1-4: Convert each image in the image frame set S into a grayscale image and establish a grayscale histogram corresponding to each grayscale image. Then, obtain the histogram similarity between any two adjacent grayscale images in the image frame set S, and calculate the average of all histogram similarities to obtain the degree of image change in the image frame set S. Based on the several image frame sets obtained from the video V, draw a scatter plot of the degree of scene change as a function of the degree of image change.

[0046] Step S2: Obtain the target video currently used for 3D model reconstruction. Based on the grayscale histogram and scatter plot of each frame in the target video, obtain the target frame interval of the target video. Then, extract the key frames and non-key frames in the target video according to the target frame interval.

[0047] Step S2-1: Obtain the interval range [W1, W2] for extracting frames from the target video, where W1 and W2 are the preset minimum and maximum frame intervals, respectively; obtain the set of image frames extracted from the target video with a frame interval of W0, convert each image in the set of image frames into a grayscale image, and then obtain the degree of image change Y0 of the set of image frames.

[0048] Step S2-2: Extract the scene change degree of N scatter points that are closest to the scene change degree Y0 from the scatter plot, and calculate the average value to obtain the scene change degree X0 of the image frame set, and then obtain the target value W0 / X0 with a frame number interval of W0; obtain the target value corresponding to each frame number interval within the interval range, and take the frame number interval corresponding to the maximum value as the target frame number interval; then extract key frames from the image sequence of the target video according to the target frame number interval, and the rest are non-key frames.

[0049] Step S3: Initially train the keyframes, establish and update the Gaussian set, and determine the foreground and background point clouds in each keyframe by analyzing the changes in the point clouds in the keyframes; extract the camera shooting angle of the keyframes, render the background point cloud, and obtain the final rendered static background.

[0050] Step S3 includes: establishing a Gaussian meta-set G, obtaining a 3D point cloud model corresponding to each keyframe, and randomly extracting several point clouds from the 3D point cloud model and inputting them into the Gaussian meta-set G; extracting the camera shooting perspective corresponding to each keyframe, rendering a prediction image based on the Gaussian meta-set G, then calculating the rendering loss based on the prediction image and the real image of the keyframe, and finally updating the feature attributes of all point clouds in the Gaussian meta-set G; determining the foreground and background point clouds of each keyframe in the target video by analyzing the degree of change of the point clouds in the time series; pre-rendering and caching the background image based on the camera shooting perspective, establishing a background image cache library, and obtaining the final rendered static background.

[0051] This step processes keyframes. In this scheme, based on the camera's shooting angle v and the Gaussian set G, the generated predicted image is I_pred = Render(G, camera_pose[v]). Then, based on the predicted image I_pred and the real image I_gt[v], the rendering loss between the two images is loss += Loss(I_pred, I_gt[v]). Finally, the position, covariance, color, transparency, and other feature attributes of all point clouds in the Gaussian set G are updated. Using multi-frame point cloud registration technology, the degree of change of the point clouds over time is obtained, determining the foreground point cloud G_fg and the background point cloud G_bg. The background point cloud is the point cloud generated in all frames. The static background remains unchanged, such as fixed environmental elements like walls and the ground. The foreground point cloud contains dynamically changing foreground objects, such as people and moving objects. To avoid redundant calculations in subsequent training, the background image needs to be pre-rendered and cached for each training viewpoint. Based on the camera's shooting viewpoint v, the mathematical representation of the rendered background image is I_bg[v] = Render(G_bg, camera_pose[v]). The predicted image generated by rendering, the calculation of rendering loss, and the multi-frame point cloud registration technique are all existing technologies and will not be elaborated here. Since the static background only needs to be calculated once and can be reused in multiple non-key frames, the rendering strategy obtained in this scheme can significantly reduce computational overhead.

[0052] Step S4: Initialize the Gaussian point cloud in the non-keyframes. Based on the rendered static background, extract and optimize the foreground point cloud in the non-keyframes, thereby achieving the fusion optimization of the foreground and background of the scene in the target video.

[0053] Step S4 includes: using the foreground point cloud of the keyframe as an initialization template, and performing Gaussian primitive initialization on the foreground point cloud of the non-keyframe; training the foreground point cloud of the non-keyframe based on the obtained rendered static background, specifically including rendering the foreground, compositing it with the rendered static background to obtain a composite image, and calculating the loss between the composite image and the ground truth image, thereby updating the feature attributes of the foreground point cloud of the non-keyframe; and simultaneously rendering the foreground point cloud and background point cloud according to the obtained foreground point cloud and background point cloud in the keyframe and non-keyframe to obtain a complete scene, thereby accurately synthesizing a complete scene model.

[0054] This step processes non-keyframes. First, the Gaussian point cloud of the non-keyframe is initialized using the foreground point cloud of the keyframe. Specifically, the foreground point cloud G_fg extracted from the keyframe is used as the initialization template to initialize the Gaussian point cloud of the foreground region of the non-keyframe. By inheriting the spatial distribution and basic attributes of the foreground point cloud of the keyframe, a good foundation can be laid for subsequent independent training of the foreground. For the dynamic part of the non-keyframe, only the foreground point cloud is optimized, and efficient compositing is performed using a pre-rendered background. Specifically: 1. Render the foreground of the non-keyframe; 2. Composite it with the rendered static background to obtain a composite image; 3. Calculate the loss between the composite image and the ground truth image; 4. Update only the feature attributes of the foreground point cloud of the non-keyframe, while keeping the background unchanged. This strategy can focus on optimizing the dynamic foreground and avoid redundant calculations of the static background. Then, based on the foreground point cloud and background point cloud in the obtained keyframes and non-keyframes, the foreground point cloud and background point cloud are rendered simultaneously to obtain the complete scene, and then the complete scene model is accurately synthesized. The optimization strategy for the foreground point cloud and background point cloud adopts differentiated processing. The foreground point cloud is optimized with full feature attributes, while the background is only optimized with transparency. At the same time, the natural transition and seamless integration of the edge area are ensured, and any possible compositing traces are eliminated.

[0055] This invention also provides a Gaussian point cloud training system based on background separation, comprising: a scatter plot establishment module, a target frame interval calculation module, a static background rendering module, and a scene fusion optimization module. The target frame interval calculation module includes an image change degree calculation unit and a target frame interval calculation unit, and the scene fusion optimization module includes a scene fusion optimization unit. When the system executes a computer program, it implements the aforementioned Gaussian point cloud training method based on background separation. Since this Gaussian point cloud training method based on background separation has been described in detail above, it will not be repeated here.

[0056] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A background separation based Gaussian point cloud training method, characterized in that, The method comprises the following steps: Step S1: obtaining a historical construction record of a three-dimensional model based on a Gaussian point cloud, generating a corresponding image frame set according to an image sequence in a video corresponding to the construction record, and extracting corresponding feature points from the generated three-dimensional model; obtaining a scene change degree of the image frame set based on the coordinate positions of the feature points in a spatial coordinate system, obtaining an image change degree of the image frame set based on a gray histogram of each image in the image frame set, and drawing a scatter plot between the scene change degree and the image change degree; Step S2: obtaining a target frame interval of a target video currently used for three-dimensional model reconstruction according to a gray histogram of each frame of image in the target video and the scatter plot, and then extracting key frames and non-key frames in the target video according to the target frame interval; Step S3: performing initial training on the key frames, establishing and updating a Gaussian primitive set, determining foreground point clouds and background point clouds in each key frame by analyzing the change of the point clouds in the key frames, extracting a camera shooting angle of the key frames, rendering the background point clouds to obtain a final rendered static background; Step S4: initializing the point clouds in the non-key frames, extracting and optimizing the foreground point clouds in the non-key frames based on the rendered static background, and then realizing fusion and optimization of the foreground and the background in the scene in the target video. 2.The background separation based Gaussian point cloud training method of claim 1, wherein, Step S1 comprises: Step S1-1: obtaining a historical construction record, wherein the construction record is an image sequence generated by a video with only a background region as input, a three-dimensional model is constructed by Gaussian point cloud technology, and a record of the model construction process is recorded; all image sequences in a video V corresponding to a certain construction record R are extracted, a certain number of images in the image sequences are taken as an image frame set according to a preset frame interval, and a certain number of image frame sets are obtained; the position, RGB value and covariance matrix of each point cloud in each model formed by each image in a certain image frame set S are obtained, the brightness of each point cloud is obtained according to the RGB value, and the covariance matrix is subjected to eigenvalue decomposition to obtain corresponding eigenvalues and eigenvectors; Step S1-2: obtaining a point cloud set formed by a certain number of adjacent point clouds, if the variance between the eigenvalues of each point cloud in the point cloud set in any eigenvector direction is less than a preset variance threshold, the point cloud set is marked; if the brightness of a certain point cloud P in the marked point cloud set is the maximum compared with the brightness of a certain number of adjacent point clouds, the point cloud P is taken as a feature point, and then all feature points in the model formed according to each frame of image are obtained. 3.The background separation based Gaussian point cloud training method of claim 2, wherein, Step S1 further comprises the following steps: Step S1-3: establishing a spatial coordinate system corresponding to the scene in the video, obtaining all feature points in the model formed by each frame of image in the image frame set S, and obtaining the corresponding coordinate points of each feature point in the spatial coordinate system through coordinate alignment technology; The coordinate point corresponding to a feature point F1 in a model formed by a certain frame image T1 in the image frame set S is extracted, the next frame image of the image T1 in the image frame set S is taken as T2, the feature point closest to the feature point F1 in the model formed by the image T2 is taken as F2, and if the coordinate distance L between the feature point F1 and the feature point F2 is less than a preset distance threshold, the coordinate distance L is taken as the target distance of the feature point F1; the target distances of all feature points in the image T1 are obtained, the average distance of the image T1 is obtained by averaging, the average distances of all images in the image frame set S are obtained, and the variance between the average distances is taken as the scene change degree of the image frame set S. 12 12 The coordinate distance L is taken as the target distance of the feature point F1; the target distances of all feature points in the image T1 are obtained, the average distance of the image T1 is obtained by averaging, the average distances of all images in the image frame set S are obtained, and the variance between the average distances is taken as the scene change degree of the image frame set S.​ Step S1-4: converting each image in the image frame set S into a grayscale image, establishing a grayscale histogram corresponding to each grayscale image, and then obtaining the histogram similarity between any two adjacent grayscale images in the image frame set S, and averaging all the histogram similarities to obtain the image change degree of the image frame set S; based on the plurality of image frame sets obtained from the video V, a scatter plot of scene change degree versus image change degree is drawn.

4. The Gaussian point cloud training method based on background separation according to claim 3, characterized in that, Step S2 includes the following steps: Step S2-1: obtaining the interval range [W1, W2] for frame extraction of images in the target video, W1 and W2 being the preset minimum frame interval and maximum frame interval, respectively; obtaining the image frame set extracted from the target video at the frame interval W0, converting each image in the image frame set into a grayscale image, and then obtaining the image change degree Y0 of the image frame set; Step S2-2: extracting the scene change degrees of the N closest scatter points to the image change degree Y0 from the scatter plot, and averaging to obtain the scene change degree X0 of the image frame set, and then obtaining the target value of the frame interval W0 as W0 / X0; obtaining the target value corresponding to each frame interval in the interval range, and taking the frame interval corresponding to the maximum value as the target frame interval; and then extracting key frames from the image sequence of the target video at the target frame interval, and the rest are non-key frames.

5. The method of claim 1, wherein, Step S3 includes: establishing a Gaussian cell set G, obtaining a three-dimensional point cloud model corresponding to each key frame, and randomly extracting a plurality of point clouds from the three-dimensional point cloud model and inputting them into the Gaussian cell set G; extracting the camera shooting angle corresponding to each key frame, rendering a predicted image based on the Gaussian cell set G, and then calculating the rendering loss according to the predicted image and the real image of the key frame, and finally updating the feature attributes of all point clouds in the Gaussian cell set G; determining the foreground point cloud and the background point cloud of each key frame in the target video by analyzing the change degree of the point cloud in the time sequence; based on the camera shooting angle, pre-rendering and caching the background image to establish a background image cache library, and obtaining the final rendered static background.

6. The method of claim 1, wherein, Step S4 includes: taking the foreground point cloud of the key frame as an initialization template to initialize the foreground point cloud of the non-key frame; based on the obtained rendered static background, training the foreground point cloud of the non-key frame, including rendering the foreground, synthesizing with the rendered static background to obtain a synthesized image, and calculating the loss between the synthesized image and the real image, and then updating the feature attributes of the foreground point cloud of the non-key frame; according to the obtained foreground point cloud and background point cloud in the key frame and non-key frame, rendering the foreground point cloud and background point cloud to obtain a complete scene, and then accurately synthesizing a complete scene model.

7. A background separation based Gaussian point cloud training system for performing the background separation based Gaussian point cloud training method of any one of claims 1-6. The system includes a scatter plot establishment module, a target frame interval calculation module, a static background rendering module, and a scene fusion optimization module. The scatter plot establishment module is configured to obtain a construction record of a historical Gaussian point cloud-based three-dimensional model construction, generate a corresponding image frame set according to an image sequence in a video corresponding to the construction record, and extract corresponding feature points from the generated three-dimensional model; obtain a scene change degree of the image frame set based on coordinate positions of the feature points in a spatial coordinate system, obtain an image change degree of the image frame set based on a gray histogram of each image in the image frame set, and draw a scatter plot between the scene change degree and the image change degree; The target frame interval calculation module is configured to obtain a target video currently used for three-dimensional model reconstruction, obtain a target frame interval of the target video according to a gray histogram of each image in the target video and the scatter plot, and then extract key frames and non-key frames in the target video according to the target frame interval; The static background rendering module is configured to perform initial training on the key frames, establish and update a Gaussian primitive set, determine foreground point clouds and background point clouds in each key frame by analyzing a change of the point clouds in the key frames, extract a camera shooting angle of the key frames, render the background point clouds, and obtain a final rendered static background; The scene fusion optimization module is configured to initialize the point clouds in the non-key frames, extract and optimize the foreground point clouds in the non-key frames based on the rendered static background, and then realize fusion and optimization of foreground and background scenes in the target video.

8. The background separation based Gaussian point cloud training system of claim 7, wherein, The target frame interval calculation module includes an image change degree calculation unit and a target frame interval calculation unit; The image change degree calculation unit is configured to obtain an interval range for frame extraction of images in the target video, obtain an image frame set extracted from the target video at a certain frame interval, convert each image in the image frame set into a gray image, and then obtain an image change degree of the image frame set; The target frame interval calculation unit is configured to obtain a scene change degree of the image frame set according to the scatter plot and the image change degree, then obtain the target frame interval, and extract key frames from an image sequence of the target video at the target frame interval, and the rest are non-key frames.

9. The background separation based Gaussian point cloud training system of claim 7, wherein, The scene fusion optimization module includes a scene fusion optimization unit; The scene fusion optimization unit is configured to use the foreground point clouds of the key frames as an initialization template to initialize the foreground point clouds of the non-key frames, train the foreground point clouds of the non-key frames based on the obtained rendered static background, specifically including rendering foreground, synthesizing with the rendered static background to obtain a synthesis image, calculating a loss of the synthesis image and a real image, and then updating feature attributes of the foreground point clouds of the non-key frames; The foreground point clouds and the background point clouds in the obtained key frames and non-key frames are simultaneously rendered to obtain a complete scene, and then a complete scene model is accurately synthesized.

Images