4Dgs model training method and three-dimensional scene reconstruction method

Abstract

The present disclosure provides a 4DGS model training method and a three-dimensional scene reconstruction method, and relates to the technical fields of three-dimensional modeling and computer vision. The method comprises: acquiring laser radar point cloud data of a target scene; generating depth supervision information based on depth information obtained by projecting the laser radar point cloud data; constructing a geometric supervision loss of a 4DGS model based on the depth supervision information; and training the 4DGS model based on the geometric supervision loss and an image reconstruction loss. The present disclosure can improve the representation ability of the 4DGS model for the spatial structure of the target scene and improve the accuracy of the three-dimensional scene reconstruction result by introducing the depth supervision information generated based on the laser radar point cloud data to train the 4DGS model.

Description

Technical Field

[0001] This disclosure relates to the fields of 3D modeling and computer vision technology, and more specifically, to a 4DGS model training method and a 3D scene reconstruction method. Background Technology

[0002] With the development of computer vision, 3D scene modeling, and neural rendering technologies, the digital reconstruction of target scenes using acquired data has been widely applied in fields such as autonomous driving, virtual reality, digital twins, and 3D content generation. In order to improve the realism and usability of scene reconstruction results, existing solutions typically construct scene representation models to train and represent the target scene, thereby obtaining the corresponding scene reconstruction results. Especially in complex scenes, how to balance the ability to represent scene details, model training efficiency, and the quality of reconstruction results has become a continuously concerning technical issue in this field.

[0003] Currently, some 3D scene reconstruction schemes primarily train scene representation models based on image data and optimize the image reconstruction results to allow the model to gradually learn the appearance and structural information of the target scene. Under this approach, the model training process typically revolves around image reconstruction errors, ensuring that the final scene reconstruction result visually approximates the target scene well. This type of scheme is relatively mature and applicable to various scene reconstruction tasks, thus it has been widely used in existing technologies.

[0004] However, since these methods primarily rely on image data for training, and the training constraints are largely derived from the image imaging results themselves, they still have limitations in representing spatial hierarchy, distance relationships, and local structural relationships within the target scene. In cases of complex scene structures and significant changes in spatial relationships, problems such as inaccurate scene structure representation and unstable local region reconstruction results can easily arise, thus affecting the accuracy of the 3D scene reconstruction results.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a 4DGS model training method and a 3D scene reconstruction method. By introducing depth supervision information generated based on LiDAR point cloud data to train the 4DGS model, the ability of the 4DGS model to represent the spatial structure of the target scene can be improved, and the accuracy of the 3D scene reconstruction results can be improved.

[0007] According to a first aspect of the present disclosure, a 4DGS model training method is provided, comprising: Acquire LiDAR point cloud data of the target scene; Depth supervision information is generated based on the depth information obtained by projecting the lidar point cloud data into the point cloud. Based on the aforementioned deep supervision information, the geometric supervision loss of the 4DGS model is constructed. The 4DGS model is trained based on the geometric supervision loss and image reconstruction loss.

[0008] In one exemplary embodiment of this disclosure, generating depth supervision information based on the depth information obtained by projecting the lidar point cloud data includes: Obtain the camera parameters corresponding to the lidar point cloud data; Based on the camera parameters, the lidar point cloud data is projected onto the camera image plane corresponding to the camera parameters to obtain the depth information of each pixel position in the camera image plane; The depth supervision information is generated based on the depth information of each pixel position.

[0009] In one exemplary embodiment of this disclosure, the step of projecting the lidar point cloud data onto a camera image plane corresponding to the camera parameters, based on the camera parameters, to obtain depth information of each pixel position in the camera image plane, includes: Obtain the three-dimensional coordinate information of each sampling point in the lidar point cloud data; Based on the camera parameters, coordinate transformation and projection mapping are performed on the three-dimensional coordinate information of each sampling point to obtain the projection position and corresponding depth value of each sampling point in the camera image plane. The depth information of each pixel position in the camera image plane is determined based on the projection position of each sampling point in the camera image plane and the corresponding depth value.

[0010] In an exemplary embodiment of this disclosure, the step of performing coordinate transformation and projection mapping on the three-dimensional coordinate information of each sampling point based on the camera parameters to obtain the projection position and corresponding depth value of each sampling point in the camera image plane includes: Based on the extrinsic parameter matrix in the camera parameters, the three-dimensional coordinate information of each sampling point is transformed into the camera coordinate system to obtain the camera coordinate information of each sampling point; Based on the camera coordinate information of each sampling point, determine the depth value corresponding to each sampling point; Based on the intrinsic parameter matrix in the camera parameters, the camera coordinate information of each sampling point is projected onto the camera image plane to obtain the projection position of each sampling point in the camera image plane.

[0011] In one exemplary embodiment of this disclosure, generating the depth supervision information based on the depth information of each pixel position includes: Based on the depth information of each pixel position, the target pixel position that meets the preset depth condition and the target depth value corresponding to each target pixel position are determined. The depth supervision information is generated based on the target pixel position and the target depth value.

[0012] In one exemplary embodiment of this disclosure, constructing the geometric supervision loss of the 4DGS model based on the depth supervision information includes: Obtain the predicted depth information output by the 4DGS model from the target's perspective; The predicted depth information is compared with the depth supervision information to determine the depth error; Based on the depth error, the geometric supervision loss of the 4DGS model is constructed.

[0013] In one exemplary embodiment of this disclosure, training the 4DGS model based on the geometric supervision loss and image reconstruction loss includes: Acquire image data of the target scene; The image data is input into the 4DGS model to obtain the rendered image; Based on the rendered image and the image data, the image reconstruction loss is determined; The 4DGS model is trained based on the geometric supervision loss and the image reconstruction loss.

[0014] In one exemplary embodiment of this disclosure, the method further includes: Obtain the Gaussian parameters in the 4DGS model; Based on the Gaussian parameters, determine the Gaussian sparsity loss; The target loss is determined based on the geometric supervision loss, the image reconstruction loss, and the Gaussian sparsity loss. The parameters of the 4DGS model are updated based on the target loss.

[0015] In one exemplary embodiment of this disclosure, the method further includes: The lidar point cloud data is subjected to noise point filtering processing to obtain the first point cloud data; Outlier removal is performed on the first point cloud data to obtain the second point cloud data; The second point cloud data is processed using coordinate system one to obtain preprocessed lidar point cloud data, and the depth supervision information is generated based on the preprocessed lidar point cloud data.

[0016] According to a second aspect of the present disclosure, a three-dimensional scene reconstruction method is provided, comprising: Acquire image data of the target scene; The image data is input into a pre-trained 4DGS model, and the target scene is reconstructed based on the 4DGS model to obtain a three-dimensional scene model of the target scene. The 4DGS model is obtained according to the 4DGS model training method described in the first aspect of the present disclosure.

[0017] The technical solutions provided in this disclosure may have the following beneficial effects: The 4DGS model training method provided in this exemplary embodiment introduces a supervisory basis that differs from simple image imaging results during the model training phase. This allows the constraints during training to no longer focus solely on image appearance information, but to further reflect spatial distance and hierarchical relationships within the target scene. Consequently, the model learns the spatial structure of the target scene more fully during training, improving its ability to represent spatial relationships within the scene.

[0018] Building upon this, since the introduced constraints further participate in the model parameter optimization process, the model optimization not only focuses on image appearance reconstruction but is also constrained by spatial geometric relationships. Therefore, in complex scenes, the structural representation of local regions, edge regions, and regions with significant hierarchical changes will more closely resemble the real spatial distribution. This reduces the deviation in scene structural representation and improves the accuracy and stability of structural representation.

[0019] Furthermore, by expanding the type of information used during training from image appearance information to spatial depth information, the model's learning foundation for the geometric structure of the target scene is enhanced, making the resulting 3D scene reconstruction results closer to the actual spatial structure of the target scene overall. This improves the accuracy of the 3D scene reconstruction results.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0022] Figure 1 A system architecture diagram is shown for a 4DGS model training method and a 3D scene reconstruction method that can be applied to embodiments of this disclosure.

[0023] Figure 2 A flowchart illustrating a 4DGS model training method according to an embodiment of this disclosure is shown.

[0024] Figure 3 A schematic diagram of a process for generating deep supervision information is shown in an embodiment of this disclosure.

[0025] Figure 4 A schematic diagram of a process for determining depth information of the position of each pixel in a camera image plane is shown in an embodiment of this disclosure.

[0026] Figure 5 A flowchart illustrating a three-dimensional scene reconstruction method according to an embodiment of this disclosure is shown.

[0027] Figure 6 A schematic diagram of a 4DGS model training apparatus according to an embodiment of the present disclosure is shown.

[0028] Figure 7 A schematic diagram of a three-dimensional scene reconstruction apparatus according to an embodiment of the present disclosure is shown.

[0029] Figure 8 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown.

[0030] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0031] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0032] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0033] First, the relevant terms used in the exemplary embodiments of this disclosure will be explained: The 4DGS model is a scene representation model that uses four-dimensional Gaussian scattering to represent and model a target scene. The "4D" in 4DGS indicates that the model can further describe the scene by incorporating a temporal dimension on top of a three-dimensional spatial representation, and "GS" stands for Gaussian Splatting, which uses multiple Gaussian elements to express the spatial distribution of the target scene. This model can learn the spatial structure and appearance information of the target scene based on input data, and after training, it can be used to generate a corresponding 3D scene model or scene reconstruction result.

[0034] Specifically, the four-dimensional Gaussian scattering method represents the scene using a large number of three-dimensional Gaussian ellipsoids and combines this with volumetric rendering technology to generate images. Therefore, it has advantages such as fast rendering speed, strong scene representation capabilities, and the ability to achieve real-time display. Building upon this, the four-dimensional Gaussian scattering method further expands the expressive power of the Gaussian field, making it applicable to the representation of more complex scene structures.

[0035] However, existing 3D scene reconstruction methods based on 4DGS models still have certain shortcomings in practical applications. Current solutions typically rely primarily on RGB images for model supervision, using image reconstruction error as the main optimization objective. Due to the lack of constraints on the true 3D geometry of the target scene, the model tends to focus on fitting the image appearance during training, while learning the scene's spatial structure is relatively insufficient. This can lead to reconstruction results with large depth errors, object contour distortion, and unstable spatial structures, thus affecting the accuracy of 3D scene reconstruction.

[0036] Furthermore, when relying solely on image supervision, 4DGS models may approximate 2D imaging results by adjusting the position and scale of the Gaussian distribution to fit image colors, which can easily lead to depth structure drift. Specifically, this can result in issues such as blurred object edges, object position shifts, and incorrect foreground-background depth relationships, and these problems are often more pronounced in complex scenes. At the same time, due to the lack of further constraints on scene structure from real sensor data, the reconstruction results may, in some cases, be inconsistent with the real physical space, exhibiting phenomena such as floating objects, uneven ground, and distorted building structures, thus affecting the realism and usability of the 3D scene reconstruction results.

[0037] Based on one or more of the above-mentioned technical problems, this disclosure provides a 4DGS model training method and a 3D scene reconstruction method. By introducing depth supervision information generated based on LiDAR point cloud data to train the 4DGS model, the ability of the 4DGS model to represent the spatial structure of the target scene can be improved, and the accuracy of the 3D scene reconstruction results can be improved.

[0038] Figure 1 A schematic diagram of a system architecture for a 4DGS model training method and a 3D scene reconstruction method that can be applied to embodiments of the present disclosure is shown.

[0039] like Figure 1 As shown, system architecture 100 may include one or more terminal devices such as smartphone 101, portable computer 102, and desktop computer 103, network 104, and server 105. Network 104 is used as a medium to provide a communication link between the terminal devices and server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0040] The terminal device can be any electronic device with data processing capabilities, and can provide data display and interactive functions through a graphical user interface. For example, it can display image data of the target scene, LiDAR point cloud data, model training progress, and 3D scene reconstruction results, and can also support the configuration, viewing, or adjustment of information related to model training or scene reconstruction. It is understood that the electronic device includes, but is not limited to, the aforementioned desktop computers, portable computers, smartphones, and tablets, and may also include vehicle terminals, intelligent camera devices, etc.

[0041] Server 105 can be a device with data processing and model training capabilities, used to receive LiDAR point cloud data of a target scene and execute the 4DGS model training method provided in this embodiment. Specifically, server 105 can generate depth supervision information based on the depth information obtained by projecting the LiDAR point cloud data, and construct the geometric supervision loss of the 4DGS model based on the depth supervision information, so as to train the 4DGS model based on the geometric supervision loss and the image reconstruction loss, thereby obtaining a pre-trained 4DGS model.

[0042] After obtaining the pre-trained 4DGS model, the server 105 can also execute the 3D scene reconstruction method provided in this embodiment of the disclosure, that is, acquire image data of the target scene and input the image data into the pre-trained 4DGS model to obtain a 3D scene model corresponding to the target scene. Of course, the trained 4DGS model can also be deployed in a terminal device for 3D scene reconstruction.

[0043] In some embodiments, image data and lidar point cloud data can be collected by the same terminal device; in other embodiments, image data and lidar point cloud data can be collected by different devices and then aggregated to server 105 for processing. This disclosure does not limit this.

[0044] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0045] This disclosure provides a 4DGS model training method, referring to... Figure 2 As shown, the method may include steps S210 to S240: Step S210: Obtain lidar point cloud data of the target scene; Step S220: Generate depth supervision information based on the depth information obtained by projecting the lidar point cloud data into the point cloud. Step S230: Based on deep supervision information, construct the geometric supervision loss of the 4DGS model; Step S240: Train the 4DGS model based on geometric supervision loss and image reconstruction loss.

[0046] The 4DGS model training method provided in this disclosure expands the information used for model training from image imaging results to information that can reflect the spatial distance and hierarchical relationships of the target scene. This allows the parameter optimization process to be constrained by the geometric structure of the scene, thereby enhancing the model's ability to learn the spatial structure of the target scene. This makes the structural representation in complex scenes closer to the real spatial distribution relationship, and thus improves the accuracy of the 3D scene reconstruction results.

[0047] The 4DGS model training method in this example embodiment will be described in detail below.

[0048] In step S210, the lidar point cloud data of the target scene is acquired.

[0049] In the exemplary embodiments of this disclosure, the target scene can be a road scene, an indoor scene, a park scene, or other scenes to be modeled in 3D. LiDAR point cloud data is a data set obtained after a LiDAR device scans the target scene, including spatial coordinate information corresponding to multiple sampling points in the target scene. This allows the data to characterize the spatial distribution and structural contours of object surfaces in the target scene, providing real-world geometric information. In some embodiments, each sampling point in the LiDAR point cloud data may include not only 3D coordinate information but also reflection intensity information, timestamp information, etc., which is not limited in this disclosure.

[0050] For example, a LiDAR device can be used to scan a target scene in real time to obtain LiDAR point cloud data corresponding to the target scene. For instance, in autonomous driving applications, a LiDAR device installed on a vehicle can scan targets such as roads, vehicles, pedestrians, and buildings around the vehicle in real time; as another example, in a 3D modeling scenario of a park, a LiDAR device set up on a mobile acquisition platform can scan roads, green areas, building facades, etc. within the park.

[0051] LiDAR equipment emits laser beams into a target scene and receives the laser echoes reflected from the surfaces of objects in the target scene to determine the spatial coordinates of each sampling point, thereby forming LiDAR point cloud data.

[0052] In other embodiments, lidar point cloud data corresponding to the target scene can be read directly from a pre-stored dataset, cache unit, or storage device, and this disclosure does not limit this.

[0053] By obtaining information that can characterize the spatial structure of the target scene, a data foundation is provided for subsequent point cloud projection based on LiDAR point cloud data, generation of depth supervision information, and construction of geometric supervision loss.

[0054] In step S220, depth supervision information is generated based on the depth information obtained by projecting the lidar point cloud data into the point cloud.

[0055] After obtaining the LiDAR point cloud data of the target scene, further point cloud projection processing can be performed to obtain depth information corresponding to the target scene. Point cloud projection processing involves mapping the spatial location information of each sampling point in the LiDAR point cloud data to its corresponding position on an image plane or a preset projection plane. This establishes a correspondence between spatial points in the scene and planar positions, and determines the depth at each corresponding position. Through point cloud projection processing, the spatial structure information in the original point cloud data can be converted into depth information that is easily utilized in subsequent training processes. This depth information can be used to characterize the spatial distance relationships between different locations in the target scene.

[0056] Next, deep supervision information can be generated based on the depth information so that it can serve as a supervision basis in the subsequent model training stage.

[0057] It is understandable that depth supervision information is not a direct copy of the original point cloud data, but rather information further formed from the depth information obtained by point cloud projection processing, used to constrain the 4DGS model training process. In other words, depth information focuses more on reflecting the depth result obtained after point cloud projection, while depth supervision information focuses on representing the data form that can be directly used for model training supervision.

[0058] By transforming the scene geometric structure information contained in LiDAR point cloud data into deep supervision information that can be used for training, a foundation can be provided for the subsequent construction of geometric supervision loss.

[0059] In some example implementations, after acquiring the LiDAR point cloud data of the target scene, the LiDAR point cloud data can be preprocessed to improve the data quality upon which subsequent depth supervision information is based. For example, the preprocessing process may include noise point filtering, outlier removal, and coordinate system unification. By performing such preprocessing on the original LiDAR point cloud data, the impact of outliers, isolated points, and coordinate system inconsistencies on the subsequent point cloud projection results can be reduced.

[0060] Specifically, the LiDAR point cloud data can first be processed by filtering out noise points to obtain the first point cloud data. Next, outlier removal is performed on the first point cloud data to obtain the second point cloud data. Finally, coordinate system one processing is applied to the second point cloud data to obtain preprocessed LiDAR point cloud data, which is then used to generate depth supervision information.

[0061] The noise filtering process for LiDAR point cloud data refers to identifying and filtering noise points generated during the LiDAR scanning process due to environmental interference, equipment measurement errors, or abnormal echoes, in order to obtain the first point cloud data. Noise points can be characterized by abnormal spatial distribution, poor continuity with surrounding point sets, or points that do not conform to the actual structural distribution characteristics of the target scene. This disclosure does not limit the specific implementation method of noise filtering; for example, filtering methods based on neighborhood statistical features, filtering methods based on distance thresholds, or other applicable data filtering methods can be used.

[0062] After obtaining the first point cloud data, outlier removal can be performed to obtain the second point cloud data. Outliers typically refer to points that deviate significantly from the main point cloud distribution and are difficult to reflect the true structural information of the target scene. Compared with noise points, outliers, although also anomalous data, are more characterized by their deviation from the effective point set in the overall spatial distribution. By removing outliers, the remaining point cloud data can be made more concentrated and continuous in spatial distribution, thereby improving the reliability of the depth information obtained from subsequent point cloud projection. This disclosure does not limit the specific implementation method of outlier removal; for example, a neighborhood density-based removal method, a distance distribution statistical removal method, or other applicable outlier removal methods can be used.

[0063] After obtaining the second point cloud data, coordinate system one processing can be performed on it to obtain preprocessed LiDAR point cloud data. Coordinate system one processing refers to transforming the coordinate representation of the second point cloud data to the target coordinate system required for subsequent processing, so as to facilitate the subsequent point cloud projection and depth information generation. For example, the target coordinate system can be a coordinate system corresponding to the camera parameters, or it can be a preset unified scene coordinate system; this disclosure does not limit this. By performing coordinate system one processing, the preprocessed LiDAR point cloud data can maintain consistency in coordinate representation, thereby facilitating the subsequent generation of depth supervision information based on the preprocessed LiDAR point cloud data.

[0064] By preprocessing LiDAR point cloud data, we can obtain preprocessed LiDAR point cloud data with higher data quality, more stable spatial distribution, and consistent coordinate representation, thus providing a foundation for the subsequent generation of depth supervision information.

[0065] In some example implementations, references Figure 3 As shown, the process of generating deep supervision information may further include steps S310 to S330: Step S310: Obtain the camera parameters corresponding to the lidar point cloud data.

[0066] As can be understood, camera parameters refer to parameter information used to characterize camera imaging relationships. They can be used to reflect the correspondence between spatial points in the target scene and image positions in the camera image plane, thus providing a parameter basis for subsequently mapping LiDAR point cloud data to the camera image plane. The camera image plane refers to the plane corresponding to the camera imaging result, used to carry the image content of the target scene from the camera's perspective.

[0067] In this example, the camera parameters can be derived from an image acquisition device that works in conjunction with the LiDAR and correspond to the current LiDAR point cloud data. It should be noted that the camera parameters and the LiDAR point cloud data are acquired for the same target scene, and together they reflect scene information under the same acquisition time, the same acquisition stage, or the same viewpoint relationship. This allows the camera parameters to be used to characterize the mapping relationship between the LiDAR point cloud data and the camera image plane.

[0068] Step S320: Based on the camera parameters, project the LiDAR point cloud data onto the camera image plane corresponding to the camera parameters to obtain the depth information of each pixel position in the camera image plane.

[0069] In the exemplary embodiments of this disclosure, a correspondence between LiDAR point cloud data and the camera image plane can be established using camera parameters, so that the spatial location information represented by each sampling point in the LiDAR point cloud data is mapped to the corresponding position in the camera image plane. Here, pixel position refers to a position unit in the camera image plane, used to represent a specific location in the image.

[0070] Based on this mapping relationship, the depth information corresponding to each pixel position in the camera image plane can be determined. Accordingly, depth information refers to information used to characterize the spatial distance relationship between the corresponding pixel position in the target scene and the camera, thereby reflecting the spatial depth distribution of the target scene from the camera's perspective.

[0071] For example, refer to Figure 4 As shown, step S320 may further include steps S410 to S430: Step S410: Obtain the three-dimensional coordinate information of each sampling point in the lidar point cloud data.

[0072] It is understandable that the three-dimensional coordinate information of each sampling point is used to characterize the position of the corresponding sampling point in three-dimensional space. In some implementations, the three-dimensional coordinate information can be represented as ( x , y , z ),in, x , y and z These represent the position coordinates of the corresponding sampling point in three spatial dimensions, which are used together to determine the position of the sampling point in three-dimensional space.

[0073] By acquiring the three-dimensional coordinate information of each sampling point in the lidar point cloud data, we can provide basic data for subsequent coordinate transformation and projection mapping of each sampling point.

[0074] Step S420: Based on the camera parameters, perform coordinate transformation and projection mapping on the three-dimensional coordinate information of each sampling point to obtain the projection position and corresponding depth value of each sampling point in the camera image plane.

[0075] In this embodiment, coordinate transformation refers to transforming the three-dimensional coordinate information of each sampling point according to the camera parameters so that the spatial position information of each sampling point meets the requirements of subsequent projection processing; projection mapping refers to mapping each sampling point after coordinate transformation to the corresponding position in the camera image plane; correspondingly, the projection position refers to the position of each sampling point after being mapped to the camera image plane; and the depth value refers to the value corresponding to each sampling point during the projection process, which is used to characterize the spatial distance relationship between the sampling point and the camera.

[0076] Specifically, based on the extrinsic parameter matrix in the camera parameters, the 3D coordinate information of each sampling point can be transformed to the camera coordinate system to obtain the camera coordinate information of each sampling point. In other words, the extrinsic parameter matrix can be used to perform coordinate system transformation on the original spatial coordinates of each sampling point in the LiDAR point cloud data, so that the spatial position information of each sampling point is transformed from the original coordinate representation to the coordinate representation in the camera coordinate system.

[0077] The extrinsic parameter matrix is ​​a parameter matrix used to characterize the spatial relationship between the lidar coordinate system and the camera coordinate system. It can include rotation and translation parameters, and is used to describe the attitude and positional relationship of the lidar coordinate system relative to the camera coordinate system. Based on the extrinsic parameter matrix, the three-dimensional coordinate information of each sampling point in the lidar coordinate system can be mapped to the camera coordinate system, thereby achieving a unified transformation between different coordinate systems.

[0078] For example, matrix operations can be performed between the 3D coordinate information of each sampling point and the extrinsic parameter matrix to achieve coordinate system transformation. For instance, for any sampling point, its 3D coordinates in the lidar coordinate system can be expressed as ( x , y , zAfter homogeneous representation, a corresponding coordinate vector can be constructed. Then, based on the rotation parameters in the extrinsic parameter matrix, this coordinate vector undergoes an orientation transformation. Finally, based on the translation parameters in the extrinsic parameter matrix, the transformed coordinate result is offset to obtain the camera coordinate information of the sampling point in the camera coordinate system. It can be seen that the coordinate system transformation process includes adjusting both the orientation and positional relationships of the sampling points, so that the sampling point positions originally represented in the LiDAR coordinate system are converted to positions represented in the camera coordinate system.

[0079] After obtaining the camera coordinate information of each sampling point, the depth value corresponding to each sampling point can be determined based on the camera coordinate information of each sampling point. Specifically, the depth value is determined based on the spatial position information of the sampling point in the camera coordinate system. Since the positional relationship of each sampling point has been unified to the camera coordinate system after the coordinate system transformation, the spatial distance information of each sampling point relative to the camera can be further extracted or determined from the camera coordinate information, and used as the depth value of the corresponding sampling point.

[0080] Furthermore, based on the intrinsic parameter matrix in the camera parameters, the camera coordinate information of each sampling point can be projected onto the camera image plane to obtain the projected position of each sampling point in the camera image plane. In other words, since each sampling point already has a coordinate representation in the camera coordinate system, the intrinsic parameter matrix can be used to establish a positional mapping relationship between it and the camera image plane, thereby determining the specific projected position of each sampling point in the camera image plane.

[0081] The intrinsic parameter matrix is ​​a parameter matrix used to characterize the imaging relationship of the camera itself. It may include parameters such as focal length and principal point, and describes the mapping relationship between spatial points in the camera coordinate system and image positions in the camera image plane. Based on the intrinsic parameter matrix, the camera coordinate information of each sampling point in the camera coordinate system can be mapped to the camera image plane, thereby realizing the transformation from three-dimensional spatial coordinates to two-dimensional image positions.

[0082] For example, matrix operations can be performed between the camera coordinate information of each sampling point and the intrinsic parameter matrix to achieve projection mapping. For instance, for any sampling point, after obtaining its camera coordinate information, the normalized positional relationship of the sampling point in the camera coordinate system can be determined first based on the camera coordinate information. Then, combined with the focal length parameter and principal point parameter in the intrinsic parameter matrix, the sampling point is mapped to its corresponding position in the camera image plane, thus obtaining the projected position of the sampling point in the camera image plane. The projection mapping process converts the spatial position information in the camera coordinate system into a planar position representation in the camera image plane, characterizing the landing position of the corresponding sampling point in the image.

[0083] This embodiment allows us to obtain the depth value corresponding to each sampling point and the projection position of each sampling point on the camera image plane. This establishes the correspondence between the spatial position of the sampling point, the depth result, and the position on the image plane, providing a foundation for generating depth information of each pixel position in the camera image plane.

[0084] Step S430: Determine the depth information of each pixel position in the camera image plane based on the projection position of each sampling point in the camera image plane and the corresponding depth value.

[0085] After obtaining the projection position and corresponding depth value of each sampling point in the camera image plane, the pixel position corresponding to each sampling point in the camera image plane can be determined based on the projection position of each sampling point. The depth value corresponding to each sampling point is then associated with the corresponding pixel position to obtain the depth information of each pixel position in the camera image plane. In other words, the pixel position to which each sampling point falls can be determined based on its planar position representation in the camera image plane, and the depth value corresponding to that sampling point can then be used as the depth result for that pixel position.

[0086] For example, when a pixel location corresponds to one sampling point, the depth value corresponding to that sampling point can be directly determined as the depth information of that pixel location. When the same pixel location corresponds to multiple sampling points, a target depth value can be determined from the depth values ​​corresponding to the multiple sampling points, and the target depth value can be used as the depth information of that pixel location. For example, the minimum depth value among multiple depth values ​​can be selected as the depth information of that pixel location to represent a scene point closer to the camera from the camera's perspective; alternatively, a target depth value can be selected from multiple depth values ​​according to preset rules, and this disclosure does not limit this approach.

[0087] For example, when some pixel positions in the camera image plane do not correspond to any sampling points, the depth information of those pixel positions can be set to a preset null value, a default value, or marked as an invalid depth position; this disclosure does not limit this. Through the above processing, each pixel position in the camera image plane can be associated with the corresponding depth result, thereby forming a pixel-level depth information distribution in the camera image plane.

[0088] By performing this step, the spatial location information and depth values ​​at the sampling point level can be further converted into pixel-level depth information in the camera image plane, thus providing a basis for the subsequent generation of depth supervision information.

[0089] Step S330: Generate depth supervision information based on the depth information of each pixel position.

[0090] After obtaining the depth information of each pixel position in the camera image plane, depth supervision information for model training can be further formed based on the depth information, so that the scene geometric structure information contained in the LiDAR point cloud data can participate in the subsequent training process in the form of supervision information.

[0091] For example, based on the depth information of each pixel location, target pixel locations that meet preset depth conditions and the corresponding target depth values ​​for each target pixel location are determined. The preset depth conditions can be used to filter valid depth results that can participate in subsequent training supervision. This disclosure does not limit the specific form of the preset depth conditions. In some embodiments, the preset depth condition can be that the corresponding pixel location has a non-empty depth value, thereby filtering out pixel locations with depth results. The preset depth condition can also be that the depth value of the corresponding pixel location falls within a preset depth range, thereby filtering out pixel locations whose depth results are within a reasonable range.

[0092] Specifically, the process iterates through the pixel positions in the camera image plane and determines whether the depth information corresponding to each pixel position meets a preset depth condition. If the depth information corresponding to a pixel position meets the preset depth condition, then that pixel position is determined as the target pixel position, and the depth value corresponding to that pixel position is determined as the target depth value. If the depth information corresponding to a pixel position does not meet the preset depth condition, then that pixel position is not included in the subsequent depth supervision information generation process.

[0093] After determining the target pixel positions and their corresponding target depth values, depth supervision information can be generated based on these positions and depth values. Specifically, a correspondence between target pixel positions and target depth values ​​can be established, and supervision data for model training can be constructed based on this correspondence.

[0094] For example, the target pixel locations and their corresponding target depth values ​​can be combined to form a depth-supervised dataset; alternatively, the target depth values ​​corresponding to each target pixel location can be filled into the supervision result corresponding to the camera image plane to form depth-supervised information for training. This disclosure does not limit this approach. Through this processing, the pixel locations that meet the conditions and their corresponding depth results can participate in the subsequent training process in a structured form.

[0095] In step S230, the geometric supervision loss of the 4DGS model is constructed based on the deep supervision information.

[0096] After generating deep supervision information, a geometric supervision loss for the 4DGS model can be further constructed based on this information. The geometric supervision loss is a loss term used to characterize the degree of difference between the 4DGS model output and the deep supervision information, and it serves to constrain model parameter updates during training.

[0097] By introducing geometric supervised loss, the optimization criteria during model training can no longer be limited to the image appearance reconstruction results, but can also further reflect the spatial depth relationship and geometric structure information in the target scene.

[0098] Understandably, depth supervision information represents the depth result at a corresponding location in the target scene, while geometric supervision loss is based on the difference between the model output and the depth supervision information. Accordingly, a corresponding loss term can be constructed by comparing the consistency between the model's output depth result and the depth supervision information, and this loss term can be used as the optimization basis during the 4DGS model training process. Through this process, the 4DGS model can further learn the spatial geometry of the target scene during training, thus providing a geometric constraint basis for subsequent model parameter optimization.

[0099] In some example implementations, the predicted depth information output by the 4DGS model from the target viewpoint can be obtained. The target viewpoint refers to the viewpoint information used to perform the current 4DGS model output or training computation. After the 4DGS model performs scene representation or rendering processing from the target viewpoint, it can output predicted depth information corresponding to that target viewpoint. This predicted depth information characterizes the 4DGS model's prediction of depth results at various locations in the target scene from the current viewpoint, thus serving as the basis for subsequent comparisons with depth supervision information.

[0100] After obtaining the predicted depth information, it can be further compared with the depth supervision information to obtain the depth error. The depth error refers to the degree of difference between the predicted depth information and the depth supervision information, used to characterize the deviation between the model's current output and the supervised results. Optionally, the difference between the predicted depth result and the supervised depth result at each corresponding position can be calculated to determine the depth error at each position; alternatively, the depth error can be determined based on the difference between the overall predicted result and the overall supervised result. This disclosure does not limit this approach.

[0101] Finally, based on the depth error, a geometric supervision loss for the 4DGS model is constructed. That is, the depth error can be used as the basis for constructing the loss term, allowing the geometric supervision loss to reflect the degree of difference between the 4DGS model output and the depth supervision information. The larger the depth error, the larger the constructed geometric supervision loss; the smaller the depth error, the smaller the constructed geometric supervision loss. Through this process, the depth supervision information can be transformed into a loss constraint term that can directly participate in the 4DGS model training process.

[0102] This embodiment allows for the construction of a corresponding geometric supervision loss based on the difference between the predicted depth information and the depth supervision information output by the 4DGS model. This enables the 4DGS model to optimize its parameters based on the depth relationships and geometric structure information of the target scene during training.

[0103] In step S240, the 4DGS model is trained based on geometric supervision loss and image reconstruction loss.

[0104] Model training refers to the process of adjusting the parameters of a 4DGS model according to a preset optimization objective. Geometric supervision loss serves as one of the constraints in the model parameter update process. Model parameters can include scene representation parameters used to characterize the spatial distribution relationships and appearance attributes of the target scene. By introducing geometric supervision loss into the training process of the 4DGS model, the model can not only focus on the expression of scene appearance information during parameter optimization, but also take into account the spatial depth relationships and geometric structure information in the target scene.

[0105] In practical implementation, geometric supervision loss can be used as the optimization basis in the training process of the 4DGS model to iteratively update the parameters of the 4DGS model.

[0106] For example, based on the difference between the model output reflected by the geometric supervision loss and the depth supervision information, parameters related to scene spatial location, spatial distribution range, and scene representation attributes can be adjusted so that the updated model output gradually approximates the scene geometry represented by the depth supervision information. In other words, during training, the 4DGS model's representation of the target scene's spatial structure can be continuously corrected based on changes in the geometric supervision loss to reduce the difference between the model output and the depth supervision information.

[0107] As the training process continues, the scene representation parameters in the 4DGS model can gradually converge, making the model's expression of the spatial structure features of the target scene more in line with the supervision requirements, and forming a scene representation result that is adapted to the target scene.

[0108] Understandably, the inclusion of geometrically supervised loss in the training process essentially involves further influencing the parameter optimization of the 4DGS model with the scene's geometric structure information reflected by the LiDAR point cloud data. This allows the model training to move beyond solely relying on the image imaging results and introduce constraints on spatial structure during the training phase. Through this process, the 4DGS model can achieve a better representation of scene geometry after training.

[0109] In some example implementations, in order to take into account both the appearance and geometric information of the target scene during the training process of the 4DGS model, an image reconstruction loss corresponding to the image data can be further introduced on the basis of training the 4DGS model based on geometric supervision loss, so as to jointly constrain the 4DGS model.

[0110] First, image data of the target scene is acquired. The image data can be a single frame or multiple frames acquired from the target scene, used to characterize the appearance information of the target scene from a corresponding viewpoint. The image data can originate from an image acquisition device working in conjunction with a LiDAR, or from a pre-stored dataset; this disclosure does not limit its source.

[0111] After obtaining the image data of the target scene, the image data can be input into the 4DGS model. The 4DGS model can perform scene representation and image generation processing based on the image data, thereby outputting a rendered image corresponding to the image data. The rendered image refers to the image result generated by the 4DGS model based on the current model parameters, which reflects the model's representation of the appearance information of the target scene during the current training phase.

[0112] Furthermore, the image reconstruction loss can be determined based on the rendered image and the image data. The image reconstruction loss refers to a loss term that characterizes the degree of difference between the rendered image and the image data, reflecting the deviation between the current output of the 4DGS model and the real image of the target scene. For example, the image reconstruction loss can be determined by calculating the difference between the image content at corresponding locations in the rendered image and the image data; alternatively, it can be determined based on the difference between the overall image results, without limitation in this disclosure.

[0113] After obtaining the geometric supervision loss and image reconstruction loss, the 4DGS model can be further trained based on the geometric supervision loss and image reconstruction loss.

[0114] Specifically, both geometric supervision loss and image reconstruction loss can be used as optimization criteria during model training to iteratively update the parameters in the 4DGS model. This allows for constraints on both the appearance representation of the target scene using image reconstruction loss and the spatial geometric structure representation of the target scene using geometric supervision loss, ensuring that the model training process simultaneously considers both the appearance and geometric information of the target scene. As training continues, the 4DGS model can gradually develop a scene representation that adapts to the target scene.

[0115] In some example implementations, in order to further constrain the scene representation in the 4DGS model and make the model take into account the scene appearance representation, geometric structure representation and Gaussian distribution compactness during training, Gaussian sparsity loss can be further introduced on the basis of training the 4DGS model based on geometric supervision loss and image reconstruction loss.

[0116] Specifically, firstly, the Gaussian parameters in the 4DGS model are obtained. These Gaussian parameters characterize the state information of each Gaussian distribution in the 4DGS model and reflect how the current model represents the target scene.

[0117] After obtaining the Gaussian parameters, the Gaussian sparsity loss can be determined based on these parameters. The Gaussian sparsity loss is a loss term constructed based on the distribution of the Gaussian parameters and is related to the sparsity of the Gaussian distribution. It can characterize the compactness or redundancy of the Gaussian distribution in the current 4DGS model.

[0118] For example, the Gaussian sparsity loss can be determined based on the quantity distribution, spatial distribution, or other parameter characteristics related to sparsity of the Gaussian parameters, but this disclosure does not limit this.

[0119] Furthermore, the target loss can be determined based on geometric supervision loss, image reconstruction loss, and Gaussian sparsity loss. For example, combining geometric supervision loss, image reconstruction loss, and Gaussian sparsity loss can form the target loss for model training, enabling the target loss to comprehensively represent the optimization needs of the 4DGS model in terms of geometric structure representation, image appearance representation, and Gaussian sparsity distribution.

[0120] For example, the target loss can be expressed as: Loss = Lrgb + λLdepth + Lsparse Where Lrgb represents the image reconstruction loss, Ldepth represents the geometric supervision loss, Lsparse represents the Gaussian sparsity loss, and λ represents the coefficient used to balance the weights of the geometric supervision loss.

[0121] Finally, the parameters of the 4DGS model can be updated based on the target loss. For example, an iterative optimization approach can be used to adjust the parameters of the 4DGS model according to the changes in the target loss. This ensures that the model is simultaneously subject to geometric supervision, image reconstruction constraints, and Gaussian sparsity constraints during training, thereby optimizing both image reconstruction quality and geometric accuracy. As training continues, the target loss gradually decreases, and the parameters of the 4DGS model gradually converge, resulting in a scene representation that is adapted to the target scene.

[0122] This embodiment of the disclosure introduces LiDAR point cloud data into the 4DGS model training process, enabling the information used for model training to include spatial geometric information of the target scene in addition to image appearance information. Simultaneously, by generating depth supervision information through LiDAR point cloud projection and constructing geometric supervision loss based on this depth supervision information, the model parameter update process is further constrained by spatial depth relationships and geometric structures while focusing on image reconstruction. This results in the scene representation formed by Gaussian field parameters being closer to the actual spatial structure of the target scene, which helps to improve the spatial structural accuracy of the 3D scene reconstruction results and reduce the impact of depth drift and structural distortion.

[0123] This disclosure also provides an exemplary implementation of a three-dimensional scene reconstruction method, referencing... Figure 5 As shown, the method may include the following steps S510 and S520: Step S510: Obtain image data of the target scene.

[0124] Image data of the target scene can be a single frame or multiple frames acquired by an image acquisition device, used to characterize the appearance information of the target scene from a corresponding viewpoint. For example, image data of the target scene can be acquired in real time by a camera, video camera, or other image acquisition device, or image data corresponding to the target scene can be read directly from a pre-stored dataset, cache unit, or storage device.

[0125] Step S520: Input the image data into the pre-trained 4DGS model, and reconstruct the target scene based on the 4DGS model to obtain a three-dimensional scene model of the target scene.

[0126] The pre-trained 4DGS model is obtained according to the 4DGS model training method in another example embodiment of this disclosure. Since the 4DGS model has already learned the appearance information and spatial structure information of the target scene during the training phase, in the application phase, image data can be input into the pre-trained 4DGS model, so that the 4DGS model can perform scene representation and reconstruction processing on the target scene based on the input image data, thereby outputting a three-dimensional scene model corresponding to the target scene.

[0127] For example, after inputting image data into a pre-trained 4DGS model, the 4DGS model can recover and represent the spatial structure and appearance information of the target scene based on the appearance content of the target scene represented in the image data, combined with the parameters already learned in the model. Specifically, the 4DGS model can use the scene representation results formed during the training phase to parse the scene content from the viewpoint corresponding to the input image data, and generate a three-dimensional representation result of the target scene based on the converged parameters in the model, thereby obtaining a three-dimensional scene model of the target scene.

[0128] In some implementations, the 3D scene model may include spatial structure information and corresponding appearance information of each region in the target scene, thereby characterizing the 3D spatial distribution of the target scene. After obtaining the 3D scene model, it can be further displayed, stored, transmitted, or subjected to subsequent analysis and processing, which is not limited in this disclosure.

[0129] By executing steps S510 and S520, the target scene can be reconstructed based on the pre-trained 4DGS model, resulting in a 3D scene model corresponding to the target scene. Since the pre-trained 4DGS model has already undergone parameter optimization incorporating the spatial geometric information of the target scene during the training phase, the resulting 3D scene model, when reconstructing a 3D scene based on image data during the application phase, can more closely approximate the actual spatial structure of the target scene, thereby improving the accuracy of the 3D scene reconstruction results.

[0130] This example implementation also provides a 4DGS model training apparatus. (See reference...) Figure 6 As shown, the device may include a point cloud data acquisition module 610, a supervision information generation module 620, a supervision loss construction module 630, and a 4DGS model training module 640, wherein: The point cloud data acquisition module 610 is used to acquire LiDAR point cloud data of the target scene; The supervision information generation module 620 is used to generate depth supervision information based on the depth information obtained by projecting the lidar point cloud data into the point cloud. The supervised loss construction module 630 is used to construct the geometric supervised loss of the 4DGS model based on deep supervised information; The 4DGS model training module 640 is used to train the 4DGS model based on geometric supervision loss and image reconstruction loss.

[0131] The specific details of each module of the above-mentioned 4DGS model training device have been described in detail in the corresponding 4DGS model training method, so they will not be repeated here.

[0132] This example embodiment also provides a three-dimensional scene reconstruction apparatus. (See reference...) Figure 7 As shown, the device may include an image data acquisition module 710 and a 3D scene reconstruction module 720, wherein: Image data acquisition module 710 is used to acquire image data of the target scene; The 3D scene reconstruction module 720 is used to input image data into a pre-trained 4DGS model, reconstruct the target scene based on the 4DGS model, and obtain a 3D scene model of the target scene. The 4DGS model is obtained according to the 4DGS model training method in another exemplary embodiment of this disclosure.

[0133] The specific details of each module of the aforementioned 3D scene reconstruction device have been described in detail in the corresponding 3D scene reconstruction methods, so they will not be repeated here.

[0134] Exemplary embodiments of this disclosure also provide a computer-readable storage medium having a program product stored thereon capable of implementing the methods described above in this specification. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on an electronic device, causes the electronic device to perform the steps described in the "Exemplary Methods" section of this specification according to the various exemplary embodiments of this disclosure.

[0135] The program product may be a portable compact disc read-only memory (CD-ROM) containing program code and may run on an electronic device, such as a personal computer. However, the program product disclosed herein is not limited thereto. In this document, a readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0136] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0137] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0138] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0139] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C#, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0140] Furthermore, exemplary embodiments of this disclosure also provide an electronic device capable of implementing the above-described 4DGS model training method.

[0141] The following reference Figure 8 To describe an electronic device 800 according to such an embodiment of the present disclosure. Figure 8 The electronic device 800 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0142] like Figure 8 As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810, at least one storage unit 820, a bus 830 connecting different system components (including storage unit 820 and processing unit 810), and a display unit 840.

[0143] The storage unit 820 stores program code that can be executed by the processing unit 810, causing the processing unit 810 to perform the steps described in the "Exemplary Methods" section above, according to various exemplary embodiments of this disclosure. For example, the processing unit 810 can perform method steps in exemplary embodiments of this disclosure.

[0144] Storage unit 820 may include readable media in the form of volatile storage units, such as random access memory (RAM) 821 and / or cache memory (Cache) 822, and may further include read-only memory (ROM) 823.

[0145] The storage unit 820 may also include a program / utility 824 having a set (at least one) of program modules 825, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0146] Bus 830 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0147] Electronic device 800 can also communicate with one or more external devices 870 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0148] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0149] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0150] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0151] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A 4DGS model training method, characterized in that, include: Acquire LiDAR point cloud data of the target scene; Depth supervision information is generated based on the depth information obtained by projecting the lidar point cloud data into the point cloud. Based on the aforementioned deep supervision information, the geometric supervision loss of the 4DGS model is constructed. The 4DGS model is trained based on the geometric supervision loss and image reconstruction loss.

2. The 4DGS model training method according to claim 1, characterized in that, The generation of depth supervision information based on the depth information obtained by projecting the lidar point cloud data includes: Obtain the camera parameters corresponding to the lidar point cloud data; Based on the camera parameters, the lidar point cloud data is projected onto the camera image plane corresponding to the camera parameters to obtain the depth information of each pixel position in the camera image plane; The depth supervision information is generated based on the depth information of each pixel position.

3. The 4DGS model training method according to claim 2, characterized in that, The step of projecting the lidar point cloud data onto a camera image plane corresponding to the camera parameters based on the camera parameters to obtain depth information of each pixel position in the camera image plane includes: Obtain the three-dimensional coordinate information of each sampling point in the lidar point cloud data; Based on the camera parameters, coordinate transformation and projection mapping are performed on the three-dimensional coordinate information of each sampling point to obtain the projection position and corresponding depth value of each sampling point in the camera image plane. The depth information of each pixel position in the camera image plane is determined based on the projection position of each sampling point in the camera image plane and the corresponding depth value.

4. The 4DGS model training method according to claim 3, characterized in that, The step of performing coordinate transformation and projection mapping on the three-dimensional coordinate information of each sampling point based on the camera parameters to obtain the projection position and corresponding depth value of each sampling point in the camera image plane includes: Based on the extrinsic parameter matrix in the camera parameters, the three-dimensional coordinate information of each sampling point is transformed into the camera coordinate system to obtain the camera coordinate information of each sampling point; Based on the camera coordinate information of each sampling point, determine the depth value corresponding to each sampling point; Based on the intrinsic parameter matrix in the camera parameters, the camera coordinate information of each sampling point is projected onto the camera image plane to obtain the projection position of each sampling point in the camera image plane.

5. The 4DGS model training method according to claim 2, characterized in that, The generation of depth supervision information based on the depth information of each pixel position includes: Based on the depth information of each pixel position, the target pixel position that meets the preset depth condition and the target depth value corresponding to each target pixel position are determined. The depth supervision information is generated based on the target pixel position and the target depth value.

6. The 4DGS model training method according to claim 1, characterized in that, The construction of the geometric supervision loss of the 4DGS model based on the depth supervision information includes: Obtain the predicted depth information output by the 4DGS model from the target's perspective; The predicted depth information is compared with the depth supervision information to determine the depth error; Based on the depth error, the geometric supervision loss of the 4DGS model is constructed.

7. The 4DGS model training method according to claim 1, characterized in that, The training of the 4DGS model based on the geometric supervision loss and image reconstruction loss includes: Acquire image data of the target scene; The image data is input into the 4DGS model to obtain the rendered image; Based on the rendered image and the image data, the image reconstruction loss is determined; The 4DGS model is trained based on the geometric supervision loss and the image reconstruction loss.

8. The 4DGS model training method according to claim 7, characterized in that, The method further includes: Obtain the Gaussian parameters in the 4DGS model; Based on the Gaussian parameters, determine the Gaussian sparsity loss; The target loss is determined based on the geometric supervision loss, the image reconstruction loss, and the Gaussian sparsity loss. The parameters of the 4DGS model are updated based on the target loss.

9. The 4DGS model training method according to claim 1, characterized in that, The method further includes: The lidar point cloud data is subjected to noise point filtering processing to obtain the first point cloud data; Outlier removal is performed on the first point cloud data to obtain the second point cloud data; The second point cloud data is processed using coordinate system one to obtain preprocessed lidar point cloud data, and the depth supervision information is generated based on the preprocessed lidar point cloud data.

10. A method for reconstructing a three-dimensional scene, characterized in that, include: Acquire image data of the target scene; The image data is input into a pre-trained 4DGS model, and the target scene is reconstructed based on the 4DGS model to obtain a three-dimensional scene model of the target scene. The 4DGS model is obtained by the 4DGS model training method according to any one of claims 1 to 9.

Images