A scene optimization method and robot

Abstract

This application provides a scene optimization method and robot. By acquiring the robot's LiDAR point cloud data, inertial measurement unit data, and visual image data, a spatial index structure is constructed based on the LiDAR point cloud data to establish an initial scene map. The current frame visual image is initialized to generate initial Gaussian points, which are then inserted into the initial scene map to obtain a global scene map. The current pose information is then determined based on the LiDAR point cloud data, inertial measurement unit data, and visual image data. The initial Gaussian points in the global scene map are optimized based on the current pose information to obtain an optimized global scene. This application clearly distinguishes between static and dynamic Gaussian points and integrates and manages static and dynamic representations through a unified global scene map representation, reducing the complexity of separate processing. Furthermore, by combining multimodal data fusion to estimate and optimize pose, more stable pose information can be provided, achieving spatiotemporally consistent dynamic scene reconstruction.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a scene optimization method and a robot. Background Technology

[0002] With the rapid development of robotics and autonomous navigation systems, real-time high-precision scene perception and reconstruction has become one of the core research directions in the field of robotics. Among them, real-time scene reconstruction technology is a key support for realizing environmental perception and autonomous decision-making. In application scenarios such as intelligent robots and augmented reality, devices need to accurately model the surrounding environment to support tasks such as localization, navigation, obstacle avoidance, and interaction. However, the large number of dynamic elements in the real world, such as pedestrians, vehicles, and swaying plants, increases the difficulty of reconstruction technology.

[0003] Current mainstream scene reconstruction methods are primarily based on simultaneous localization and mapping (SMR) or neural radiation field (NRF) techniques. However, SMR methods typically assume a static environment. When moving objects exist in the scene, these dynamic elements are incorrectly included in the map building process, leading to ghosting and other issues that reduce map quality and positioning accuracy. NRF techniques, on the other hand, usually require long training times and significant computational resources, making them unsuitable for real-time applications. Furthermore, for dynamic scenes, the motion structure often needs to be pre-defined, resulting in high computational complexity. Additionally, in practice, mainstream scene reconstruction techniques often use traditional map representations for static areas while tracking or ignoring dynamic areas separately. This not only increases system complexity but also disrupts the overall consistency of the scene, making it difficult to support spatiotemporally consistent dynamic analysis. Summary of the Invention

[0004] The purpose of this application is to provide a scene optimization method and a robot to solve the above-mentioned technical problems.

[0005] This application provides a scene optimization method, which includes: acquiring LiDAR point cloud data, inertial measurement unit data, and visual image data of a robot; constructing a spatial index structure based on the LiDAR point cloud data, and establishing an initial scene map based on the spatial index structure; performing Gaussian point initialization processing on the visual image of the current frame in the visual image data to generate initial Gaussian points, and inserting the initial Gaussian points into the initial scene map to obtain a global scene map, wherein the initial Gaussian points include static attribute Gaussian points and dynamic attribute Gaussian points; determining the current pose information based on the LiDAR point cloud data, inertial measurement unit data, and visual image data through a filtering algorithm; and optimizing the initial Gaussian points in the global scene map according to the current pose information to obtain an optimized global scene. Beneficial effects include clearly distinguishing between static and dynamic Gaussian points, integrating and managing static and dynamic representations through a unified global scene map representation, reducing the complexity of separate processing, and combining multimodal data fusion to estimate and optimize pose, providing more stable pose information and achieving spatiotemporally consistent dynamic scene reconstruction.

[0006] In one embodiment of this application, constructing a spatial index structure based on the LiDAR point cloud data and establishing an initial scene map based on the spatial index structure includes: determining a processable three-dimensional space according to the three-dimensional coordinates of each point cloud in the LiDAR point cloud data; dividing the processable three-dimensional space into multiple octree nodes using spatial hash indexes according to a preset spatial resolution, generating a spatial index structure, where each octree node corresponds to a spatial sub-region, and the spatial hash index uses the position and size of the spatial sub-region corresponding to the octree node as the key; associating time parameters with the octree nodes in the spatial index structure, and using the associated spatial index structure as the initial scene map. The beneficial effects include efficient management of large-scale point cloud data, enabling rapid spatial queries, and by associating time parameters with the octree nodes, the initial scene map possesses time dimension information, improving the map construction's adaptability to dynamic environments.

[0007] In one embodiment of this application, Gaussian point initialization processing is performed on the visual image of the current frame in the visual image data to generate initialized Gaussian points. This includes: initializing the visual image of the current frame with basic parameters to obtain candidate Gaussian points with basic parameter descriptions, wherein the basic parameters include position parameters and physical attribute parameters; and assigning static or dynamic attributes to each candidate Gaussian point according to image semantics and motion information to complete dynamic-static layered initialization, thereby obtaining the initialized Gaussian point. The beneficial effects include refined basic parameter initialization of Gaussian points, ensuring accurate initial position and appearance descriptions of Gaussian points in space, and dynamic-static layered initialization of Gaussian points, which improves the accuracy of the initial scene map and the ability to model dynamic environments.

[0008] In one embodiment of this application, the basic parameter initialization of the visual image of the current frame to obtain candidate Gaussian points with basic parameter descriptions includes: in the initialization of position parameters, the visual image of the current frame is divided based on the coverage state of the LiDAR point cloud; for areas covered by LiDAR point cloud, the three-dimensional position parameters of the initialized Gaussian points are determined based on the points mapped from the LiDAR point cloud data to the visual image of the current frame; for areas not covered by LiDAR point cloud, the three-dimensional position parameters of the initialized Gaussian points are determined by performing feature point matching and triangulation measurement on the visual image of the current frame and the visual image of the previous frame, wherein the visual image of the previous frame is the visual image whose time series is located in the frame preceding the current frame in the frame sequence; in the initialization of physical attribute parameters, based on the position parameters of the initialized Gaussian points, neighboring Gaussian points are retrieved in the global scene map, and values ​​are assigned based on the physical attribute parameters corresponding to the neighboring Gaussian points, wherein the physical attribute parameters include at least one of color parameters, scale parameters, and opacity parameters. Beneficial effects include dividing the visual image of the current frame based on the coverage state of the LiDAR point cloud, reducing the impact of limited accuracy in complex scenes, and accelerating the convergence of Gaussian point parameters by retrieving and assigning neighboring Gaussian points, thereby improving the efficiency and accuracy of scene map construction.

[0009] In one embodiment of this application, assigning static or dynamic attributes to each candidate Gaussian point based on image semantics and motion information includes: performing semantic segmentation on the visual image of the current frame, determining static and dynamic regions in the image based on the category of the target object in the image; calculating optical flow on the visual image of the current frame and the visual image of the previous frame to obtain the optical flow vector amplitude of each pixel; and dividing the candidate Gaussian point into static attribute Gaussian points and dynamic attribute Gaussian points based on the static region, the dynamic region, and the optical flow vector amplitude. The dynamic attribute Gaussian points include slow dynamic Gaussian points and fast dynamic Gaussian points, both of which include a time parameter. Beneficial effects include the ability to effectively distinguish between static backgrounds, slow-moving objects, and fast-moving objects by combining image semantic information and optical flow, facilitating the adoption of adaptive optimization processing strategies for different types of objects.

[0010] In one embodiment of this application, inserting the initialized Gaussian point into the initial scene map includes: for the static attribute Gaussian point, locating the root voxel within the field of view based on spatial hash index; when the leaf voxel within the root voxel is not filled, initializing a new Gaussian point based on LiDAR point cloud data and visual image data, and inserting the new Gaussian point into the leaf voxel; for the dynamic attribute Gaussian point, inserting the initialized Gaussian point into the corresponding leaf voxel. Beneficial effects include adopting different insertion strategies based on the attributes of the Gaussian point, achieving refined management of the global scene map, and better balancing the level of detail and resource consumption of the map when dealing with complex dynamic environments.

[0011] In one embodiment of this application, determining the current pose information based on the lidar point cloud data, inertial measurement unit (IMU) data, and visual image data using a filtering algorithm includes: extracting lidar point cloud features from the lidar point cloud data, extracting visual image features from the visual image data, and extracting IMU motion features from the IMU data; spatiotemporally aligning the extracted lidar point cloud features, visual image features, and IMU motion features, and performing feature-level fusion to obtain a joint observation vector; using the joint observation vector as input, performing prediction through a filtering algorithm to obtain a state prediction value, the state prediction value including the device's attitude, position, velocity, and sensor bias; and updating the state prediction value based on the joint observation vector, calculating the Kalman gain and correcting the state prediction value to obtain a state estimate value, and determining the state estimate value as the current pose information. Beneficial effects include fusing heterogeneous data, enabling the filtering algorithm to obtain more comprehensive and reliable observation information, and improving the estimation accuracy and stability of the current pose information.

[0012] In one embodiment of this application, optimizing the initialized Gaussian points in the global scene map based on the current pose information to obtain an optimized global scene includes: optimizing the static and dynamic attribute Gaussian points in the global scene map based on a preset loss function and the current pose information, respectively, to obtain statically optimized Gaussian points and dynamically optimized Gaussian points; and performing rendering processing based on the statically optimized Gaussian points and dynamically optimized Gaussian points to generate an optimized global scene. The beneficial effects include classifying and optimizing the initialized Gaussian points based on the loss function, achieving a targeted strategy for global optimization.

[0013] In one embodiment of this application, the preset loss function includes a photometric loss function, a rigidity-preserving loss function, and a temporal smoothing loss function. Based on the preset loss function and according to the current pose information, the optimization of the static attribute Gaussian points and dynamic attribute Gaussian points initialized in the global scene map includes: for the static attribute Gaussian points, optimizing the position, color, and opacity of the static attribute Gaussian points based on the position information in the current pose information using multi-view photometric consistency constraints and the photometric loss function, and updating the static attribute Gaussian points. The multi-view photometric consistency constraint refers to minimizing the difference between the projected photometric value and the actual image under different viewpoints; for the dynamic attribute Gaussian points, constructing a deformation model based on the current pose information using a four-dimensional hash grid structure and a multilayer perceptron model; predicting the position and pose changes of the dynamic attribute Gaussian points at different times based on the deformation model, the rigidity-preserving loss function, and the temporal smoothing loss function, and updating the dynamic attribute Gaussian points. The four-dimensional hash grid structure consists of a spatial hash grid and a spatial-temporal hash grid. Beneficial effects include ensuring more accurate and stable geometric and photometric reconstruction of static scene parts, accurately capturing and predicting complex movements and deformations of dynamic objects, avoiding unreasonable deformations and abrupt changes in movement, and improving the ability to perceive and understand the environment.

[0014] This application also provides a robot, characterized in that the robot includes a memory and a processor, the memory being used to store computer instructions that can be executed on the processor, and the processor being used to implement the scene optimization method as described in any one of the above embodiments when executing the computer instructions.

[0015] The beneficial effects of this application are as follows: This application provides a scene optimization method and robot. By acquiring the robot's LiDAR point cloud data, inertial measurement unit data, and visual image data, a spatial index structure is constructed based on the LiDAR point cloud data, and an initial scene map is established. The current frame visual image is initialized to generate initial Gaussian points, which are then inserted into the initial scene map to obtain a global scene map. The current pose information is then determined based on the LiDAR point cloud data, inertial measurement unit data, and visual image data. Based on the current pose information, the initial Gaussian points in the global scene map are optimized to obtain an optimized global scene. This application introduces initial Gaussian points... The system initializes Gaussian points and clearly distinguishes between static and dynamic attribute Gaussian points. Then, through a unified global scene map representation, it achieves integrated management of static and dynamic elements, reducing the coordination complexity between the static map and the dynamic tracking module. Secondly, by using a multimodal data fusion strategy combined with a filtering algorithm to determine the current pose information, it provides more stable pose information, further supporting rapid optimization of subsequent Gaussian points. Moreover, by optimizing the initialized Gaussian points in the global scene map based on the current pose information, it generates a spatiotemporally consistent dynamic scene representation, improving the accuracy and stability of dynamic scene reconstruction while ensuring real-time performance.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0018] Figure 1 This is a schematic diagram illustrating an exemplary system architecture as shown in an exemplary embodiment of this application; Figure 2 This is a flowchart illustrating a scene optimization method in an exemplary embodiment of this application; Figure 3 This is a schematic diagram illustrating a specific scenario optimization method architecture, as shown in an exemplary embodiment of this application. Figure 4 This is a schematic diagram of a scene optimization device shown in an exemplary embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computer system for an electronic device, as illustrated in an exemplary embodiment of this application. Detailed Implementation

[0019] The embodiments of this application will be described below with reference to the accompanying drawings and specific examples. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and are not intended to limit the scope of protection of this application.

[0020] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the illustrations only show the components related to this application and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0021] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0022] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an exemplary system architecture as shown in an exemplary embodiment of this application.

[0023] Reference Figure 1 As shown, the system architecture may include a robot 110 and a computer device 120. The computer device 120 acquires LiDAR point cloud data, inertial measurement unit data, and visual image data from the robot 110. Based on the LiDAR point cloud data, it constructs a spatial index structure and establishes an initial scene map. It initializes the visual image of the current frame to generate initial Gaussian points, which are then inserted into the initial scene map to obtain a global scene map. Next, based on the LiDAR point cloud data, inertial measurement unit data, and visual image data, it determines the current pose information of the robot 110. Based on the current pose information, it optimizes the initial Gaussian points in the global scene map to obtain an optimized global scene. The aforementioned computer device 120 refers to a computing power support terminal device used to carry the program and implementation environment for the scene optimization method, including but not limited to microcomputers, tablet devices, embedded computers, server clusters, and cloud servers.

[0024] The aforementioned robot 110 includes, but is not limited to, a bipedal robot, which at least includes a robot body, multi-source sensors, and a data transmission module. The multi-source sensors are used to acquire multi-source sensor data such as LiDAR point cloud data, inertial measurement unit data, and visual image data, and include, but are not limited to, fisheye cameras, wide-angle cameras, LiDAR, depth cameras, and inertial measurement units. The data transmission module is used to transmit the aforementioned multi-source sensor data to a computer device 120. In a practical implementation environment, the aforementioned robot 110 is a bipedal intelligent robot, equipped with one fisheye camera and two wide-angle cameras for visual sensing. The fisheye camera is 1920mm. With a resolution of 1536, a frame rate of 30 frames per second, and a field of view of 196°×154°, it is positioned behind the head of robot 110; the two wide-angle cameras also have a resolution of 1920. With a resolution of 1536, a frame rate of 30 frames per second, and a field of view of 118° × 92°, the cameras are positioned at the eye locations on the head of robot 110. For depth perception, it is equipped with two depth cameras with a resolution of 1920. The camera operates at 800 frames per second (fps), with a field of view of 94° × 68°. It is positioned on the upper chest of robot 110, near the shoulders on both sides. For LiDAR, two LiDARs are used, each with a field of view of 120° × 190°, a maximum ranging distance of 75 meters, and an output point cloud of 260,000 per second. They are positioned on either side of the chest of robot 110. For motion sensing, one inertial measurement unit (IMU) is provided. This IMU has an attitude measurement accuracy of 0.2° for roll and pitch, 1° for yaw, and zero-bias stability of 10 micrograms horizontally and 15 micrograms vertically. The above configuration of the fisheye camera, wide-angle camera, LiDAR, depth camera, and inertial unit is only one possible implementation in this application. The model selection, parameter requirements, and configuration quantity can be modified according to the actual needs of the implementation process, and are not specifically limited here.

[0025] This technical solution can be further discussed and analyzed in multiple steps; for specific steps, please refer to [link / reference needed]. Figure 2 , Figure 2 This is a flowchart illustrating a scene optimization method in an exemplary embodiment of this application. This scene optimization method can be executed in implementation environments supported by various operating systems, and no specific limitation is made to the implementation environment herein. (Refer to...) Figure 2 As shown, the flowchart of this scenario optimization method includes at least steps S210 to S250, which are described in detail below: In step S210, the robot's lidar point cloud data, inertial measurement unit data, and visual image data are acquired.

[0026] In one embodiment of this application, the lidar point cloud data is a set of three-dimensional spatial points in the environment acquired by a lidar sensor, each point including at least three-dimensional coordinate information, which can be used to perceive the geometric structure of the environment; the inertial measurement unit data refers to data about the motion state of the device collected by the inertial measurement unit (IMU), including acceleration and angular velocity information, which is used to estimate the attitude and motion of the device; the visual image data is a sequence of two-dimensional images acquired by a camera, containing various texture, color and semantic information, which can be used to identify objects and features in the environment.

[0027] In one embodiment of this application, the robot is equipped with independent LiDAR sensors, inertial measurement unit (IMU) sensors, and a vision camera, each collecting its own type of data. The LiDAR sensor periodically emits laser light and receives reflected signals, generating LiDAR point cloud data containing distance and intensity information. The IMU sensor measures IMU data in real time, including the robot's angular velocity and acceleration. The vision camera captures a sequence of two-dimensional images of the environment at a specific frame rate, forming visual image data. These data streams are transmitted to a processing unit and initially timestamped. In other feasible environments, the robot may also employ an integrated sensor module. This module integrates multiple sensors and provides a unified data interface. Through this interface, pre-packaged multimodal raw data can be output. This multimodal raw data is then parsed to separate the LiDAR point cloud data, IMU data, and visual image data.

[0028] In one embodiment of this application, after acquiring lidar point cloud data, inertial measurement unit data, and visual image data, a data preprocessing step is included to improve the quality of these data. For the lidar point cloud data, denoising is first performed to remove outliers caused by environmental interference or sensor noise. Then, voxel filtering is applied to the point cloud data. By dividing the three-dimensional space into a voxel grid, representative point clouds are retained within each voxel, thereby reducing the density of the point cloud data, decreasing redundant points, and effectively compressing the data volume while preserving geometric features. To further improve point cloud quality, an adaptive filtering algorithm can be used instead of voxel filtering. This algorithm can better preserve the edge and detail information of the point cloud while removing noise, and can be specifically configured according to the data acquisition requirements.

[0029] For inertial measurement unit (IMU) data, the collected acceleration and angular velocity data are first integrated to obtain the device's attitude changes and trajectory over time. The integrated results are then filtered using algorithms such as Kalman filtering to suppress noise accumulation and obtain more accurate attitude and motion information. Further optimization algorithms are applied to the IMU data integration and filtering to improve the estimation accuracy of attitude and motion information. For visual image data, the original images are first corrected based on pre-calibrated camera intrinsic parameters and distortion coefficients. The distortion-corrected images are then converted to grayscale. To further improve image quality, deep image processing algorithms such as convolutional neural networks can be used for super-resolution reconstruction, contrast enhancement, and sharpness improvement.

[0030] In step S220, a spatial index structure is constructed based on the lidar point cloud data, and an initial scene map is established based on the spatial index structure.

[0031] In one embodiment of this application, a processable three-dimensional space is determined based on the three-dimensional coordinates of each point cloud in the lidar point cloud data; the processable three-dimensional space is divided into multiple octree nodes using spatial hash indexes according to a preset spatial resolution, and a spatial index structure is generated, with each octree node corresponding to a spatial sub-region, and the spatial hash index using the position and size of the spatial sub-region corresponding to the octree node as the key value; time parameters are associated with the octree nodes in the spatial index structure, and the associated spatial index structure is used as the initial scene map.

[0032] In one embodiment of this application, when determining a processable 3D space, the minimum axis-aligned bounding box encompassing all point cloud data can be determined by analyzing the maximum and minimum X, Y, and Z coordinate values ​​of all received LiDAR point cloud data. The processable 3D space is then divided into multiple octree nodes using spatial hash indexing. When generating the spatial index structure, each node in the octree structure represents a cubic space, recursively subdivided into eight child nodes until a preset spatial resolution or a threshold number of point clouds is reached. The spatial hash index encodes the spatial coordinates and hierarchical information of the octree nodes into hash keys.

[0033] In other feasible environments, this also includes employing dynamic octrees and adaptively adjusting the node subdivision based on the density and distribution of the LiDAR point cloud data. Spatial hash indexes can utilize Z-order curves to map three-dimensional coordinates to one-dimensional hash values, improving cache locality. Z-order curves are a technique for mapping multi-dimensional data to one-dimensional space, maintaining data locality so that nearby multi-dimensional points are also adjacent in one-dimensional space.

[0034] In one embodiment of this application, associating time parameters with octree nodes in the spatial index structure means storing the timestamp of the last update of each octree node. By associating time parameters with octree nodes, the initial scene map has time dimension information, which makes it easier to distinguish between static and dynamic environmental elements.

[0035] In the embodiments of this application, the space is efficiently divided into multiple octree nodes using a preset spatial resolution, and combined with a spatial hash index, hierarchical management of large-scale point cloud data is realized, improving the efficiency of spatial data query and enabling the insertion and management of Gaussian points to quickly locate the corresponding spatial sub-regions; secondly, each octree node is associated with a time parameter, so that the initial scene map not only includes spatial information but also incorporates the time dimension, which can better adapt to changes in the dynamic environment.

[0036] In the embodiments of this application, Gaussian point data management and retrieval are performed using a combination of spatial hash indexing and a sliding window mechanism. For retrieval, the spatial hash index's rapid positioning capability is used to query Gaussian points. When data within a specific spatial region needs to be accessed, the spatial hash index of that region is calculated. This index is then used to quickly locate the corresponding root voxel in the global scene map, allowing access to the leaf voxels contained within the root voxel. Based on this, all Gaussian points within a specified spatial region can be efficiently retrieved, providing timely and accurate data support for real-time device positioning, incremental updates of the global scene map, and matching and association with currently observed data. Furthermore, the management and retrieval of Gaussian points are implemented using a sliding window mechanism. An incremental maintenance method can be used to manage Gaussian points within a sliding window. Management operations are performed primarily within the sliding window, utilizing the Gaussian point data already loaded in the sliding window to complete the processing task of the current frame. If the area to be processed is not within the coverage of the current sliding window, the Gaussian points related to the target area are loaded from the global scene map into the sliding window using the spatial index structure of the global scene map, ensuring that the current processing task can access complete data. By dynamically loading and unloading based on a sliding window mechanism, memory usage can be reduced and data processing efficiency can be improved while ensuring data access integrity.

[0037] In step S230, Gaussian point initialization processing is performed on the visual image of the current frame in the visual image data to generate initial Gaussian points, and the initial Gaussian points are inserted into the initial scene map to obtain the global scene map.

[0038] During local map initialization, the visual image of the current frame needs to be partitioned into multiple image regions. For each image region, the luminance loss value between the rendered image and the real image in that region is obtained. Based on the relationship between this luminance loss value and a preset threshold, a differentiated Gaussian point initialization strategy is adopted. When the luminance loss of an image region is less than the preset threshold, it indicates that the rendering quality of that region is high and the scene representation is relatively accurate. Based on the screen space gradient information and the distribution ratio of dynamic attribute Gaussian points in neighboring points, some static attribute Gaussian points are adaptively transformed to better adapt to local changes in the current scene. At the same time, the Gaussian points of the fast dynamic layer are dynamically updated according to the time parameter so that the fast dynamic layer Gaussian points can reflect the latest motion state of objects at that time point. When the luminance loss of an image region is greater than or equal to the preset threshold, it indicates that the rendering quality of that region is low. After performing the adaptive transformation of static attribute Gaussian points based on the screen space gradient and the distribution ratio of dynamic points in neighboring points, and the dynamic update of the fast dynamic layer Gaussian points based on the time parameter, it is also necessary to add Gaussian points to that image region to supplement the detailed information of the scene representation.

[0039] In one embodiment of this application, the visual image of the current frame is initialized with basic parameters to obtain candidate Gaussian points with basic parameter descriptions, including position parameters and physical attribute parameters; for each candidate Gaussian point, static or dynamic attributes are assigned according to image semantics and motion information to complete the static-dynamic layer initialization and obtain initialized Gaussian points.

[0040] In one embodiment of this application, during the initialization of position parameters, the visual image of the current frame is divided based on the coverage state of the LiDAR point cloud; for areas covered by LiDAR point cloud, the three-dimensional position parameters of the initial Gaussian points are determined based on the points mapped from the LiDAR point cloud data to the visual image of the current frame; for areas not covered by LiDAR point cloud, the three-dimensional position parameters of the initial Gaussian points are determined by performing feature point matching and triangulation measurement on the visual image of the current frame and the visual image of the previous frame, wherein the visual image of the previous frame is the visual image in the frame sequence whose time series is the frame preceding the current frame.

[0041] In one embodiment of this application, the visual image of the current frame is divided based on the coverage status of the LiDAR point cloud. This is achieved by adopting a differentiated initialization strategy based on the data reliability of different regions. This includes dividing the regions in the visual image into different processing categories based on whether there is corresponding LiDAR point cloud data coverage. This can be done by projecting the LiDAR point cloud data onto the visual image plane of the current frame and marking the pixel regions covered by the point cloud. For areas covered by LiDAR point clouds, high-precision LiDAR point cloud data is used to initialize the 3D positions of Gaussian points. This involves projecting the LiDAR point cloud data onto the visual image plane using camera intrinsic and extrinsic parameters to obtain the 2D coordinates of each point cloud on the image. For areas covered by point clouds in the image, the 3D coordinates of these point clouds are interpolated and used as the initial 3D positions of the corresponding Gaussian points. For areas not covered by LiDAR point clouds, feature point matching and triangulation measurements are performed on the visual image of the current frame and the visual image of the previous frame to determine the 3D position parameters for initializing Gaussian points. This includes detecting and matching feature points between the visual image of the current frame and the visual image of the previous frame, and then using the camera pose and the matched feature point pairs, calculating the positions of these feature points in 3D space using the triangulation principle, and using these 3D positions as the initial 3D positions of the corresponding Gaussian points.

[0042] In one embodiment of this application, during the initialization of physical attribute parameters, neighboring Gaussian points are retrieved in the global scene map based on the position parameters of the Gaussian points obtained during initialization, and the physical attribute parameters corresponding to the neighboring Gaussian points are assigned values. The physical attribute parameters include at least one of color parameters, scale parameters, and opacity parameters.

[0043] In the initialization of physical property parameters, based on the position parameters of the Gaussian point obtained during initialization, neighboring Gaussian points are searched in the global scene map, and the physical property parameters corresponding to the neighboring Gaussian points are assigned values ​​to provide reasonable physical properties for the newly initialized Gaussian point, avoiding learning from scratch, thereby accelerating convergence and improving the initialization quality. This includes using the 3D position parameters of the newly initialized Gaussian point to perform a spatial nearest neighbor search in the constructed global scene map to find several Gaussian points with the closest distance; then, the physical property parameters of these neighboring Gaussian points are weighted and averaged as the initial physical properties of the new Gaussian point.

[0044] In the actual implementation of this embodiment, when performing Gaussian point initialization processing on the visual image of the current frame, the image region is divided into a laser-covered region and a non-laser-covered region based on whether there is laser point cloud coverage in the image. For the laser-covered region, Gaussian points are initialized based on the points mapped from the laser point cloud data to the current frame visual image. This includes using the three-dimensional position of the points in the laser point cloud data, combined with pre-calibrated sensor extrinsic parameters and camera intrinsic parameters, to project the laser points onto the image plane, determine their corresponding pixel positions in the image, and use these three-dimensional positions as the position parameters for initializing the Gaussian points. For the non-laser-covered region, since there is a lack of direct laser depth information, the three-dimensional position of the Gaussian points is initialized using visual feature triangulation. Feature points are extracted from the current frame visual image and the previous frame visual image, and the extracted feature points are matched to determine the matching feature point pairs in the two frames. Then, based on the matching feature point pairs and the pre-calibrated camera parameters, the three-dimensional spatial position corresponding to the feature points is calculated through triangulation measurement, and this three-dimensional position is used as the position parameters for initializing the Gaussian points.

[0045] After initializing the position parameters, the system further assigns values ​​to other physical attribute parameters of the initialized Gaussian point, including at least one of the color, scale, and opacity parameters. The assignment method involves searching for neighboring Gaussian points in the global scene map based on their spatial proximity, according to the 3D position of the initialized Gaussian point, and assigning the physical attribute parameters corresponding to the searched neighboring Gaussian points to the currently initialized Gaussian point as its initial physical attribute parameters.

[0046] In the embodiments of this application, when initializing the basic parameters of the visual image of the current frame in the visual image data, the visual image of the current frame is divided based on the coverage state of the LiDAR point cloud, and the three-dimensional position parameters of the Gaussian points are initialized by LiDAR data mapping or visual feature point triangulation measurement. This solves the problem of insufficient coverage of single sensor data in complex scenes. Furthermore, by searching for neighboring Gaussian points in the global scene map and assigning values ​​based on their physical attribute parameters, the newly generated Gaussian points can inherit the existing and optimized physical attribute information in the environment, reducing the uncertainty caused by random initialization and accelerating the convergence of Gaussian point parameters.

[0047] In one embodiment of this application, assigning static or dynamic attributes based on image semantics and motion information includes semantic segmentation of the visual image of the current frame, determining static and dynamic regions in the image based on the category of the target object in the image; performing optical flow calculation on the visual image of the current frame and the visual image of the previous frame to obtain the optical flow vector amplitude of each pixel; and dividing candidate Gaussian points into static attribute Gaussian points and dynamic attribute Gaussian points based on the static region, dynamic region, and optical flow vector amplitude. The dynamic attribute Gaussian points include slow dynamic Gaussian points and fast dynamic Gaussian points, both of which include time parameters.

[0048] The process involves semantic segmentation of the current frame's visual image. Static and dynamic regions are identified based on the category of the target object in the image. Each pixel is categorized into a predefined semantic category, recognizing the boundaries and categories of different objects and identifying which regions may contain dynamic objects and which are static background. Optical flow calculations are performed on the current and previous frames' visual images to obtain the optical flow vector amplitude for each pixel. This quantifies the speed and direction of motion of objects in the image. Optical flow is the motion information of pixel intensity patterns in an image sequence, describing the displacement vector of each pixel between consecutive frames. It directly reflects the dynamics of elements in the scene and is a key basis for distinguishing between static and dynamic attributes. The optical flow vector amplitude can be estimated using the Lucas-Kanade optical flow algorithm to estimate pixel motion. Based on this, the aforementioned region categories and optical flow are combined to perform a final dynamic and static attribute determination on each candidate Gaussian point, thereby achieving refined dynamic and static stratification of Gaussian points. If a candidate Gaussian point is located in a dynamic region determined by semantic segmentation and its optical flow vector amplitude is greater than or equal to a preset amplitude threshold, it is marked as a dynamic attribute Gaussian point; if it is located in a static region or its optical flow amplitude is less than the preset amplitude threshold, it is marked as a static attribute Gaussian point.

[0049] Further subdividing dynamic attribute Gaussian points into slow-dynamic and fast-dynamic categories allows for more precise handling of dynamic objects with varying speeds. In a feasible environment, static Gaussian points, including but not limited to fixed objects such as walls and the ground, are modeled as 3D Gaussian points whose position, color, and scale do not change over time. Slow-dynamic Gaussian points, defined as Gaussian points with an optical flow of less than 10 pixels per frame, include but are not limited to fluttering leaves and walking pedestrians. These are modeled using 4D Gaussian points and include a time-axis scale parameter. This time-axis scale parameter is controlled by an exponential function with a base of the natural constant to adjust the coverage of Gaussian points in the time dimension, enabling the slow-dynamic layer Gaussian points to represent the object's trajectory over a period of time. Fast-dynamic Gaussian points, defined as Gaussian points with an optical flow greater than or equal to 10 pixels per frame, include but are not limited to moving vehicles and fast-moving objects. These are also modeled using 4D Gaussian points and include a time-axis scale parameter. Based on this, the Gaussian points in the fast dynamic layer are also associated with a dynamic deformation model, which is used to track the motion of the Gaussian points in the fast dynamic layer independently. The parameters of the dynamic deformation model include position offset and rotation offset. By predicting the position offset of the Gaussian point relative to the reference position and the rotation offset relative to the reference attitude at different times, the accurate motion expression of the Gaussian points in the fast dynamic layer over time is realized.

[0050] In the embodiments of this application, a coarse-grained division of static and dynamic regions in the scene is obtained through semantic segmentation. At the same time, the optical flow between the current frame and the previous frame is calculated to obtain the optical flow vector amplitude. For each candidate Gaussian point, the semantic region category and its corresponding optical flow vector amplitude are comprehensively judged, so that the dynamic and static layering of Gaussian points is more accurate, and the situation of misidentifying dynamic objects as static objects can be effectively reduced in complex dynamic scenes.

[0051] In one embodiment of this application, for static attribute Gaussian points, the root voxel within the field of view is located based on the spatial hash index. When the leaf voxel in the root voxel is not filled, a new Gaussian point is initialized based on the lidar point cloud data and visual image data, and the new Gaussian point is inserted into the leaf voxel. For dynamic attribute Gaussian points, the initialized Gaussian point is inserted into the corresponding leaf voxel.

[0052] In this context, the root voxel refers to the smallest spatial unit in the octree that is at a higher level and within the current sensor's field of view. It can effectively narrow the search range and improve data processing efficiency. The root voxel can be determined by projecting the sensor's frustum onto the spatial index structure and identifying all intersecting top-level voxels. The leaf voxel, on the other hand, is the smallest and finest-grained spatial unit in the spatial data structure, namely the Gaussian point ontology. In the octree structure, an unfilled leaf voxel means that the number of Gaussian points currently stored in the leaf voxel has not yet reached its preset maximum capacity. In the embodiments of this application, for static attribute Gaussian points, the root voxel within the current field of view is first quickly located using a spatial hash index, thereby limiting the processing range to the relevant area. Then, it is checked whether the leaf voxels under the root voxel are full. If the leaf voxels are not full, new Gaussian points are initialized based on LiDAR point cloud data and visual image data, and these new Gaussian points are inserted into the corresponding leaf voxels, ensuring that the details of the static scene can be gradually enriched. For dynamic attribute Gaussian points, since they are time-varying, they are directly inserted into the corresponding leaf voxels so that their motion parameters can be independently tracked and optimized.

[0053] In one embodiment of this application, after Gaussian point insertion, the method further includes continuously updating the Gaussian point parameters in the global scene map. When updating the global map, voxels still within the current field of view in the previous frame's Gaussian sliding window are first identified. For the Gaussian points contained within these voxels, optimization processing is performed based on the observation data of the current frame. The optimization includes the Gaussian point's position parameters, color parameters, scale parameters, opacity parameters, and the time parameters and deformation offset parameters associated with the dynamic attribute Gaussian points. After optimization, the optimized Gaussian point parameters are copied back to the corresponding voxel positions in the global scene map to replace the original old parameters, thereby enabling the global scene map to reflect environmental changes in real time.

[0054] In step S240, the current pose information is determined by a filtering algorithm based on lidar point cloud data, inertial measurement unit data, and visual image data.

[0055] In one embodiment of this application, lidar point cloud features are extracted from lidar point cloud data, visual image features are extracted from visual image data, and motion features of inertial measurement units are extracted from inertial measurement unit data. The extracted lidar point cloud features, visual image features, and motion features of inertial measurement units are spatiotemporally aligned and feature-level fusion is performed to obtain a joint observation vector.

[0056] The lidar point cloud features include geometric structural features such as planar features, edge features, and corner features; visual image features include descriptors such as corner points, edges, scale-invariant feature transformations, acceleration robustness features, and orientation-based rapid rotation descriptors; and the motion features of the inertial measurement unit (IMU) refer to acceleration and angular velocity data. Different sensors typically acquire data at different sampling frequencies and timestamps, and their coordinate systems are inconsistent. Spatiotemporal alignment ensures that all sensor data is processed under the same time reference and unified coordinate system, eliminating time delays and spatial biases. Feature-level fusion integrates information from different sensors at the original feature level, fully utilizing the complementarity of each sensor and improving the richness of observation information. The aligned lidar point cloud features, visual image features, and IMU motion features can be stitched together to form a high-dimensional vector. Furthermore, weighted averaging and covariance matrix fusion can be used to form a joint observation vector containing multi-source information.

[0057] In one embodiment of this application, the joint observation vector is used as input, and a filtering algorithm is used to predict the state prediction value, which includes the attitude, position, velocity and sensor bias of the device; and the state prediction value is updated based on the joint observation vector, the Kalman gain is calculated and the state prediction value is corrected to obtain the state estimate value, and the state estimate value is determined as the current pose information.

[0058] In one embodiment of this application, the filtering algorithm is the core of pose estimation. It predicts the state of the device at the next moment using a motion model. This state prediction is an estimate derived from the state at the previous moment and the motion model, and can be represented as a comprehensive vector containing multiple key parameters of the robot's motion, as well as bias terms that may affect sensor measurements. The prediction step includes integrating the motion characteristics of the inertial measurement unit through a kinematic model, combining the state estimate from the previous moment to infer the attitude, position, and velocity at the current moment. Then, the state prediction obtained in the prediction step is corrected based on the actual joint observation vector. The Kalman gain is a weighting factor that balances the confidence between the predicted and observed values, ensuring that the corrected state estimate considers both the model's prediction and the actual measurements. The final corrected state estimate represents the pose information with the highest confidence at the current moment.

[0059] In the actual implementation of this embodiment, this step is located in the tracking thread. Specifically, it uses the Extended Sparse Kalman Filter (ESF) algorithm as the core algorithm to estimate the real-time pose of the device. Based on the preprocessed LiDAR point cloud data, inertial measurement unit data, and visual image data, the ESF is fused and input into the ESF algorithm. Through the prediction and update steps in the ESF algorithm, the attitude and position are estimated and iteratively updated in real time. The motion transformation relationship of the device between consecutive frames is calculated to obtain real-time odometry information, which includes the attitude angle and three-dimensional spatial position of the device at each moment.

[0060] In one embodiment of this application, a joint observation vector is constructed by extracting features from multiple sensors, spatiotemporal alignment, and feature-level fusion, and is used as the input to the filtering algorithm. This allows the filtering process to fully utilize the LiDAR's perception of the environmental geometry, the visual image's description of texture features, and the inertial measurement unit's response to instantaneous motion. After the filtering algorithm predicts, the observation update stage corrects the predicted value by calculating the Kalman gain, resulting in a more accurate state estimate and improving the stability of pose estimation.

[0061] In step S250, the initial Gaussian points in the global scene map are optimized based on the current pose information to obtain an optimized global scene.

[0062] In one embodiment of this application, based on a preset loss function and according to the current pose information, the static and dynamic attribute Gaussian points initialized in the global scene map are optimized respectively to obtain statically optimized Gaussian points and dynamically optimized Gaussian points. Then, rendering processing is performed based on the statically optimized Gaussian points and dynamically optimized Gaussian points to generate an optimized global scene. The above optimization steps are executed in the mapping thread.

[0063] In one embodiment of this application, the preset loss function includes a photometric loss function, a rigidity-preserving loss function, and a time-smoothing loss function.

[0064] The photometric loss function measures the pixel-level difference between the rendered image and the actual observed image. By minimizing this loss, it ensures that the rendered images of the scene model are as consistent as possible with the real images from different viewpoints. This includes calculating the absolute or squared difference between the corresponding pixel values ​​of the rendered and actual images. The As-Rigid-As-Possible (ARAP) loss function ensures that rigid objects or parts thereof maintain their geometric shape and relative positional relationships during optimization. It encourages adjacent dynamic points to have similar deformations at each deformation time stamp, maintaining the stability of local geometry and avoiding unreasonable deformations. The temporal smoothing loss function ensures that the motion of dynamic objects is continuous and smooth in the time series. It forces the deformation of each point within a short time window to be linearly smooth, avoiding abrupt or unnatural jumps. It forces the smoothness of the motion trajectory by penalizing the dynamic Gaussian point parameters between adjacent time steps.

[0065] In one embodiment of this application, when optimizing the static attribute Gaussian point, the position, color, and opacity of the static attribute Gaussian point are optimized based on the position information in the current pose information through multi-view photometric consistency constraints and photometric loss functions, and the static attribute Gaussian point is updated. The multi-view photometric consistency constraint refers to minimizing the difference between the projected photometric value and the actual image under different viewpoints. Specifically, it requires that when the scene model is rendered from different viewpoints, the photometric value projected onto the image is consistent with the photometric value of the actually observed image. This can be achieved by sampling pixels from images from multiple different viewpoints in the optimization iteration, calculating the photometric loss between these sampled pixels and the model rendering result, and then accumulating the losses for optimization.

[0066] In the actual implementation of this embodiment, the static attribute Gaussian points maintain relatively fixed parameters during reconstruction, reducing computational optimization and storage resource consumption, and only making necessary optimization adjustments. Based on the principle of multi-view photometric consistency constraints, the static attribute Gaussian points are projected onto real images from multiple different perspectives. The consistency between the projection position and image features is compared, and the 3D position of the static attribute Gaussian points is adjusted according to the projection differences. Simultaneously, based on the lighting information of the surrounding environment under different perspectives, the color and opacity parameters of the static attribute Gaussian points are optimized to make them more closely match the appearance of the actual scene. In real-world scenes, before optimization, the reconstructed colors of static buildings may differ under different perspectives, and their geometric contours may also be offset. However, through multi-view photometric consistency constraints, the system projects the static attribute Gaussian points corresponding to the building onto real images from multiple perspectives. By minimizing the photometric differences of the projection points under different perspectives, the position, color, and opacity of the Gaussian points are adjusted, ensuring that the reconstructed building model presents accurate and consistent geometric shapes and color textures under different perspectives, thus improving the detail performance of static scene modeling.

[0067] In one embodiment of this application, when optimizing the dynamic attribute Gaussian point, a deformation model is constructed based on the current pose information through a four-dimensional hash grid structure and a multilayer perceptron model; based on the deformation model, the rigidity-preserving loss function and the temporal smoothing loss function, the position and pose changes of the dynamic attribute Gaussian point at different times are predicted, and the dynamic attribute Gaussian point is updated to optimize the parameters of the dynamic attribute Gaussian point. The aforementioned four-dimensional hash grid structure consists of a spatial hash grid and a spatial-temporal hash grid.

[0068] Among them, the four-dimensional hash grid structure is a discretized data structure used to store and query spatial-temporal information. It combines three-dimensional space and one-dimensional time, and uses a hash function to map four-dimensional coordinates to indices in the grid, thereby achieving an effective representation of dynamic scenes. In the embodiments of this application, the four-dimensional hash grid structure (4D-hashgrid) consists of one spatial and three spatial-temporal hash grids. The spatial hash grid corresponds to three-dimensional spatial coordinates and is used to encode the geometric features of Gaussian points in the spatial dimension. The three spatial-temporal hash grids correspond to pairwise combinations of spatial and temporal coordinates, specifically including a hash grid combining the spatial x-axis, y-axis, and temporal t-axis; a hash grid combining the spatial y-axis, z-axis, and temporal t-axis; and a hash grid combining the spatial x-axis, z-axis, and temporal t-axis, used to encode the joint change features of Gaussian points in the spatial and temporal dimensions. When constructing the hash grid, considering that the motion deformation of dynamic objects has the characteristic of continuous smoothness in the temporal dimension, the resolution of the temporal t-axis is set relatively low, that is, a coarser grid division is used in the temporal dimension to reduce the number of parameters and force the model to learn smooth changes in the temporal dimension. For example, the resolution of the temporal axis can be set to one-quarter to one-half of that of the spatial axis.

[0069] The aforementioned multilayer perceptron model is a feedforward artificial neural network, consisting of at least input, hidden, and output layer neurons. These neurons are connected via nonlinear activation functions and weights to form a fully connected neural network with multiple hidden layers. Its input is the feature vector in a four-dimensional hash grid structure, and its output is the deformation parameters of a dynamic Gaussian point. In the embodiments of this application, a multi-head micro-multilayer perceptron model is used as the feature decoder. It receives the output features of the four-dimensional hash grid structure as input and decodes the grid features into deformation offsets of dynamic attribute Gaussian points through multiple parallel network heads. One head outputs a position offset to describe the spatial position change of the dynamic attribute Gaussian point relative to a reference position at different times; another head outputs a rotation offset to describe the rotation angle change of the dynamic attribute Gaussian point relative to a reference pose at different times. The combination of position and rotation offsets can fully express the motion change of the dynamic attribute Gaussian point over time.

[0070] In one embodiment of this application, for a dynamically attributed Gaussian point, due to its time-varying characteristics, this application first constructs a deformation model, uses a four-dimensional hash grid structure to encode spatial and temporal information, and combines a multilayer perceptron model to predict the position and attitude changes of the dynamically attributed Gaussian point at different time points. Based on this, a rigidity-preserving loss function and a time-smoothing loss function are introduced to optimize the parameters of the dynamically attributed Gaussian point. This not only ensures that the local rigid structure remains unchanged during deformation but also guarantees the continuity of the dynamic Gaussian point's trajectory in the time dimension, reducing the occurrence of abrupt motion changes. Taking a vehicle in a real-world scenario as an example, during the optimization of the dynamic attribute Gaussian point, based on the current observation data and the motion patterns learned by the deformation model, the changes in the vehicle's position and attitude at different time points are predicted, and the position offset and rotation offset are output. This describes the vehicle's motion trajectory and attitude adjustment process in a continuous time series. Furthermore, based on the prediction results, the position and rotation parameters of the fast dynamic Gaussian point are updated in real time, enabling the Gaussian point to accurately follow the actual movement of the vehicle. In the reconstructed scene, the vehicle's trajectory is continuous and smooth, and the attitude adjustment processes such as turning, acceleration, and deceleration are realistically reproduced, achieving accurate dynamic modeling of fast-moving objects.

[0071] In one embodiment of this application, the optimization process of the global scene map employs a strategy combining adaptive densification and pruning operations, and continuously improves map accuracy through iterative optimization. The adaptive densification operation dynamically adjusts the distribution density of Gaussian points based on the actual needs of scene reconstruction, evaluating in real time how well the current Gaussian points fit the scene's geometry and texture details. For areas with complex geometry or drastic texture changes, the number and density of Gaussian points are automatically increased. By inserting new Gaussian points, the detailed information of the area is better fitted, improving the precision of the reconstruction results. The pruning operation removes Gaussian points that contribute little to the overall reconstruction or do not meet the model's requirements. By evaluating the importance and parameter rationality of each Gaussian point during the rendering process, Gaussian points with low contribution, high redundancy, or abnormal parameters are removed from the global scene map, reducing the computational load and memory consumption of optimization, and maintaining the simplicity of the map.

[0072] Based on this, the global scene map is iteratively optimized. In the first iteration, the global scene map is empty, so global reconstruction is performed based on LiDAR point cloud data. Using the LiDAR point cloud data as initial input, an initial global scene map is constructed through Gaussian point initialization and insertion. In each iteration of the optimization phase, the reconstruction result from the previous iteration is used as the initial value for the current optimization, and the Gaussian point parameters in the global scene map are further optimized in conjunction with the observation data of the current frame. By continuously adjusting the position, color, scale, opacity, and the time and deformation offset parameters of the dynamic attribute Gaussian points, the geometric accuracy and appearance consistency of the map are gradually improved. This iterative optimization process continues until the reconstruction accuracy of all frames reaches the preset requirements, such as reaching a maximum of 500 iterations, to obtain a high-precision global scene map, providing a highly accurate scene representation for subsequent positioning, navigation, and interaction tasks.

[0073] In one embodiment of this application, the mapping thread further includes fusing the current frame's LiDAR point cloud data with the global scene map using a variational Gaussian process regression algorithm to establish a probabilistic correspondence between the LiDAR point cloud and Gaussian points in the map. Then, based on the parameters of the Gaussian points in the optimized global scene map, the current pose information is further corrected to reverse-correct any potential accumulated errors, making the device's pose and position estimation more accurate.

[0074] Please see Figure 3 , Figure 3 This is an exemplary embodiment of the present application illustrating a specific scene optimization method architecture. In this specific embodiment, the system hardware configuration includes an inertial measurement unit and a lidar. The inertial measurement unit operates at a frequency of 200 Hz, and the lidar operates at a frequency of 10 Hz. To achieve accurate fusion of multi-sensor data, a hard synchronization mechanism is used to time-align the inertial measurement unit data and the lidar data.

[0075] During map construction, Gaussian point parameters are initialized using both LiDAR point cloud data and visual image data. These Gaussian points serve as the basic representation units of the scene and are stored in a two-level storage structure: a compact Gaussian library in computer memory and a global Gaussian map in video memory. The global Gaussian map is organized using an octree structure with a hash index, supporting Gaussian point insertion and retrieval operations. The compact Gaussian library and the global Gaussian map are synchronized via data transfer between the host and device, ensuring timely updates to the Gaussian map in video memory. For data retrieval, a spatial hash table is used for efficient Gaussian point lookup. This table stores the priority information, key-value information, and corresponding Gaussian point data of each point. By calculating the spatial hash index of the query area, the target Gaussian point can be quickly located, supporting subsequent positioning, map updates, and data matching operations.

[0076] In the pose estimation process, state propagation is first performed, including forward propagation and state estimation. The motion state of the device is predicted using inertial measurement unit data. After obtaining the initial state estimate, a point-to-plane residual is constructed using point cloud data. The pose estimation is optimized by minimizing this residual. Simultaneously, motion distortion correction is performed on the LiDAR point cloud data to eliminate the influence of device motion on the point cloud data. In the pose optimization process, an iterative extended Kalman filter algorithm is used to implement LiDAR and vision updates. LiDAR updates utilize the corrected point cloud data to construct an observation model; vision updates first require distortion correction of the image data, then constructing a photometric residual, and optimizing the pose estimation by minimizing the photometric residual. Within a sliding window, Gaussian points can be optimized in real time. The optimization process constructs a loss function based on the difference between the rendered image and the real image, and updates the parameters of the Gaussian points by minimizing the loss function. Specifically, for all Gaussian points from time 0 to time k, the parameters of the Gaussian points are optimized by minimizing the difference between the rendered image at time k-1 and the real image at time k-1.

[0077] This application provides a scene optimization method that acquires LiDAR point cloud data, inertial measurement unit data, and visual image data from a robot. Based on the LiDAR point cloud data, a spatial index structure is constructed and an initial scene map is established. The visual image of the current frame is initialized to generate initial Gaussian points, which are then inserted into the initial scene map to obtain a global scene map. Next, the current pose information is determined based on the LiDAR point cloud data, inertial measurement unit data, and visual image data. The initialized Gaussian points in the global scene map are then optimized based on the current pose information to obtain an optimized global scene. This application introduces Gaussian point initialization processing, and... By clearly distinguishing between static and dynamic Gaussian points and using a unified global scene map representation, integrated management of static and dynamic elements is achieved, reducing the coordination complexity between the static map and the dynamic tracking module. Secondly, by using a multimodal data fusion strategy combined with a filtering algorithm to determine the current pose information, more stable pose information can be provided, which can further support the rapid optimization of subsequent Gaussian points. Moreover, by optimizing the initial Gaussian points in the global scene map based on the current pose information, a spatiotemporally consistent dynamic scene representation can be generated, improving the accuracy and stability of dynamic scene reconstruction while ensuring real-time performance.

[0078] The following describes an embodiment of the apparatus described in this application, which can be used to execute the scene optimization method described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the scene optimization method described in the above embodiments of this application.

[0079] Figure 4 This is a schematic diagram illustrating a scene optimization device according to an exemplary embodiment of this application. The device can be applied to... Figure 1 The method described is implemented in a device that has the necessary conditions for execution. This embodiment does not impose specific limitations on the devices to which the device is applicable.

[0080] like Figure 4 As shown, the exemplary scene optimization device includes: a perception data acquisition module 401, a global scene construction module 402, and a global scene optimization module 403.

[0081] The system includes a perception data acquisition module 401, which acquires LiDAR point cloud data, inertial measurement unit data, and visual image data of the robot; a global scene construction module 402, which constructs a spatial index structure based on the LiDAR point cloud data and establishes an initial scene map based on the spatial index structure; it performs Gaussian point initialization processing on the visual image of the current frame in the visual image data to generate initial Gaussian points, and inserts the initial Gaussian points into the initial scene map to obtain the global scene map. The initial Gaussian points include static attribute Gaussian points and dynamic attribute Gaussian points; and a global scene optimization module 403, which determines the current pose information based on the LiDAR point cloud data, inertial measurement unit data, and visual image data through a filtering algorithm; and optimizes the initial Gaussian points in the global scene map according to the current pose information to obtain an optimized global scene.

[0082] The embodiments of this application also provide a robot, characterized in that the robot includes a memory and a processor, the memory being used to store computer instructions that can be executed on the processor, and the processor being used to implement the scene optimization methods provided in the above embodiments when executing the computer instructions.

[0083] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the scene optimization methods provided in the above embodiments.

[0084] Figure 5 This is a schematic diagram illustrating the structure of a computer system for an electronic device, as shown in an exemplary embodiment of this application. It should be noted that... Figure 5 The computer system 500 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0085] like Figure 5 As shown, the computer system 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 502 or programs loaded from storage into Random Access Memory (RAM) 503. The RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus. An I / O interface 505 is also connected to the bus 504, where the I / O interface 505 refers to an input / output interface.

[0086] The following components are connected to I / O interface 505: input section 506 including keyboard, mouse, etc.; output section 507 including cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; storage section 508 including hard disk, etc.; and communication section 509 including network interface card, such as LAN (Local Area Network) card, modem, etc. The communication section performs communication processing via a network such as the Internet. A drive is also connected to I / O interface 505 as needed. Removable media 511, such as disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 510 as needed so that computer programs read from them can be installed into storage section 508 as needed.

[0087] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs various functions defined in the system of this application.

[0088] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0089] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0090] In the corresponding figures of the above embodiments, connecting lines can represent the connection relationship between various components, indicating more constitutive signal paths and / or one or more ends of some lines having arrows to indicate the main information flow direction. Connecting lines serve as an identifier and are not a limitation on the scheme itself, but rather, using these lines in conjunction with one or more exemplary embodiments helps to more easily connect circuits or logic units. Any signal represented (determined by design requirements or preferences) can actually include one or more signals that can be transmitted in any direction and can be implemented in any suitable type of signal scheme.

[0091] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0092] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0093] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements a scene optimization method as described in any of the above embodiments.

[0094] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0095] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0096] This application can be used in a wide range of general-purpose or special-purpose computing system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0097] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0098] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A method of scene optimization, characterized by, The scenario optimization method includes: Acquire the robot's LiDAR point cloud data, inertial measurement unit data, and visual image data; A spatial index structure is constructed based on the LiDAR point cloud data, and an initial scene map is established based on the spatial index structure. The visual image of the current frame in the visual image data is initialized with Gaussian points to generate initial Gaussian points, and the initial Gaussian points are inserted into the initial scene map to obtain the global scene map. The initial Gaussian points include static attribute Gaussian points and dynamic attribute Gaussian points. Based on the lidar point cloud data, inertial measurement unit data, and visual image data, the current pose information is determined through a filtering algorithm; Based on the current pose information, the initialized Gaussian points in the global scene map are optimized to obtain an optimized global scene.

2. The scene optimization method of claim 1, wherein, Constructing a spatial index structure based on the LiDAR point cloud data, and establishing an initial scene map based on the spatial index structure, includes: Based on the three-dimensional coordinates of each point cloud in the lidar point cloud data, the processable three-dimensional space is determined; The processable three-dimensional space is divided into multiple octree nodes using spatial hash index according to the preset spatial resolution, and a spatial index structure is generated. Each octree node corresponds to a spatial sub-region. The spatial hash index uses the position and size of the spatial sub-region corresponding to the octree node as the key value. Associate time parameters with the octree nodes in the spatial index structure, and use the completed spatial index structure as the initial scene map.

3. The scene optimization method of claim 1, wherein, The process of initializing Gaussian points for the current frame of the visual image data includes: The visual image of the current frame is initialized with basic parameters to obtain candidate Gaussian points with basic parameter descriptions, including position parameters and physical property parameters; For each candidate Gaussian point, static or dynamic attributes are assigned based on image semantics and motion information to complete the static-dynamic layer initialization and obtain the initialized Gaussian point.

4. The scene optimization method of claim 3, wherein, The visual image of the current frame is initialized with basic parameters to obtain candidate Gaussian points with basic parameter descriptions, including: In the initialization of position parameters, the visual image of the current frame is divided based on the coverage state of the LiDAR point cloud; for areas covered by the LiDAR point cloud, the three-dimensional position parameters of the Gaussian points are determined based on the points mapped from the LiDAR point cloud data to the visual image of the current frame; for areas not covered by the LiDAR point cloud, the three-dimensional position parameters of the Gaussian points are determined by performing feature point matching and triangulation measurement on the visual image of the current frame and the visual image of the previous frame, wherein the visual image of the previous frame is the visual image in the frame sequence whose time series is the frame preceding the current frame; In the initialization of physical attribute parameters, based on the position parameters of the Gaussian points obtained during initialization, neighboring Gaussian points are retrieved in the global scene map, and values ​​are assigned based on the physical attribute parameters corresponding to the neighboring Gaussian points. The physical attribute parameters include at least one of color parameters, scale parameters, and opacity parameters.

5. The scene optimization method of claim 3, wherein, For each candidate Gaussian point, static or dynamic attributes are assigned based on image semantics and motion information, including: Perform semantic segmentation on the visual image of the current frame, and determine the static and dynamic regions in the image based on the category of the target object in the image; Optical flow calculations are performed on the visual images of the current frame and the previous frame to obtain the optical flow vector amplitude for each pixel. Based on the static region, the dynamic region, and the optical flow vector amplitude, the candidate Gaussian points are divided into static attribute Gaussian points and dynamic attribute Gaussian points. The dynamic attribute Gaussian points include slow dynamic Gaussian points and fast dynamic Gaussian points, both of which include time parameters.

6. The scene optimization method of claim 5, wherein, Inserting the initial Gaussian points into the initial scene map includes: For the static attribute Gaussian point, the root voxel within the field of view is located based on the spatial hash index. When the leaf voxel in the root voxel is not filled, a new Gaussian point is initialized based on the lidar point cloud data and visual image data, and the newly initialized Gaussian point is inserted into the leaf voxel. For the dynamic attribute Gaussian point, the initial Gaussian point is inserted into the corresponding leaf voxel.

7. The method of claim 1, wherein, Based on the lidar point cloud data, inertial measurement unit data, and visual image data, the current pose information is determined through a filtering algorithm, including: Lidar point cloud features are extracted from the lidar point cloud data, visual image features are extracted from the visual image data, and motion features of the inertial measurement unit are extracted from the inertial measurement unit data, respectively. The extracted lidar point cloud features, visual image features, and motion features of the inertial measurement unit are spatiotemporally aligned and then fused at the feature level to obtain a joint observation vector. Using the joint observation vector as input, a filtering algorithm is applied to predict the state, resulting in predicted state values. These predicted state values ​​include the device's attitude, position, velocity, and sensor bias. The state prediction value is updated based on the joint observation vector, the Kalman gain is calculated and the state prediction value is corrected to obtain the state estimate value, and the state estimate value is determined as the current pose information.

8. The method of claim 1, wherein, Based on the current pose information, the initial Gaussian points in the global scene map are optimized to obtain an optimized global scene, including: Based on a preset loss function, and according to the current pose information, the static attribute Gaussian points and dynamic attribute Gaussian points of the initialized Gaussian points in the global scene map are optimized respectively to obtain statically optimized Gaussian points and dynamically optimized Gaussian points. The rendering process is performed based on the statically optimized Gaussian points and the dynamically optimized Gaussian points to generate an optimized global scene.

9. The scene optimization method of claim 8, wherein, The preset loss function includes a photometric loss function, a rigidity-preserving loss function, and a temporal smoothing loss function. Based on the preset loss function and according to the current pose information, the static attribute Gaussian points and dynamic attribute Gaussian points initialized in the global scene map are optimized respectively, including: For the static attribute Gaussian point, the position, color, and opacity of the static attribute Gaussian point are optimized based on the position information in the current pose information through multi-view photometric consistency constraints and photometric loss function, and the static attribute Gaussian point is updated. The multi-view photometric consistency constraint refers to minimizing the difference between the projected photometric value and the actual image under different viewpoints. For the dynamic attribute Gaussian point, a deformation model is constructed based on the current pose information using a four-dimensional hash grid structure and a multilayer perceptron model. Based on the deformation model, a rigidity-preserving loss function, and a temporal smoothing loss function, the position and pose changes of the dynamic attribute Gaussian point at different times are predicted, and the dynamic attribute Gaussian point is updated. The four-dimensional hash grid structure consists of a spatial hash grid and a spatial-temporal hash grid.

10. A robot, characterized in that The robot includes a memory and a processor, the memory being used to store computer instructions that can be executed on the processor, and the processor being used to implement the method of any one of claims 1 to 9 when executing the computer instructions.

Images