Air-ground collaborative exploration method in unknown environment based on three-dimensional Gaussian splashing

Abstract

The invention relates to an air-ground collaborative exploration method in an unknown environment based on three-dimensional Gaussian splashing, and belongs to the field of air-ground robot collaborative exploration in a dangerous environment. The method comprises the following steps: S1, acquiring image data of a to-be-explored area by using an unmanned aerial vehicle, describing a complex scene and a differentiable rendering technology by using three-dimensional Gaussian distribution, accurately estimating a camera pose of the unmanned aerial vehicle and reconstructing a three-dimensional environment in real time by combining a simultaneous localization and mapping (SLAM) system, traversing a space and marking a series of key waypoints; step S2, the ground robot realizes accurate exploration of a local three-dimensional environment by using waypoints provided by the unmanned aerial vehicle and combining a collaborative mechanism of global planning and local planning; and S3, dynamically updating the path plan through the robot in the exploration process, and constructing a fine three-dimensional map of the exploration area. According to the invention, the air-ground robot can collaboratively explore the unknown environment and establish the environment map.

Description

Technical Field

[0001] The present invention belongs to the field of air-ground robot collaborative exploration in dangerous environments, and specifically relates to an air-ground collaborative exploration method in an unknown environment based on three-dimensional Gaussian splashing. Background Art

[0002] In robotics, exploration refers to the core function of a robot, in unfamiliar or partially familiar environments, using perception, decision-making, and path planning strategies to fully acquire environmental information and determine its own position within a reasonable timeframe. Simultaneous Localization and Mapping (SLAM) technology, closely related to exploration, uses sensor data to achieve autonomous localization and mapping in unknown environments. However, traditional SLAM relies on manual traversal of the environment to obtain a global map. Performing search, rescue, or exploration operations in extreme or hazardous environments, such as inside buildings, deep caves, narrow tunnels, and complex mines, is often a task fraught with unknown risks and challenges for humans. While current ground robots possess stable mobility and physical interaction capabilities, their relatively low field of view limits them to ground-level observation, making it difficult to acquire comprehensive environmental information. For example, in tunnel collapse search and rescue, ground robots can only construct local two-dimensional grid maps, making it difficult to identify high-lying collapsed structures or overhanging objects, resulting in critical blind spots in navigation path planning. Drones, with their three-dimensional maneuverability, can quickly acquire bird's-eye view data. However, they face significant limitations when operating independently in unknown areas. Flight stability and dynamic obstacle avoidance in complex airspace rely heavily on high-precision environmental models. In the early stages of exploration, the lack of such models and interference from dynamic obstacles can easily lead to positioning drift and even collisions. Therefore, establishing a collaborative interaction mechanism between drones and ground robots is key to improving system reliability. Summary of the Invention

[0003] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide an air-ground collaborative exploration method in an unknown environment based on three-dimensional Gaussian splashing. By transplanting the SLAM algorithm based on three-dimensional Gaussian splashing by unmanned aerial vehicles, rapid coverage area scanning is achieved in three-dimensional space, key waypoints are marked, and ground robots receive the waypoints and reach the target area for refined exploration.

[0004] To achieve the above-mentioned purpose, the technical solution of the present invention is: a method for collaborative air-ground exploration in an unknown environment based on three-dimensional Gaussian splashing, comprising the following steps:

[0005] Step S1: Use a drone to acquire image data of the area to be explored, use a 3D Gaussian distribution to describe the complex scene, combine differentiable rendering technology with a Simultaneous Localization and Mapping (SLAM) system, accurately estimate the drone camera pose and reconstruct the 3D environment in real time, traverse the space and mark a series of key waypoints;

[0006] Step S2: The ground robot uses the waypoints provided by the drone and combines the global planned path with the local planned path to achieve accurate exploration of the local three-dimensional environment.

[0007] Step S3: Dynamically update the path planning during the exploration process by the ground robot to build a detailed three-dimensional map of the exploration area.

[0008] Furthermore, a three-dimensional Gaussian distribution is used to describe complex scenes, specifically:

[0009] 3D Gaussian is a mathematical tool for describing the distribution of three-dimensional spatial data. Its standard form is:

[0010]

[0011] Among them, (x, y, z) are the coordinate components of the point in three-dimensional space, (μ x ,μ y ,μ z ) is the coordinate of the center point, (σ X ,σ y ,σ z ) is the standard deviation in three directions, describing the probability density of the three-dimensional normal distribution.

[0012] When using a 3D Gaussian distribution to describe complex scenes, the Structure from Motion (SFM) technique is used to recover a 3D point cloud from a set of 2D images. Each point in the point cloud is converted into a 3D Gaussian distribution, including parameters such as position, color, covariance matrix, and opacity. The covariance matrix describes the shape and direction of the Gaussian distribution in 3D space. To make it more suitable for scene description, the 3D Gaussian distribution is represented as follows:

[0013]

[0014] Where x is any point in three-dimensional space; μ∈R 3 are the coordinates of the center point of the three-dimensional Gaussian distribution, which are consistent with the coordinates of the center point in the above standard form; r is the radius, which is used to describe the range of the distribution; θ∈[0,1] is the opacity, which controls the visibility of the distribution.

[0015] Furthermore, the differentiable rendering technology is used to perform the following operations, specifically:

[0016] Using a set of 3D Gaussian distributions and the camera pose, we first sort all Gaussian distributions along the depth direction. Then, we perform an alpha blending operation on the 2D projections of each Gaussian distribution in depth order in pixel space, effectively synthesizing the final color value w = (u, v) for each pixel:

[0017]

[0018] Where C(w) is the final color value of pixel w, u is the horizontal coordinate value of the pixel point, and v is the vertical coordinate value of the pixel point; i is each Gaussian distribution in the traversal set n, and n is the set of three-dimensional Gaussian distributions involved in rendering; j is the Gaussian distribution before traversal i (i.e., 1 to i-1) to determine the part of pixel w that is not covered by the previous Gaussian distribution; c i is the color value of the i-th Gaussian distribution, and Q i (w) multiplication represents the color contribution of the i-th Gaussian distribution to pixel w; Q i The calculation method of (w) is the same as that of the Gaussian function, which is the color contribution weight of the i-th Gaussian distribution at pixel w; Q j (w) is the color contribution weight of the j-th Gaussian distribution at pixel w, which is similar to 1-Q j (w) jointly calculate the uncovered degree of the previous Gaussian distribution for pixel w; To represent the portion of the pixel not covered by the previous j=1 to i-1 Gaussian distributions when calculating the color contribution of the i-th Gaussian distribution to the pixel w. Note that μ and r are two-dimensional Gaussian functions splashed in the pixel space:

[0019]

[0020] Among them, μ 2D is the center coordinate of the two-dimensional Gaussian distribution spliced ​​in the pixel space, K is the camera internal parameter, E λ is the external parameter of camera rotation and translation at the λth frame, μ is the coordinate of the center point of the three-dimensional Gaussian distribution, r 2D is the radius of the two-dimensional Gaussian distribution spliced ​​in the pixel space, which is used to describe the range of the two-dimensional Gaussian distribution in the pixel space; f r is the focal length, and d is the depth of the i-th Gaussian value in camera coordinates. Differentiable rendering technology is used:

[0021]

[0022] Where D(w) is the depth value of the pixel under the differentiable depth rendering technology obtained by accumulating the depth contribution of all Gaussian distributions to the pixel w; i is each Gaussian distribution in the traversal set n, and n is the set of three-dimensional Gaussian distributions involved in the rendering; d i is the depth parameter associated with the i-th Gaussian distribution, and Q i (w) multiplication represents the depth contribution of the i-th Gaussian distribution to pixel w; Q i The calculation method of (w) is the same as that of the Gaussian function, which is the weight of the i-th Gaussian distribution contributing to the pixel depth at pixel w; Q j (w) is the pixel depth contribution weight of the j-th Gaussian distribution at pixel w, which is the same as 1-Q j (w) jointly calculate the uncovered degree of the previous Gaussian distribution for pixel w; It represents the part of the pixel not covered by the previous j=1 to i-1 Gaussian distributions when calculating the depth contribution of the i-th Gaussian distribution to the pixel w.

[0023] Compare with the input depth map and return the gradient relative to the 3D map. And use differentiable rendering technology to get the outline image to determine visibility:

[0024]

[0025] Where S(w) is the visibility value at pixel w, i is each Gaussian distribution in the traversal set n, n is the set of three-dimensional Gaussian distributions involved in rendering, Q i The calculation method of (w) is the same as that of the Gaussian function, which is the transparency contribution weight of the i-th Gaussian model at pixel w; Q j (w) is the transparency contribution weight of the i-th Gaussian model at pixel w, which is similar to 1-Q j (w) jointly calculate the uncovered degree of the previous Gaussian distribution for pixel w; When calculating the visibility contribution of the i-th Gaussian distribution to the pixel w, the portion of the pixel that is not covered by the previous j=1 to i-1 Gaussian distributions.

[0026] Furthermore, the simultaneous positioning and mapping system is implemented as follows:

[0027] a) Use a drone to acquire the first frame of RGB-D data from the area to be explored, extract the RGB image, depth image, and camera intrinsic parameters, initialize the camera pose to the unit matrix, and normalize the RGB image. Use the camera intrinsic unit pose matrix (extrinsic parameters) to calculate the transformation matrix from the camera to the world coordinate system. Then, based on the depth image information and the camera intrinsic and extrinsic parameters, construct the initial 3D point cloud. Associate a Gaussian model with each point in the point cloud, and calculate the model parameters through depth projection to form an initial 3D Gaussian model set;

[0028] b) Based on the current drone camera pose and Gaussian model parameters, differentiable rendering technology is used to efficiently render and generate virtual RGB images and depth images. Pixels within the visible contour map are evaluated, and a tracking loss function consisting of a weighted sum of RGB loss and depth loss is constructed. Utilizing the differentiable characteristics of differentiable rendering, the gradient of the loss function with respect to the camera pose is calculated through backpropagation. Based on the gradient, continuous iterative optimization is performed to adjust the camera pose parameters to minimize the difference between the rendered image and the actual observed image. In each iteration, the system updates the pose according to the gradient direction, gradually approaching the true pose, and achieving an accurate estimation of the camera pose.

[0029] c) Generate contour masks to identify areas with insufficient coverage or geometric holes in the map, generate depth difference masks to detect areas where the rendered depth deviates significantly from the observed depth, fuse the contour coverage mask with the depth difference mask to locate the updated area, and expand the Gaussian model according to the proximity rule to ensure the map's accurate description and dynamic update of the scene, thereby improving the map's integrity and accuracy;

[0030] d) Based on disparity, scene change, and overlap, key frames are selected (as in Equation (7)). Parameter block selective joint optimization is performed. Starting from the most recently constructed map, frames that may affect the newly added Gaussian function are optimized, reducing redundant calculations and rapidly responding to environmental changes. Finally, the camera pose and map are jointly optimized to ensure that the map is consistent with the current observation. As new data is input, an accurate and dense 3D scene representation is gradually constructed, improving the map's adaptability and accuracy in complex environments.

[0031]

[0032] Among them, Keyframe is a key frame set, which is used to filter key frames that meet the conditions; P I The percentage of points remaining inside the image is used to measure the proportion of valid points (points that do not exceed the image range) in the current frame to ensure that the key frame contains sufficient scene information; k is the translation change value of the kth frame, tra k+1 is the translation change value of the k+1th frame; tra τ is the threshold of translation change, which is used to judge whether the translation change between two adjacent frames is large enough. k+1 -tra k )>tra τ When , it means that the translation change between the two frames is significant; rot k is the rotation change value of the kth frame, rot k+1 is the rotation change value of the k+1th frame; rot τ The rotation change threshold is used to determine whether the rotation change between two adjacent frames is large enough. k+1 -rotk )>rot τ , it indicates that the rotation changes between the two frames are significant.

[0033] By organically combining complex scene descriptions using 3D Gaussian distributions, differentiable rendering technology, and a simultaneous localization and mapping system, and incorporating image data from drones exploring unknown areas, the team continuously advances the processes of data acquisition, pose estimation, map updates, and joint optimization, achieving accurate estimation of the drone's camera pose and real-time reconstruction of the 3D environment. This allows the reconstruction results to be gradually refined as the drone explores, ultimately constructing an accurate and dense 3D scene representation, providing reliable support for air-ground collaborative exploration in unknown environments.

[0034] Furthermore, the global planning path is specifically as follows:

[0035] a) Define the boundary point f at the junction of the known and unknown areas, and also consider the waypoint w as the boundary point. Define the boundary point set F as the union of the boundary of the known / unknown areas and the drone waypoints F = {f1,f2,…f m ,w1,w2,…w l}={F1,F2…F q}, construct a sparse random roadmap covering the subspace to be explored;

[0036] b) Based on the A* algorithm, the shortest path from the current viewpoint to each boundary point is calculated in parallel, the path distance matrix D' is generated and mapped to the fitness weight ada;

[0037]

[0038] Among them, D' [i][j] From the boundary point F i To the boundary point F j distance; a ij is the element D' in the distance matrix [i][j] The reciprocal of a ij =1 / D' [i][j] ), which is used to measure the fitness in path planning. The smaller the distance, the larger the reciprocal, and the higher the fitness.

[0039] c) Define the binary decision variable x ij Mark the boundary point access status, construct the fitness function with the inverse of the path distance, and construct the objective optimization function that minimizes the total path:

[0040]

[0041] Among them, D' [i][j] From the boundary point F i To the boundary point F jdistance; [i] is the boundary point set F={F1,F2,…,F q The i-th boundary point F in i As the starting point for distance calculation; [j] is the jth boundary point F in the boundary point set F j , as the end point of distance calculation; when the robot moves from the boundary point F i Arrival at boundary point F j When x ij =1; when the robot does not leave the boundary point F i Arrival at boundary point F j When x ij =0, which helps the robot identify the explored area during path planning.

[0042] d) Use genetic algorithms to solve the dynamic traveling salesman problem to calculate the global planning path trajectory of the robot exploration.

[0043] Furthermore, the local planning path is specifically as follows:

[0044] a) Define the viewpoint set V = {v1, v2, ... v n Each viewpoint is configured with fan-shaped field of view parameters, radius r, center angle θ, number of viewpoints on the local path n, and establishes the benchmark value of the unobstructed field of view area

[0045] b) Obtain the field of view overlap area R and obstacle occlusion area O through discretized grid analysis and calculate the effective coverage area

[0046] c) Construct the viewpoint distance matrix D, with the dual objectives of maximizing coverage area and minimizing path;

[0047]

[0048] where d [i][j] Represents the viewpoint v i To viewpoint v j distance;

[0049] d) The A* algorithm is used to generate the optimal path between viewpoints, and the traveling salesman problem is combined to calculate the local planning path trajectory that meets the coverage requirement.

[0050] Furthermore, the solution to the dynamic traveling salesman problem is:

[0051] a) First, define the priority of waypoints in the set as higher than that of ordinary boundary points, classify the priority of the boundary point set F, and quantify the weight difference between key waypoints and ordinary boundary points;

[0052] b) Generate a high-priority waypoint detection sequence as the parent generation through a roulette wheel selection mechanism, perform path segment intersection and random edge point position mutation to generate offspring;

[0053] c) Introducing a rigid constraint mechanism, setting the fitness of individuals that do not contain mandatory waypoints to zero and eliminating them, while retaining the current optimal solution and fixing its path segment;

[0054] d) As the robot moves, the boundary point set F and the distance matrix D' are updated in real time, and new nodes are dynamically integrated into the optimization process, gradually expanding the path coverage through incremental solution.

[0055] Furthermore, a detailed 3D map of the exploration area is constructed as follows:

[0056] The point cloud data of the surrounding environment is acquired through LiDAR and then processed. The point cloud registration algorithm is used to estimate the movement of the sensor between consecutive frames to achieve real-time pose estimation. At the same time, the point cloud data of the current frame is fused into the global map according to the estimated pose to construct a three-dimensional point cloud map of the environment.

[0057] The present invention also provides an air-ground collaborative exploration system based on an unknown environment, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, it can implement any of the method steps described above.

[0058] The present invention also provides a computer-readable storage medium on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, any of the method steps described above can be implemented.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. The present invention can obtain relatively rich environmental conditions for unknown environments;

[0061] 2. The drone is equipped with a 3D Gaussian splatter SLAM algorithm, which uses multi-view feature matching and density-weighted optimization to generate a highly accurate 3D reconstructed map model and mark key waypoints.

[0062] 3. The ground robot implements precise path planning based on the marked waypoints transmitted in real time by the drone;

[0063] 4. The collaborative exploration mechanism reduces the risk of single point failure through distributed computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0065] Figure 2 A flowchart of a three-dimensional Gaussian distribution for describing complex scenes and differentiable rendering technology according to an embodiment of the present invention;

[0066] Figure 3 3D Gaussian splash SLAM system flow chart of an embodiment of the present invention;

[0067] Figure 4 This is a diagram showing the path planning components of a ground mobile robot according to an embodiment of the present invention;

[0068] Figure 5 A schematic diagram of global path planning for a ground mobile robot according to an embodiment of the present invention;

[0069] Figure 6 A schematic diagram of local path planning for a ground mobile robot according to an embodiment of the present invention;

[0070] Figure 7 An RGB image captured by a drone camera in a simulation environment according to an embodiment of the present invention;

[0071] Figure 8 A depth image captured by a drone camera in a simulation environment according to an embodiment of the present invention;

[0072] Figure 9 This is a 3D Gaussian SLAM reconstruction image of a UAV in a simulation environment according to an embodiment of the present invention;

[0073] Figure 10 This is a diagram of the experimental environment scene in a simulation scenario according to an embodiment of the present invention;

[0074] Figure 11 A point cloud image of the experimental environment of an embodiment of the present invention;

[0075] Figure 12 Marking key areas of the experimental environment for the drone of an embodiment of the present invention;

[0076] Figure 13 The ground robot of the embodiment of the present invention performs global planning based on the UAV waypoints;

[0077] Figure 14 This is the path trajectory generated globally by the ground robot in an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0079] The present invention provides an air-ground collaborative exploration method in an unknown environment based on three-dimensional Gaussian splashing, comprising the following steps:

[0080] Step S1: Use a drone to acquire image data of the area to be explored, use a 3D Gaussian distribution to describe the complex scene, combine differentiable rendering technology with a Simultaneous Localization and Mapping (SLAM) system, accurately estimate the drone camera pose and reconstruct the 3D environment in real time, traverse the space and mark a series of key waypoints;

[0081] Step S2: The ground robot uses the waypoints provided by the drone and combines the global planned path with the local planned path to achieve accurate exploration of the local three-dimensional environment.

[0082] Step S3: Dynamically update the path planning during the exploration process by the ground robot to build a detailed three-dimensional map of the exploration area.

[0083] The following is a specific implementation process of the present invention.

[0084] Please refer to Figure 1 The present invention provides an air-ground collaborative exploration method in an unknown environment based on three-dimensional Gaussian splashing, which specifically includes the following steps:

[0085] Step S1: Use a drone to acquire image data of the area to be explored, use three-dimensional Gaussian distribution to describe complex scenes and differentiable rendering technology, combine with the SLAM system, accurately estimate the drone camera pose and reconstruct the three-dimensional environment in real time, traverse the space and mark a series of key waypoints;

[0086] Step S2: The ground robot uses the waypoints provided by the UAV and combines the collaborative mechanism of global planning and local planning to achieve accurate exploration of the local three-dimensional environment;

[0087] Step S3: The robot dynamically updates the path planning during the exploration process to build a detailed three-dimensional map of the exploration area.

[0088] In this embodiment, in order to accurately estimate the drone camera pose and reconstruct the 3D environment in real time and provide waypoints to the ground robot, a 3D Gaussian distribution is used to describe the complex scene and a differentiable rendering technique is used. Figure 2 As shown, specifically:

[0089] 3D Gaussian is a mathematical tool for describing the distribution of three-dimensional spatial data. Its standard form is:

[0090]

[0091] Among them, (x, y, z) are the coordinate components of the point in three-dimensional space, (μ x ,μ y ,μ z ) is the coordinate of the center point, (σ x ,σy ,σ z ) is the standard deviation in three directions, describing the three-dimensional normal distribution probability density.

[0092] When using a 3D Gaussian distribution to describe complex scenes, a 3D point cloud is recovered from a set of 2D images using the Structure from Motion (SFM) technique. Each point in the point cloud is converted into a 3D Gaussian distribution, which contains parameters such as position, color, covariance matrix, and opacity. The covariance matrix describes the shape and orientation of the Gaussian distribution in 3D space. To make it more suitable for scene description, the 3D Gaussian distribution is represented as follows:

[0093]

[0094] Where x is any point in three-dimensional space; μ∈R 3 are the coordinates of the center point of the three-dimensional Gaussian distribution, which are consistent with the coordinates of the center point in the above standard form; r is the radius, which is used to describe the range of the distribution; θ∈[0,1] is the opacity, which controls the visibility of the distribution.

[0095] Given a set of 3D Gaussian distributions and the camera pose, we first sort all the Gaussians along the depth direction. Then, in pixel space, we perform an alpha blending operation on the 2D projections of each Gaussian distribution in depth order, effectively synthesizing the final color value w = (u, v) for each pixel:

[0096]

[0097] Where c(w) is the final color value of pixel w, u represents the horizontal coordinate value of the pixel point, and v represents the vertical coordinate value of the pixel point; i is each Gaussian distribution in the traversal set n, and n is the set of three-dimensional Gaussian distributions involved in rendering; j is the Gaussian distribution before traversal i (i.e., 1 to i-1) to determine the part of pixel w that is not covered by the previous Gaussian distribution; c i is the color value of the i-th Gaussian distribution, and Q i (w) multiplication represents the color contribution of the i-th Gaussian distribution to pixel w; Q i The calculation method of (w) is the same as that of the Gaussian function, which is the color contribution weight of the i-th Gaussian distribution at pixel w; Q j (w) is the color contribution weight of the j-th Gaussian distribution at pixel w, which is similar to 1-Q j (w) jointly calculate the uncovered degree of the previous Gaussian distribution for pixel w; is the portion of the pixel not covered by the previous j=1 to i-1 Gaussian distributions when calculating the color contribution of the i-th Gaussian distribution to the pixel w. Note that μ and r are two-dimensional Gaussian functions splashed in the pixel space:

[0098]

[0099] Among them, μ 2D is the center coordinate of the two-dimensional Gaussian distribution spliced ​​in the pixel space, K is the camera internal parameter, E λ is the external parameter of camera rotation and translation at the λth frame, μ is the coordinate of the center point of the three-dimensional Gaussian distribution, r 2D is the radius of the two-dimensional Gaussian distribution spliced ​​in the pixel space, which is used to describe the range of the two-dimensional Gaussian distribution in the pixel space; f r is the focal length, and d is the depth of the i-th Gaussian value in camera coordinates. Differentiable rendering technology is used:

[0100]

[0101] Where D(w) is the depth value of the pixel under the differentiable depth rendering technology obtained by accumulating the depth contribution of all Gaussian distributions to the pixel w; i is each Gaussian distribution in the traversal set n, and n is the set of three-dimensional Gaussian distributions involved in the rendering; d i is the depth parameter associated with the i-th Gaussian distribution, and Q i (w) multiplication represents the depth contribution of the i-th Gaussian distribution to pixel w; Q i The calculation method of (w) is the same as that of the Gaussian function, which is the weight of the i-th Gaussian distribution contributing to the pixel depth at pixel w; Q j (w) is the pixel depth contribution weight of the j-th Gaussian distribution at pixel w, which is the same as 1-Q j (w) jointly calculate the uncovered degree of the previous Gaussian distribution for pixel w; When calculating the depth contribution of the i-th Gaussian distribution to the pixel w, the portion of the pixel that is not covered by the previous j=1 to i-1 Gaussian distributions.

[0102] Compare with the input depth map and return the gradient relative to the 3D map. And use differentiable rendering technology to get the outline image to determine visibility:

[0103]

[0104] Where S(w) is the visibility value at pixel w, i is used to traverse each Gaussian distribution in the set n, n represents the set of three-dimensional Gaussian distributions involved in rendering, Q i The calculation method of (w) is the same as that of the Gaussian function, which is the transparency contribution weight of the i-th Gaussian model at pixel w; Q j (w) is the transparency contribution weight of the i-th Gaussian model at pixel w, which is similar to 1-Q j (w) jointly calculate the uncovered degree of the previous Gaussian distribution for pixel w; When calculating the visibility contribution of the i-th Gaussian distribution to the pixel w, the portion of the pixel that is not covered by the previous j=1 to i-1 Gaussian distributions.

[0105] Based on three-dimensional Gaussian distribution to describe complex scenes and differentiable rendering technology, a complete SLAM system is built.

[0106] In this embodiment, the process of the simultaneous positioning and mapping system of the ground robot is as follows: Figure 4 As shown, the specific steps are as follows:

[0107] a) Use a drone to acquire the first frame of RGB-D data from the area to be explored, extract the RGB image, depth image, and camera intrinsic parameters, initialize the camera pose to the unit matrix, and normalize the RGB image. Use the camera intrinsic unit pose matrix (extrinsic parameters) to calculate the transformation matrix from the camera to the world coordinate system. Then, based on the depth image information and the camera intrinsic and extrinsic parameters, construct the initial 3D point cloud. Associate a Gaussian model with each point in the point cloud, and calculate the model parameters through depth projection to form an initial 3D Gaussian model set;

[0108] b) Based on the current drone camera pose and Gaussian model parameters, differentiable rendering technology is used to efficiently render and generate virtual RGB images and depth images. Pixels within the visible contour map are evaluated, and a tracking loss function consisting of a weighted sum of RGB loss and depth loss is constructed. Utilizing the differentiable characteristics of differentiable rendering, the gradient of the loss function with respect to the camera pose is calculated through backpropagation. Based on the gradient, continuous iterative optimization is performed to adjust the camera pose parameters to minimize the difference between the rendered image and the actual observed image. In each iteration, the system updates the pose according to the gradient direction, gradually approaching the true pose, and achieving an accurate estimation of the camera pose.

[0109] c) Generate a contour mask to identify areas with insufficient coverage or geometric holes in the map, generate a depth difference mask to detect areas where the rendered depth deviates significantly from the observed depth, fuse the contour coverage mask with the depth difference mask to locate the updated area, and expand the Gaussian model according to the proximity rule to ensure the map's accurate description and dynamic update of the scene, thereby improving the map's integrity and accuracy;

[0110] d) Keyframes are selected based on disparity, scene change, and overlap ratio (as in Equation 7). A selective joint optimization of parameter blocks is then performed. Starting with the most recently constructed map, frames that may affect newly added Gaussian functions are optimized, reducing redundant computations and enabling rapid response to environmental changes. Finally, the camera pose and map are jointly optimized to ensure consistency between the map and the current observations. As new data is input, an accurate and dense 3D scene representation is gradually constructed, improving the map's adaptability and accuracy in complex environments.

[0111]

[0112] Among them, Keyframe is a key frame set, which is used to filter key frames that meet the conditions; P I The percentage of points remaining inside the image is used to measure the proportion of valid points (points that do not exceed the image range) in the current frame to ensure that the key frame contains sufficient scene information; k is the translation change value of the kth frame, tra k+1 is the translation change value of the k+1th frame; tra τ is the threshold of translation change, which is used to judge whether the translation change between two adjacent frames is large enough. k+1 -tra k )>tra τ When , it means that the translation change between the two frames is significant; rot k is the rotation change value of the kth frame, rot k+1 is the rotation change value of the k+1th frame; rot τ The rotation change threshold is used to determine whether the rotation change between two adjacent frames is large enough. k+1 -rot k )>rot τ , it indicates that the rotation changes between the two frames are significant.

[0113] By organically combining complex scene descriptions using 3D Gaussian distributions, differentiable rendering technology, and a simultaneous localization and mapping system, and incorporating image data from drones exploring unknown areas, the team continuously advances the processes of data acquisition, pose estimation, map updates, and joint optimization, achieving accurate estimation of the drone's camera pose and real-time reconstruction of the 3D environment. This allows the reconstruction results to be gradually refined as the drone explores, ultimately constructing an accurate and dense 3D scene representation, providing reliable support for air-ground collaborative exploration in unknown environments.

[0114] In this embodiment, in order to improve the efficiency of the ground robot's autonomous exploration, key waypoints are marked, global guidance is added to the ground robot's autonomous exploration strategy, and a high-resolution area defined around the robot's current position is generated to meet the requirements of the robot's motion control. Therefore, step S2 is specifically as follows: the ground robot uses the waypoints provided by the drone and combines the collaborative mechanism of global planning and local planning to achieve accurate exploration of the local three-dimensional environment. Figure 4 As shown in the path planning diagram, for global path planning of a ground robot, the set of boundary points F within the global planning scope consists of the boundary points f at the junction of the known and unknown areas and the waypoints w provided by the drone. In the local scope, the locations where boundary points can be observed are defined as viewpoints v.

[0115] like Figure 5As shown in the figure, global path planning is specifically divided into the following steps:

[0116] a) Define the boundary point f at the junction of the known and unknown areas, and also consider the waypoint w as the boundary point. Define the boundary point set F as the union of the boundary of the known / unknown areas and the drone waypoints F = {f1,f2,…f m ,w1,w2,…w l}={F1,F2…F q}, construct a sparse random roadmap covering the subspace to be explored;

[0117] b) Based on the A* algorithm, the shortest path from the current viewpoint to each boundary point is calculated in parallel, the path distance matrix D' is generated and mapped to the fitness weight ada;

[0118]

[0119] Among them, D' [i][j] From the boundary point F i To the boundary point F j distance; a ij is the element D' in the distance matrix [i][j] The reciprocal of a ij =1 / D' [i][j] ), which is used to measure the fitness in path planning. The smaller the distance, the larger the reciprocal, and the higher the fitness.

[0120] c) Define the binary decision variable x ij Mark the boundary point access status, construct the fitness function with the inverse of the path distance, and construct the objective optimization function that minimizes the total path:

[0121]

[0122] Among them, D' [i][j] From the boundary point F i To the boundary point F j distance; [i] is the boundary point set F={F1,F2,…,F q The i-th boundary point F in i As the starting point for distance calculation; [j] is the jth boundary point F in the boundary point set F j , as the end point of distance calculation; when the robot moves from the boundary point F i Arrival at boundary point F j When x ij =1; when the robot does not leave the boundary point F i Arrival at boundary point F j When x ij =0, which helps the robot identify the explored area during path planning.

[0123] d) Use genetic algorithms to solve the dynamic traveling salesman problem to calculate the global planning path of the robot's exploration.

[0124] Furthermore, solving the dynamic traveling salesman problem is as follows:

[0125] a) First, define the priority of waypoints in the set as higher than that of ordinary boundary points, classify the priority of the boundary point set F, and quantify the weight difference between key waypoints and ordinary boundary points;

[0126] b) Generate a high-priority waypoint detection sequence as the parent generation through a roulette wheel selection mechanism, perform path segment intersection and random edge point position mutation to generate offspring;

[0127] c) Introducing a rigid constraint mechanism, setting the fitness of individuals that do not contain mandatory waypoints to zero and eliminating them, while retaining the current optimal solution and fixing its path segment;

[0128] d) As the robot moves, the boundary point set F and the distance matrix D' are updated in real time, and new nodes are dynamically integrated into the optimization process, gradually expanding the path coverage through incremental solution.

[0129] In this embodiment, in order to generate a local fine path that can be used for robot motion control, the real-time position of the robot and the point cloud information near the robot are used, such as Figure 6 As shown in the figure, local path planning is specifically divided into the following steps:

[0130] a) Define the viewpoint set V = {v1, v2, ... v n Each viewpoint is configured with fan-shaped field of view parameters, radius r, center angle θ, number of viewpoints on the local path n, and establishes the benchmark value of the unobstructed field of view area

[0131] b) Obtain the field of view overlap area R and obstacle occlusion area O through discretized grid analysis and calculate the effective coverage area

[0132] c) Construct the viewpoint distance matrix D, with the dual objectives of maximizing coverage area and minimizing path;

[0133]

[0134] where d [i][j] Represents the viewpoint v i To viewpoint v j distance.

[0135] d) The A* algorithm is used to generate the optimal path between viewpoints, and combined with the solution of the traveling salesman problem to achieve a local planning trajectory with satisfactory coverage.

[0136] In this embodiment, in step S3, based on the global and local paths obtained in step S2, the global path is clipped. The global path originally within the local planning range is replaced with the local path generated in step S2, thus connecting the global and local paths and collecting point cloud information of the environment. Specifically, step S3 involves dynamically updating the path plan during the robot's exploration process to construct a detailed three-dimensional map of the exploration area.

[0137] Furthermore, the detailed 3D map of the exploration area is constructed as follows:

[0138] By acquiring point cloud data of the surrounding environment and then processing it, a point cloud registration algorithm is used to estimate the sensor's motion between consecutive frames, enabling real-time pose estimation. Simultaneously, the point cloud data of the current frame is fused into the global map according to the estimated pose, constructing a detailed 3D map of the exploration area.

[0139] Example 1:

[0140] In this embodiment, a collaborative air-ground exploration method in an unknown environment based on three-dimensional Gaussian splashing is applied to the map exploration field of industrial mobile robots. Its purpose is to overcome the shortcomings of existing technologies and guide ground robots to autonomously explore unknown environments and establish environmental maps.

[0141] The three-dimensional Gaussian splash algorithm was transplanted to a UAV in a simulation environment for experiments. The specific parameters are shown in Table 1. Figure 7 The RGB image shown is Figure 8 The depth image shown is used as multimodal input data, and the drone reconstructs the depth image in real time during the exploration process. Figure 9 The 3D map shown in the figure is used to generate waypoints for the ground robot, which in turn drives the ground robot to make refined exploration decisions.

[0142] Table 1

[0143] Parameter name Parameter representation Parameter value Mapping frequency map_every 1 Keyframe selection frequency keyframe_every 5 The number of iterations of the tracking process tracking_iters 40 The number of iterations of the mapping process mapping_iters 60 The maximum ratio of scene radius to depth scene_radius_depth_ratio 3 Calculation method of mean square distance mean_sq_dist_method projective Gaussian distribution type gaussian_distribution isotropic Time index of the checkpoint checkpoint_time_idx 100 Adding contour thresholding of new Gaussian voxels sil_thres 0.5 Learning rate decrease exponentially decay_rate_mapping 0.95 Learning rate decrement step size decay_steps_mapping 10 Threshold for rotation between keyframes threshold_trans 0.01 Threshold for moving between keyframes threshold_rot 1

[0144] The experimental environment is as follows Figure 10 shown.

[0145] The main parameters of the ground robot are shown in Table 2. The point cloud registration algorithm is used to construct the point cloud representation of the simulation environment. Figure 11 .

[0146] Table 2

[0147]

[0148]

[0149] The experimental process is as follows: the drone first enters the experimental building environment to explore, and the camera on the drone feeds back environmental information, marking the identified room areas as key areas that need to be explored in detail. The marking process is as follows: Figure 12 As shown in the figure, the ground robot performs global path planning based on the waypoints provided by the UAV when exploring the experimental environment. The planning process is as follows: Figure 13 As shown; and finally execute the exploration trajectory covering the entire area, the trajectory visualization is as follows Figure 14 shown.

[0150] The present invention also provides an air-ground collaborative exploration system based on an unknown environment, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, it can implement any of the method steps described above.

[0151] The present invention also provides a computer-readable storage medium on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, any of the method steps described above can be implemented.

[0152] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A method for collaborative air-ground exploration in an unknown environment based on three-dimensional Gaussian splashing, characterized in that: The following steps are involved: Step S1: Use a drone to acquire image data of the area to be explored, use a 3D Gaussian distribution to describe the complex scene, combine differentiable rendering technology, and integrate a simultaneous localization and mapping (SLAM) system to accurately estimate the drone camera pose and reconstruct the 3D environment in real time, traverse the space, and mark a series of key waypoints. Step S2: The ground robot uses the waypoints provided by the drone and combines the global planned path with the local planned path to achieve accurate exploration of the local three-dimensional environment. Step S3: Dynamically update the path planning during the exploration process by the ground robot to build a detailed three-dimensional map of the exploration area.

2. The method for collaborative air-ground exploration in an unknown environment based on three-dimensional Gaussian splashing according to claim 1 is characterized in that: Use three-dimensional Gaussian distribution to describe complex scenes, specifically: 3D Gaussian is a mathematical tool for describing the distribution of three-dimensional spatial data. Its standard form is: Among them, (x, y, z) are the coordinate components of the point in three-dimensional space, (μ x ,μ y ,μ z ) is the coordinate of the center point, (σ x ,σ y ,σ z ) is the standard deviation in three directions, describing the probability density of the three-dimensional normal distribution; When using a 3D Gaussian distribution to describe complex scenes, a 3D point cloud is recovered from a set of 2D images using the structure-from-motion technique. Each point in the point cloud is converted into a 3D Gaussian distribution, including parameters such as position, color, covariance matrix, and opacity. The covariance matrix describes the shape and direction of the Gaussian distribution in 3D space. To be more suitable for scene description, the 3D Gaussian distribution is represented as follows: Where x is any point in three-dimensional space; μ∈R 3 are the coordinates of the center point of the three-dimensional Gaussian distribution, which are consistent with the coordinates of the center point in the standard form of formula (1); r is the radius, which is used to describe the range of the distribution; θ∈[0,1] is the opacity, which controls the visibility of the distribution.

3. The air-ground collaborative exploration method in an unknown environment based on three-dimensional Gaussian splashing according to claim 1 is characterized in that: Use differentiable rendering technology to do the following: Using a set of 3D Gaussian distributions and the camera pose, we first sort all Gaussian distributions along the depth direction, and then perform an alpha blending operation on the 2D projections of each Gaussian distribution in depth order in pixel space, effectively synthesizing the final color value of each pixel w = (u, v): Where C(w) is the final color value of pixel w, u is the horizontal coordinate value of the pixel point, and v is the vertical coordinate value of the pixel point; i is each Gaussian distribution in the traversal set n, and n is the set of three-dimensional Gaussian distributions involved in rendering; j is the Gaussian distribution before traversal i, that is, 1 to i-1, to determine the part of pixel w that is not covered by the previous Gaussian distribution; c i is the color value of the i-th Gaussian distribution, and Q i (w) multiplication represents the color contribution of the i-th Gaussian distribution to pixel w; Q i The calculation method of (w) is the same as that of the Gaussian function, which is the color contribution weight of the i-th Gaussian distribution at pixel w; Q j (w) is the color contribution weight of the j-th Gaussian distribution at pixel w, which is similar to 1-Q j (w) jointly calculate the uncovered degree of the previous Gaussian distribution for pixel w; is the part of the pixel not covered by the previous j=1 to i-1 Gaussian distributions when calculating the color contribution of the i-th Gaussian distribution to the pixel w, μ and r are the two-dimensional Gaussian functions splashed in the pixel space: Among them, μ 2D is the center coordinate of the two-dimensional Gaussian distribution spliced ​​in the pixel space, K is the camera internal parameter, E λ is the external parameter of camera rotation and translation at the λth frame, μ is the coordinate of the center point of the three-dimensional Gaussian distribution, r 2D is the radius of the two-dimensional Gaussian distribution spliced ​​in the pixel space, which is used to describe the range of the two-dimensional Gaussian distribution in the pixel space; f r is the focal length, d is the depth of the i-th Gaussian value in camera coordinates; differentiable rendering technology is used: Where D(w) is the depth value of the pixel under the differentiable depth rendering technology obtained by accumulating the depth contribution of all Gaussian distributions to the pixel w; i is each Gaussian distribution in the traversal set n, and n is the set of three-dimensional Gaussian distributions involved in the rendering; d i is the depth parameter associated with the i-th Gaussian distribution, and Q i (w) multiplication represents the depth contribution of the i-th Gaussian distribution to pixel w; Q i The calculation method of (w) is the same as that of the Gaussian function, which is the weight of the i-th Gaussian distribution contributing to the pixel depth at pixel w; Q j (w) is the pixel depth contribution weight of the j-th Gaussian distribution at pixel w, which is the same as 1-Q j (w) jointly calculate the uncovered degree of the previous Gaussian distribution for pixel w; To indicate the part of the pixel not covered by the previous j = 1 to i-1 Gaussian distributions when calculating the depth contribution of the i-th Gaussian distribution to the pixel w; Compare with the input depth map and return the gradient relative to the 3D map; and use differentiable rendering technology to obtain the silhouette image to determine visibility: Where S(w) is the visibility value at pixel w, i is each Gaussian distribution in the traversal set n, n is the set of three-dimensional Gaussian distributions involved in rendering, Q i The calculation method of (w) is the same as that of the Gaussian function, which is the transparency contribution weight of the i-th Gaussian model at pixel w; Q j (w) is the transparency contribution weight of the i-th Gaussian model at pixel w, which is similar to 1-Q j (w) jointly calculate the uncovered degree of the previous Gaussian distribution for pixel w; When calculating the visibility contribution of the i-th Gaussian distribution to the pixel w, the portion of the pixel that is not covered by the previous j=1 to i-1 Gaussian distributions.

4. The method of space-ground collaborative exploration in an unknown environment based on three-dimensional Gaussian splashing according to claim 3 is characterized in that: The simultaneous positioning and mapping system is implemented as follows: a) Use a drone to acquire the first frame of RGB-D data of the area to be explored, extract the RGB image, depth image, and camera intrinsic parameters, initialize the camera pose to the unit matrix, and normalize the RGB image. Use the camera intrinsics to participate in the unit pose matrix (external parameters) to calculate the transformation matrix from the camera to the world coordinate system. Then, based on the depth image information and the camera intrinsic and extrinsic parameters, construct the initial 3D point cloud; Associate a Gaussian model with each point cloud point, calculate the model parameters through depth projection, and form an initial three-dimensional Gaussian model set; b) Based on the current drone camera pose and Gaussian model parameters, differentiable rendering technology is used to efficiently render and generate virtual RGB images and depth images. Pixels within the visible contour map are evaluated, and a tracking loss function consisting of a weighted sum of RGB loss and depth loss is constructed. The differentiable nature of differentiable rendering is exploited to calculate the gradient of the loss function with respect to the camera pose through backpropagation. Based on the gradient, continuous iterative optimization is performed to adjust the camera pose parameters to minimize the difference between the rendered image and the actual observed image. In each iteration, the system updates the pose according to the gradient direction, gradually approaching the true pose and achieving an accurate estimation of the camera pose. c) Generate a contour mask to identify areas with insufficient coverage or geometric holes in the map, generate a depth difference mask to detect areas where the rendered depth deviates significantly from the observed depth, fuse the contour coverage mask with the depth difference mask to locate the updated area, and expand the Gaussian model according to the proximity rule to ensure that the map accurately describes the scene and is dynamically updated; d) Based on disparity, scene change, and overlap rate, key frames are selected as shown in Equation (7), and parameter block selective joint optimization is performed. Starting from the most recently constructed map, frames that may affect the newly added Gaussian function are optimized; finally, the camera pose and map are jointly optimized to ensure that the map is consistent with the current observation. As new data is input, an accurate and dense 3D scene representation is gradually constructed; Among them, Keyframe is a key frame set, which is used to filter key frames that meet the conditions; P I is the percentage of points remaining inside the image, which is used to measure the proportion of valid points in the current frame, i.e., points that do not exceed the image range, to ensure that the key frame contains sufficient scene information; k is the translation change value of the kth frame, tra k+1 is the translation change value of the k+1th frame; tra τ is the threshold of translation change, which is used to judge whether the translation change between two adjacent frames is large enough. k+1 -tra k )>tra τ When , it means that the translation change between the two frames is significant; rot k is the rotation change value of the kth frame, rot k+1 is the rotation change value of the k+1th frame; rot τ The rotation change threshold is used to determine whether the rotation change between two adjacent frames is large enough. k+1 -rot k )>rot τ When , it indicates that the rotation change between the two frames is significant; Through the organic combination of three-dimensional Gaussian distribution to describe complex scenes, differentiable rendering technology, and simultaneous positioning and mapping systems, as the image data obtained by the drone exploring unknown areas is added, the process of data acquisition, pose estimation, map update and joint optimization is continuously promoted, achieving accurate estimation of the drone camera pose and real-time reconstruction of the three-dimensional environment. The reconstruction results are gradually improved with the exploration of the drone, and ultimately an accurate and dense three-dimensional scene representation is constructed, providing reliable support for air-ground collaborative exploration in unknown environments.

5. The method for collaborative air-ground exploration in an unknown environment based on three-dimensional Gaussian splashing according to claim 1 is characterized in that: The global planning path is as follows: a) Define the boundary point f at the junction of the known area and the unknown area, and also consider the waypoint w as the boundary point. Define the boundary point set F as the union of the boundary of the known / unknown area and the drone waypoints F = {f1,f2,…f m ,w1,w2,…w l }={F1,F2…F q }, construct a sparse random roadmap covering the subspace to be explored; b) Based on the A* algorithm, the shortest path from the current viewpoint to each boundary point is calculated in parallel, the path distance matrix D' is generated and mapped to the fitness weight ada; Among them, D' [i][j] From the boundary point F i To the boundary point F j distance; a ij is the element D' in the distance matrix [i][j] The reciprocal of a ij =1 / D' [i][j] , used to measure the fitness in path planning. The smaller the distance, the larger the reciprocal, and the higher the fitness; c) Define the binary decision variable x ij Mark the boundary point access status, construct the fitness function with the inverse of the path distance, and construct the objective optimization function that minimizes the total path: Among them, D' [i][j] From the boundary point F i To the boundary point F j distance; [i] is the boundary point set F={F1,F2,…,F q The i-th boundary point F in i As the starting point for distance calculation; [j] is the jth boundary point F in the boundary point set F j , as the end point of distance calculation; when the robot moves from the boundary point F i Arrival at boundary point F j When x ij =1; when the robot does not leave the boundary point F i Arrival at boundary point F j When x ij =0, which helps the robot identify the explored area during path planning; d) Use genetic algorithms to solve the dynamic traveling salesman problem to calculate the global planning path trajectory of the robot exploration.

6. The method of space-ground collaborative exploration in an unknown environment based on three-dimensional Gaussian splashing according to claim 1 is characterized in that: The local planning path is as follows: a) Define the viewpoint set V = {v1, v2, ... v n Each viewpoint is configured with fan-shaped field of view parameters, radius r, center angle θ, number of viewpoints on the local path n, and establishes the benchmark value of the unobstructed field of view area b) Obtain the overlap area R of the field of view and the area blocked by obstacles O through discretized grid analysis, and calculate the effective coverage area c) Construct the viewpoint distance matrix D, with the dual objectives of maximizing coverage area and minimizing path; where d [i][j] Represents the viewpoint v i To viewpoint v j distance; d) The A* algorithm is used to generate the optimal path between viewpoints, and the traveling salesman problem is combined to calculate the local planning path trajectory that meets the coverage requirement.

7. The method for collaborative air-ground exploration in an unknown environment based on three-dimensional Gaussian splashing according to claim 5 is characterized in that: The solution to the dynamic traveling salesman problem is: a) First, define the priority of waypoints in the set as higher than that of ordinary boundary points, classify the priority of the boundary point set F, and quantify the weight difference between key waypoints and ordinary boundary points; b) Generate a high-priority waypoint detection sequence as the parent generation through a roulette wheel selection mechanism, perform path segment intersection and random edge point position mutation to generate offspring; c) Introducing a rigid constraint mechanism, setting the fitness of individuals that do not contain mandatory waypoints to zero and eliminating them, while retaining the current optimal solution and fixing its path segment; d) As the robot moves, the boundary point set F and the distance matrix D' are updated in real time, and new nodes are dynamically integrated into the optimization process, gradually expanding the path coverage through incremental solution.

8. The air-ground collaborative exploration method in an unknown environment based on three-dimensional Gaussian splashing according to claim 1 is characterized in that: A detailed 3D map of the exploration area is constructed as follows: The point cloud data of the surrounding environment is acquired through LiDAR and then processed. The point cloud registration algorithm is used to estimate the movement of the sensor between consecutive frames to achieve real-time pose estimation. At the same time, the point cloud data of the current frame is fused into the global map according to the estimated pose to construct a three-dimensional point cloud map of the environment.

9. An air-ground collaborative exploration system in an unknown environment based on three-dimensional Gaussian splashing, characterized by: The method comprises a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.

10. A computer-readable storage medium storing computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.

Images