Robot environment sensing method based on three-dimensional point cloud rendering

Abstract

The invention discloses a robot environment sensing method based on three-dimensional point cloud rendering. The robot environment sensing method comprises the following steps: collecting a multi-source environment data set; transmission and real-time synchronization are carried out on the multi-source environment data set among the robot end, the server end and the client end; constructing a three-dimensional environment rendering model at a client by combining a DS-NeRF model and an SLAM algorithm; generating an initial navigation path in the three-dimensional environment rendering model and dynamically displaying the initial navigation path; obstacles and deviation are detected in the navigation process, and navigation path data are generated; generating environment semantic representation based on the three-dimensional environment rendering model and the navigation path data; constructing a task arrangement diagram at the client, and generating a task execution state; and generating an environment perception result based on the three-dimensional environment rendering model, the navigation path data, the environment semantic representation and the task execution state. According to the method, the three-dimensional environment modeling precision and the path dynamic updating capability are improved, and efficient environment perception and stable autonomous navigation are realized.

Description

Technical Field

[0001] This invention relates to the field of environmental perception for mobile robots, and more particularly to a robot environmental perception method based on 3D point cloud rendering. Background Technology

[0002] Current robot environmental perception technologies mostly rely on single sensor data acquisition and 2D map construction, commonly using LiDAR point clouds or camera images for scene representation. While these methods are feasible in structured environments, they are prone to problems such as insufficient depth information, loss of 3D details, and limited semantic expression capabilities in complex scenes, making it difficult to support high-precision environmental modeling and interaction.

[0003] In map generation and navigation path planning, traditional algorithms often rely on two-dimensional grid maps for path calculation, lacking a comprehensive depiction of three-dimensional space. This leads to problems such as insufficient path planning accuracy and delayed obstacle avoidance in dynamic environments. Furthermore, path updates are often delayed, failing to meet the rapid response requirements of robots in real-time perception and highly dynamic scenarios.

[0004] At the user interaction and task execution configuration level, existing methods typically rely on preset scripts or fixed logic control, lacking intuitive task orchestration methods and hindering non-professional users from quickly configuring and adjusting task flows. This approach makes robots insufficiently flexible in changing task environments, making it difficult to achieve effective linkage between environmental perception and task execution.

[0005] Therefore, how to provide a robot environment perception method based on 3D point cloud rendering is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a robot environment perception method based on 3D point cloud rendering. This invention collects LiDAR point cloud data, depth camera image data, and positioning data, utilizes WebSocket to achieve low-latency synchronization across multiple devices, and constructs a depth-supervised 3D environment rendering model on the client side by combining the DS-NeRF model and SLAM algorithm. This supports interactive setting of waypoints and dynamic updating of navigation paths by the user. At the same time, a task orchestration graph can be created on the client side by dragging and dropping, binding environmental events with node execution logic to achieve linkage between environmental perception and task execution. This method has the advantages of high perception accuracy, intuitive interaction, and strong adaptability.

[0007] A robot environment perception method based on 3D point cloud rendering according to an embodiment of the present invention includes the following steps: Collect lidar point cloud data, depth camera image data, and IMU fusion positioning data to generate a multi-source environmental data set; The multi-source environmental data set is transmitted and synchronized in real time between the robot, server and client through the communication link to generate a synchronized environmental data set; The client renders the synchronous environment dataset, combines the DS-NeRF model to perform multi-source deep supervised modeling of the multi-source environment dataset, performs streaming incremental training and updates, and combines the SLAM algorithm to generate a 3D environment rendering model. The robot receives the target waypoint set by the user in the 3D environment rendering model, generates the initial navigation path from the robot's current position to the target waypoint, and dynamically displays the initial navigation path in the 3D environment rendering model. During navigation, based on a synchronous environmental data set, the deviation between real-time obstacles and the initial navigation path is detected, and navigation path data is generated. Based on the 3D environment rendering model and navigation path data, obstacle distribution information, free area information and passable area information are extracted to generate environmental semantic representation; On the client side, a task orchestration graph is constructed and bound to events in the semantic representation of the environment to generate task execution status. Based on the 3D environment rendering model, navigation path data, environmental semantic representation, and task execution status, environmental perception results are generated.

[0008] Optionally, the generation of the synchronization environment dataset includes: On the robot side, the multi-source environmental data set is segmented and timestamped according to a preset data format. The segmented multi-source environmental data set is then transmitted to the server via a WebSocket communication link. The server receives the fragmented multi-source environmental data set, performs integrity verification and order reassembly on the multi-source environmental data set, maintains the real-time connection status of the WebSocket communication link based on the heartbeat detection mechanism, and forwards the reassembled multi-source environmental data set to the client. The client receives a set of multi-source environmental data forwarded by the server, performs packet loss detection and timing calibration on the multi-source environmental data set, and completes real-time synchronization of the multi-source environmental data set based on a low-latency transmission mechanism to generate a synchronized environmental data set.

[0009] Optionally, the generation of the 3D environment rendering model includes: The client receives a set of synchronous environment data and performs format conversion on the lidar point cloud data and depth camera image data in the synchronous environment data set to generate an input data set. The client calls the DS-NeRF model, using the LiDAR point cloud data and depth camera image data in the input dataset as multi-source supervision signals to perform multi-source depth supervision modeling and generate the initial 3D point cloud rendering result; The specific process of the multi-source depth supervision modeling is as follows: First, the lidar point cloud data is projected and registered to obtain sparse depth constraints; at the same time, the pixel parallax in the depth camera image data is converted into dense depth constraints; then, during the training process of the DS-NeRF model, the sparse depth constraints and dense depth constraints are added to the depth supervision loss function and the volume rendering loss function to generate a joint optimization objective. During the continuous input of the synchronous environment dataset, the DS-NeRF model is subjected to streaming incremental training and updates. The new synchronous environment dataset is jointly calculated with the training parameters of the previous stage, and the updated 3D point cloud rendering results are iteratively output. The model is then gradually optimized into a 3D point cloud rendering model with depth supervision. The SLAM algorithm is run based on a synchronous environmental dataset to generate a two-dimensional grid map and robot trajectory. The two-dimensional raster map is fused with a three-dimensional point cloud rendering model with depth supervision to generate a three-dimensional environment rendering model, and the dynamic display output of the three-dimensional environment rendering model is completed on the client side.

[0010] Optionally, the generation of the initial navigation path includes: In the 3D environment rendering model on the client, the target waypoint set by the user interaction is received, and the current position of the robot is obtained based on the robot trajectory; In the 3D environment rendering model, the passable area is discretized to generate a set of nodes, and connections are established between the node sets based on spatial adjacency and a cost is assigned to the connection. Starting from the robot's current position and ending at the target waypoint, the A* algorithm is used to perform a search on the set of nodes and connections to generate an initial navigation path from the robot's current position to the target waypoint. The initial navigation path is written into the dynamic display layer of the 3D environment rendering model on the client side, and then dynamically displayed in the 3D environment rendering model.

[0011] Optionally, the generation of the navigation path data includes: During navigation, the robot's current position is obtained based on the synchronous environmental data set, and the shortest path segment of the initial navigation path is located in the 3D environment rendering model; Based on the synchronous environmental dataset, the obstacle positions and passable areas are updated in the 3D environment rendering model, and the occupancy verification results of the initial navigation path are generated. The deviation value is calculated based on the spatial relationship between the robot's current position and the initial navigation path in the 3D environment rendering model, and then compared with a threshold value. When the deviation value is greater than the threshold and the occupation verification result indicates that the initial navigation path is blocked by an obstacle, a search is performed on the node set and connection to generate a new navigation path in the 3D environment rendering model. In the 3D environment rendering model, the display of the initial navigation path is replaced with the new navigation path, and the new navigation path is associated with the target waypoint and the robot's current position to generate navigation path data.

[0012] Optionally, the generation of the environmental semantic representation includes: In the 3D environment rendering model, spatial point sets are classified, and obstacle distribution information and free area information are generated based on the occupancy state of the 3D environment rendering model. Based on the navigation path data, connectivity filtering is performed within the free area information. A subset of free areas connected to the navigation path data is extracted as passable area information, and elements that intersect with obstacle distribution information or whose distance from obstacle boundary is less than a preset safety distance are removed from the passable area information. By establishing a mapping relationship between obstacle distribution information, free area information, and passable area information and navigation path data according to a unified coordinate index structure, an environmental semantic representation is generated.

[0013] Optionally, the generation of the task execution status includes: Create a task orchestration graph on the client side, and load the 3D environment rendering model, navigation path data and environment semantic representation as input references into the task orchestration graph; The creation of a task orchestration diagram on the client includes selecting trigger nodes, content nodes, action nodes, and waiting nodes by dragging and dropping them onto a visual canvas, establishing execution relationships by connecting them, and allowing users to complete event binding, action selection, and delay settings in the property panel. The operation results are displayed in real time and synchronized to the 3D environment rendering model. Add trigger nodes, content nodes, action nodes, and waiting nodes to the task orchestration diagram, assign a unique identifier to each node, and establish data association with the 3D environment rendering model, navigation path data, and environmental semantic representation. The trigger node is bound to an event in the environmental semantic representation, which includes reaching the target waypoint, updating the navigation path, and entering the specified area; Set the node execution relationship in the task orchestration diagram. After the trigger node is activated, the content node and action node are executed sequentially. After the action node is completed, it enters the waiting node. After the waiting node is completed, it returns to the standby entry of the task orchestration diagram. The client runs a task orchestration graph and subscribes to event updates and navigation path data changes in the environmental semantic representation. It then executes the event-driven nodes based on event bindings and updates the interactive display in the 3D environment rendering model. The subscription indicates that the client declares that it wants to receive real-time updates of event changes and navigation path data in the environmental semantic representation; Generate a task execution status, which includes the currently triggered event and the identifier of the node being executed.

[0014] Optionally, the generation of the environmental perception results includes: The client loads the 3D environment rendering model, navigation path data, environment semantic representation and task execution status, and completes the consistency verification of timestamp and version number; Under a unified coordinate index structure, obstacle distribution information in the environmental semantic representation is aligned and associated with the 3D environment rendering model and navigation path data; On the client side, the display of the 3D environment rendering model is refreshed, the navigation path data is overlaid, and the area is highlighted with obstacle distribution information; Generate environmental perception results, which include a 3D environment rendering model, obstacle distribution information and navigation path data, and establish a correlation with the task execution status; The system publishes environmental perception results to the server and the robot, and saves and queries the environmental perception results on the client side.

[0015] The beneficial effects of this invention are: First, this invention introduces the DS-NeRF model to perform multi-source depth supervision modeling of LiDAR point cloud data and depth camera image data, and combines it with a two-dimensional grid map generated by the SLAM algorithm to achieve fusion rendering, thus constructing a three-dimensional environment rendering model with depth supervision. This effectively improves the accuracy of robot environment perception and detail restoration capability, overcoming the problem of insufficient three-dimensional perception accuracy in existing technologies.

[0016] Secondly, this invention supports user-interactive setting of target waypoints on the client side, and displays and updates the navigation path in real time based on the 3D environment rendering model. It combines the A* algorithm and deviation detection mechanism to realize dynamic adjustment of the path, ensuring the navigation stability and robustness of the robot in complex environments, and making up for the lack of real-time update capability of navigation paths in existing methods.

[0017] Furthermore, this invention allows users to create task orchestration diagrams on the client side via drag-and-drop, intuitively binding events in the environmental semantic representation with task nodes. This enables environmental perception results to directly drive task logic execution, lowers the user configuration threshold, and enhances the system's adaptability and interactivity in multiple scenarios, thereby achieving a deep integration of environmental perception and task execution. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0019] Figure 1 This is an overall flowchart of a robot environment perception method based on 3D point cloud rendering proposed in this invention. Figure 2 This is a schematic diagram of the structure of the 3D environment rendering model with depth supervision constructed by fusing the DS-NeRF model and the SLAM algorithm in this invention. Figure 3 This is a schematic diagram of the navigation path dynamic update and obstacle detection mechanism based on a 3D environment rendering model in this invention. Detailed Implementation

[0020] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0021] refer to Figure 1-3 A robot environment perception method based on 3D point cloud rendering includes the following steps: Collect lidar point cloud data, depth camera image data, and IMU fusion positioning data to generate a multi-source environmental data set; The WebSocket communication link enables low-latency, high-reliability transmission and real-time synchronization of multi-source environmental data sets between the robot, server, and client, generating a synchronized environmental data set. On the client side, the synchronous environment dataset is rendered using the Three.js rendering engine. The DS-NeRF model is used to perform multi-source depth-supervised modeling of the LiDAR point cloud data and depth camera image data in the multi-source environment dataset. Streaming incremental training and updates are performed during continuous sensor input to build a depth-supervised 3D point cloud rendering model. At the same time, the SLAM algorithm is used to generate a 2D grid map and robot trajectory. The 2D grid map and the depth-supervised 3D point cloud rendering model are then fused to generate a 3D environment rendering model. The robot receives the target waypoint set by the user in the 3D environment rendering model, generates the initial navigation path from the robot's current position to the target waypoint based on the A* algorithm, and dynamically displays the initial navigation path in the 3D environment rendering model. During navigation, based on the synchronous environmental data set, the deviation between real-time obstacles and the initial navigation path is detected. When the deviation is greater than the threshold, the navigation path is regenerated and updated in the 3D environment rendering model to generate navigation path data. Based on the 3D environment rendering model and navigation path data, obstacle distribution information, free area information and passable area information are extracted to generate environmental semantic representation; In the client, a task orchestration graph is constructed by dragging and dropping. The task orchestration graph includes trigger nodes, content nodes, action nodes, and waiting nodes. The trigger nodes are bound to events in the environmental semantic representation. The events include reaching the target waypoint, updating the navigation path, and entering the specified area. The task execution status is generated. Based on the 3D environment rendering model, navigation path data, and environmental semantic representation, an environment perception result is generated, which includes the 3D environment rendering model, obstacle distribution information, and navigation path data.

[0022] In this embodiment, the generation of the synchronization environment data set includes: On the robot side, the multi-source environmental data set is segmented and timestamped according to a preset data format. The segmented multi-source environmental data set is then transmitted to the server via a WebSocket communication link. The server receives the fragmented multi-source environmental data set, performs integrity verification and order reassembly on the multi-source environmental data set, maintains the real-time connection status of the WebSocket communication link based on the heartbeat detection mechanism, and forwards the reassembled multi-source environmental data set to the client. The client receives a set of multi-source environmental data forwarded by the server, performs packet loss detection and timing calibration on the multi-source environmental data set, and completes real-time synchronization of the multi-source environmental data set based on a low-latency transmission mechanism to generate a synchronized environmental data set. The synchronized environment data set is returned to the robot and the server to ensure that the synchronized environment data set between the robot, the server and the client always maintains consistency and real-time synchronization, forming a two-way interactive data update mechanism.

[0023] In this embodiment, the generation of the three-dimensional environment rendering model includes: The client calls the Three.js rendering engine to receive the synchronous environment data set, and performs format conversion on the LiDAR point cloud data and depth camera image data in the synchronous environment data set to generate the input data set for depth supervised modeling; The client calls the DS-NeRF model, using the LiDAR point cloud data and depth camera image data in the input dataset as multi-source supervision signals to perform multi-source depth supervision modeling and generate the initial 3D point cloud rendering result; The DS-NeRF model is an improved version of the Depth-SupervisedNeRF model; The core improvement lies in the introduction of multi-source depth supervision and online adaptation for robot environmental perception. Traditional DS-NeRF models typically rely on sparse depth information from a single source as a supervision signal. However, this method simultaneously introduces LiDAR point cloud data and depth camera image data, jointly constraining the two types of depth information with different characteristics. This allows the rendering of 3D geometry to balance the spatial density of sparse point clouds and the local texture consistency of camera images, thereby improving the accuracy and robustness of 3D reconstruction. At the same time, this method adapts the DS-NeRF model on the client side, enabling it to achieve streaming incremental training and updates with continuous sensor input, no longer relying on the traditional offline training mode. This ensures that the 3D environment rendering results can be optimized in real time as the environment dynamically changes during robot operation. The specific process of the multi-source depth-supervised modeling is as follows: First, the LiDAR point cloud data is projected and registered to obtain sparse depth constraints; at the same time, the pixel parallax in the depth camera image data is converted into dense depth constraints; then, during the training of the DS-NeRF model, the sparse depth constraints and dense depth constraints are added to the depth supervision loss function and the volume rendering loss function to generate a joint optimization objective; in this way, the model can simultaneously satisfy appearance consistency and geometric consistency when generating 3D point cloud rendering results, thereby gradually optimizing to obtain a 3D point cloud rendering model with depth supervision. The deep supervision loss function is specifically as follows: ; ; ; ; in, This represents the depth-supervised loss function, used to measure the difference between the predicted depth and the true depth from multiple sources; This represents the set of rays, indicating all rays emitted from the camera or sensor; The index represents a ray, and the ray represents a single ray in the ray set. These represent weighting coefficients, used to balance the importance of the lidar point cloud data supervision term and the depth camera image data supervision term, respectively. This represents an availability mask function, used when a LiDAR or depth camera is in a ray... The value is 1 if there is a valid depth, otherwise it is 0. This represents the Huber function, used to reduce the impact of outliers on training. This represents the error term between the predicted depth and the actual depth. This represents the Huber threshold, which controls the boundary for switching between L1 and L2 errors; Indicates along the ray The expected value of the predicted depth; This indicates the true depth provided by the lidar point cloud data; This represents the true depth provided by the depth camera image data; Indicates along the ray The The depth location of each sampling point; Indicates in the ray The Normalized termination probability distribution of each sampling point; Indicates in the ray The Volume rendering weights for each sampling point; The volume rendering loss function is specifically as follows: ; ; in, The volume rendering loss function is used to measure the difference between the rendered image and the real image and to enhance geometric consistency. The weighting parameter represents the color consistency item; The weighting parameters represent the depth variance constraint term; Indicates along the ray Predicted rendered colors; Indicates along the ray The true observed color; This represents the L1 difference between the predicted color and the actual color; This represents the depth variance term, used to constrain the ray termination distribution to be close to a single peak, ensuring a clear geometric surface. Represents rays Upper The color vector of each sampling point; This represents the expected value of the prediction depth (consistent with the depth-supervised loss function); During the continuous input of the synchronous environment dataset, the DS-NeRF model is subjected to streaming incremental training and updates. The new synchronous environment dataset is jointly calculated with the training parameters of the previous stage, and the updated 3D point cloud rendering results are iteratively output. The model is then gradually optimized into a 3D point cloud rendering model with depth supervision. Specifically, when a new synchronous environment dataset is input, the entire system is not retrained. Instead, local iterative optimization is performed on the existing training parameters using small-batch data updates. A sliding window mechanism and exponential decay strategy are used to fuse historical data distributions, thus maintaining real-time training while preventing model forgetting. During joint computation, the new synchronous environment dataset is combined with the training parameters from the previous stage within the same optimization framework, preserving the stability of the previous stage's weight distribution. Only the local parameters corresponding to the new input are corrected to prevent model mutations. During training, the incremental update loss function includes both a depth-supervised loss function and a volume rendering loss function to ensure that the geometric constraints and color consistency provided by the LiDAR point cloud data and depth camera image data work together to gradually converge the 3D point cloud rendering results. After each iteration, the client can not only output the updated 3D point cloud rendering results in real time but also continuously optimize it into a depth-supervised 3D point cloud rendering model, providing high-precision input for the subsequent 3D environment rendering model construction. The SLAM algorithm is run based on a synchronous environmental dataset to generate a two-dimensional grid map and robot trajectory. The two-dimensional raster map is fused with a three-dimensional point cloud rendering model with depth supervision to generate a three-dimensional environment rendering model, and the dynamic display output of the three-dimensional environment rendering model is completed on the client. The fusion process involves the following steps: First, using IMU-fused positioning data, the coordinate systems of the 2D grid map and the 3D point cloud rendering model are aligned to ensure they are represented in the same world coordinate system. Then, using the occupied grid cells in the 2D grid map as planar constraints, the corresponding 3D point cloud rendering model is projected and corrected to ensure the spatial point distribution in the 3D point cloud rendering model is consistent with the obstacle distribution and free area boundaries in the 2D grid map. Next, a graph optimization method is used to adjust the boundaries of local areas in the 3D point cloud rendering model, eliminating drift errors caused by sensor noise. Finally, the corrected 3D point cloud rendering model is overlaid with the occupancy information of the 2D grid map. The global navigation constraints provided by the 2D grid map and the local geometric details provided by the 3D point cloud rendering model are combined to generate a 3D environment rendering model, achieving a complete representation of the robot's environment.

[0024] In this embodiment, the generation of the initial navigation path includes: In the 3D environment rendering model on the client, the target waypoint set by the user interaction is received, and the current position of the robot is obtained based on the robot trajectory; In the 3D environment rendering model, the passable area is discretized to generate a set of nodes, and connections are established between the node sets based on spatial adjacency and a cost is assigned to the connection. Starting from the robot's current position and ending at the target waypoint, the A* algorithm is used to perform a search on the set of nodes and connections to generate an initial navigation path from the robot's current position to the target waypoint. The initial navigation path is written into the dynamic display layer of the 3D environment rendering model on the client side. The initial navigation path is dynamically displayed in the 3D environment rendering model, and the initial navigation path, target waypoint, and robot's current position are associated to support deviation detection and updates in subsequent steps. The dynamic display of the initial navigation path is specifically as follows: In the client's 3D environment rendering model, the initial navigation path is written into the dynamic display layer in the form of a node sequence. The path geometry is constructed based on the Three.js rendering engine and visualized using a combination of line rendering and pipeline rendering. The direction flow effect of the path is controlled by shader parameters. At the same time, the target waypoint and the robot's current position are displayed with independent markers in the dynamic display layer. Update information of the synchronous environment data set is received through the WebSocket communication link. When a change in the initial navigation path is detected, the affected path segments are incrementally updated to ensure that the initial navigation path can be continuously and dynamically displayed in the 3D environment rendering model and remain consistent with the target waypoint and the robot's current position.

[0025] In this embodiment, the generation of navigation path data includes: During navigation, the robot's current position is obtained based on the synchronous environmental data set, and the shortest path segment of the initial navigation path is located in the 3D environment rendering model; Based on the synchronous environmental dataset, the obstacle positions and passable areas are updated in the 3D environment rendering model, and the occupancy verification results of the initial navigation path are generated. The specific process for generating the occupancy verification result is as follows: Based on the synchronous environmental data set, the obstacle positions and passable areas are updated in real time in the 3D environment rendering model. The node set and connections of the initial navigation path are mapped to the updated 3D environment rendering model. The occupancy information provided by the 2D grid map and the local geometric details provided by the 3D point cloud rendering model are used to detect the initial navigation path segment by segment. When the node set or connection overlaps with the obstacle position, it is marked as blocked. When the node set or connection falls outside the free area, it is marked as impassable. When the safe distance between the node set or connection and the boundary of the free area is less than a preset threshold, it is marked as a risk area. Finally, these detection marks are integrated to form the occupancy verification result of the initial navigation path, which is used for subsequent deviation calculation and navigation path update. The occupancy verification result indicates whether each path segment on the initial navigation path overlaps with an obstacle, whether it is located within a passable area, and whether it maintains a safe distance from the boundary of the free area. The deviation value is calculated based on the spatial relationship between the robot's current position and the initial navigation path in the 3D environment rendering model. The deviation value is compared with a threshold to determine whether to trigger a path update. When the deviation value is greater than the threshold and the occupation verification result indicates that the initial navigation path is blocked by an obstacle, a search is performed on the node set and connection based on the A* algorithm to generate a new navigation path in the 3D environment rendering model. Specifically, the process is as follows: First, based on the synchronous environmental data set, the obstacle distribution information and passable area information in the 3D environment rendering model are updated. The passable areas are then discretized within the 3D environment rendering model to form a set of nodes and a set of connections. Next, using the robot's current position as the starting node and the target waypoint as the ending node, the A* algorithm is run on the node and connection sets. A cost function simultaneously considers the navigation path length and obstacle avoidance cost, performing a heuristic search to select the path sequence with the minimum cost. Then, the obtained node set is assembled into a new navigation path according to spatial order, and the path geometry is constructed and visualized in the dynamic display layer of the 3D environment rendering model. Finally, the new navigation path is written into the navigation path data and a coordinate mapping relationship is established between the robot's current position and the target waypoint to ensure that the new navigation path can be used in real-time for subsequent navigation processes and environmental semantic representation generation. In the 3D environment rendering model, the display of the initial navigation path is replaced with the new navigation path, and the new navigation path is associated with the target waypoint and the robot's current position to generate navigation path data; The output navigation path data is used for subsequent steps of environmental semantic representation extraction and task orchestration graph event binding.

[0026] In this embodiment, the generation of the environmental semantic representation includes: In the 3D environment rendering model, spatial point sets are classified, and obstacle distribution information and free area information are generated based on the occupancy state of the 3D environment rendering model. Based on the navigation path data, connectivity filtering is performed within the free area information. A subset of free areas connected to the navigation path data is extracted as passable area information, and elements that intersect with obstacle distribution information or whose distance from obstacle boundary is less than a preset safety distance are removed from the passable area information. By establishing a mapping relationship between obstacle distribution information, free area information, and passable area information and navigation path data according to a unified coordinate index structure, an environmental semantic representation is generated.

[0027] In this embodiment, the generation of the task execution status includes: Users can create task orchestration diagrams in the client and load 3D environment rendering models, navigation path data and environmental semantic representations as input references in the task orchestration diagrams; The process of creating a task orchestration diagram on the client side includes selecting trigger nodes, content nodes, action nodes, and waiting nodes from the node component panel by dragging and dropping, placing the nodes on a visual canvas, and establishing execution relationships between nodes by dragging and dropping lines. Users can visually configure and adjust node parameters in the property panel. Node configuration includes event binding, action selection, and delay settings. All operations are displayed in real time in the graphical interface and synchronized to the 3D environment rendering model, ensuring that non-professional users can quickly complete the configuration and dynamic adjustment of the task flow without programming. Add trigger nodes, content nodes, action nodes, and waiting nodes to the task orchestration diagram, assign a unique identifier to each node, and establish data association with the 3D environment rendering model, navigation path data, and environmental semantic representation. The trigger node is bound to an event in the environmental semantic representation, which includes reaching the target waypoint, updating the navigation path, and entering the specified area; Set the node execution relationship in the task orchestration diagram. After the trigger node is activated, the content node and action node are executed sequentially. After the action node is completed, it enters the waiting node. After the waiting node is completed, it returns to the standby entry of the task orchestration diagram. The client runs a task orchestration graph and subscribes to event updates and navigation path data changes in the environmental semantic representation. It then executes the event-driven nodes based on event bindings and updates the interactive display in the 3D environment rendering model. The subscription indicates that the client declares that it will receive real-time updates of event changes and navigation path data in the environmental semantic representation, so that the task orchestration graph can automatically drive node execution and link the interactive display of the 3D environment rendering model; Generate a task execution status, which includes the currently triggered event and the identifier of the node being executed, and provide the task execution status for use in subsequent steps.

[0028] In this embodiment, the generation of the environmental perception result includes: The client loads the 3D environment rendering model, navigation path data, environment semantic representation and task execution status, and completes the consistency verification of timestamp and version number; Under a unified coordinate index structure, obstacle distribution information in the environmental semantic representation is aligned and associated with the 3D environment rendering model and navigation path data; On the client side, the display of the 3D environment rendering model is refreshed, the navigation path data is overlaid, and the area is highlighted with obstacle distribution information; The overlay display of the navigation path data transforms the abstract path planning results into an intuitive 3D rendering representation, ensuring that the operator can intuitively see the robot's expected path in the overall environment of the 3D point cloud rendering, rather than individual data entries. This facilitates user interaction. Users can interactively check the path's rationality by combining obstacle distribution information and free area information, and adjust it as necessary through task orchestration or interactive operations. Generate environmental perception results, which include a 3D environment rendering model, obstacle distribution information and navigation path data, and establish a correlation with the task execution status; The environmental perception results are published to the server and the robot via the WebSocket communication link, and the environmental perception results are saved and queried on the client side.

[0029] Example 1: To verify the feasibility of this invention in practice, it was applied to a mobile robot's environmental perception and autonomous navigation scenario in a complex indoor environment. This environment included multiple channels, dynamic obstacles, open areas, and narrow passages, and was subject to lighting variations and noise interference. In the experiment, the robot was equipped with a LiDAR sensor, a depth camera, and an IMU module to collect environmental point cloud data, image data, and positioning data, forming a multi-source environmental data set. Through a WebSocket communication link, data was synchronized with low latency between the robot, server, and client, with the average communication latency remaining below 85 milliseconds.

[0030] On the client-side rendering side, the Three.js rendering engine, combined with the DS-NeRF model, performs multi-source depth-supervised modeling of LiDAR point clouds and depth camera images, generating a depth-supervised 3D environment rendering model. Simultaneously, the SLAM algorithm is used to generate a 2D grid map and robot trajectory, ultimately forming a 3D environment rendering model highly consistent with the real environment. In this model, users can interactively set target waypoints, and the system uses the A* algorithm to generate an initial navigation path, which is displayed in real-time on the rendering interface. When the robot encounters dynamic obstacles or deviates from the navigation path during movement, the environmental perception module triggers an obstacle detection and deviation calculation mechanism. If the deviation exceeds a set threshold, the navigation path is dynamically updated. In the experiment, the deviation threshold was set to 0.25 meters. When the robot deviates from the path due to interference, the average response time for path updates was 0.16 seconds, ensuring the stability of the navigation process.

[0031] Furthermore, the client-side drag-and-drop task orchestration interface allows users to quickly configure task flows by dragging trigger nodes, content nodes, action nodes, and waiting nodes onto the canvas and binding them to environmental semantic representations. Experimenters can define complex logic such as automatically broadcasting prompts upon entering a designated area or executing actions upon reaching a target waypoint without writing code, significantly lowering the operational threshold. The average task configuration time is reduced to less than 6 minutes, compared to approximately 15 minutes using the traditional method.

[0032] In the comparative experiments, the differences between the method of this invention and the traditional 2D SLAM-based environment perception method in terms of perception accuracy, path planning success rate, navigation response latency, and task configuration efficiency were tested. A total of 120 sets of data were collected in the experiments, and the results are shown in the table below.

[0033] Table 1. Performance comparison results between the method of the present invention and the traditional method.

[0034] As shown in Table 1, the method of this invention significantly outperforms traditional methods in multiple metrics. Regarding the accuracy of 3D environment modeling, the average error is reduced from 0.28 meters to 0.15 meters. This is mainly due to the multi-source depth supervision mechanism of the DS-NeRF model effectively fusing point cloud and image data, making the rendered model closer to the real environment. In terms of navigation path success rate, this invention achieves 89%, an improvement of approximately 9 percentage points compared to traditional methods. This is due to the introduction of obstacle occupancy verification and dynamic path update mechanisms, ensuring the robot can maintain an effective path in complex environments. Regarding obstacle detection, the accuracy of this invention reaches 89%, a significant improvement compared to the 79% of traditional methods. This is attributed to the 3D rendering model's ability to preserve spatial details. In terms of response speed, the average path update response time is shortened to 0.16 seconds, and the environmental perception latency is controlled at 125 milliseconds, far lower than traditional methods, indicating that the system can quickly respond to environmental changes. At the user operation level, the drag-and-drop task orchestration interface of this invention significantly reduces configuration difficulty, improves task configuration efficiency by more than 2 times, and achieves a user interaction satisfaction score of 4.7.

[0035] The performance improvement of this invention lies in its innovations in both environmental modeling and task interaction. In the modeling process, a DS-NeRF model is introduced to perform multi-source depth-supervised modeling of LiDAR point clouds and depth camera images, and this is combined with a 2D raster map generated by the SLAM algorithm for fusion. This results in a 3D environment rendering model that significantly outperforms traditional methods in terms of spatial detail reproduction and depth consistency, thereby significantly reducing environmental modeling errors. In the path planning and updating stage, this invention achieves timely perception of dynamic obstacles and rapid reconstruction of navigation paths through occupancy verification and deviation detection mechanisms, ensuring the continuity and effectiveness of navigation paths and improving navigation success rate and system robustness. At the user interaction level, this invention adopts a drag-and-drop task orchestration method, transforming complex logical configurations into intuitive graphical operations, greatly shortening task configuration time and improving operational convenience.

[0036] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A robot environment perception method based on three-dimensional point cloud rendering, characterized in that, The method comprises the following steps: Collecting laser radar point cloud data, depth camera image data and IMU fusion positioning data to generate a multi-source environment data set; Transmitting and synchronizing the multi-source environment data set between the robot, the server and the client through a communication link to generate a synchronized environment data set; Rendering the synchronized environment data set on the client, combining the DS-NeRF model to perform multi-source depth supervision modeling on the multi-source environment data set, performing stream-based incremental training and updating, and combining the SLAM algorithm to generate a three-dimensional environment rendering model; Receiving a target waypoint set by user interaction in the three-dimensional environment rendering model, generating an initial navigation path from the current position of the robot to the target waypoint, and dynamically displaying the initial navigation path in the three-dimensional environment rendering model; Detecting real-time obstacles and deviations of the initial navigation path based on the synchronized environment data set during navigation to generate navigation path data; Extracting obstacle distribution information, free area information and passable area information based on the three-dimensional environment rendering model and the navigation path data to generate an environment semantic representation; Building a task arrangement graph on the client and binding it with events in the environment semantic representation to generate a task execution state; Generating an environment perception result based on the three-dimensional environment rendering model, the navigation path data, the environment semantic representation and the task execution state.

2. The robot environment perception method based on three-dimensional point cloud rendering according to claim 1, characterized in that, The generation of the synchronized environment data set comprises: The robot end slices and timestamps the multi-source environment data set according to a preset data format, and the sliced multi-source environment data set is transmitted to the server through a WebSocket communication link; The server receives the sliced multi-source environment data set, performs integrity checking and sequence reorganization on the multi-source environment data set, maintains the real-time connection state of the WebSocket communication link based on a heartbeat detection mechanism, and forwards the reorganized multi-source environment data set to the client; The client receives the multi-source environment data set forwarded by the server, performs packet loss detection and timing calibration on the multi-source environment data set, completes real-time synchronization of the multi-source environment data set based on a low-latency transmission mechanism, and generates a synchronized environment data set.

3. The robot environment perception method based on three-dimensional point cloud rendering according to claim 1, characterized in that, The generation of the three-dimensional environment rendering model comprises: The client receives the synchronized environment data set and converts the laser radar point cloud data and depth camera image data in the synchronized environment data set into an input data set; The client calls the DS-NeRF model, takes the laser radar point cloud data and depth camera image data in the input data set as multi-source supervision signals, performs multi-source depth supervision modeling, and generates an initial three-dimensional point cloud rendering result; The specific process of the multi-source depth supervision modeling is as follows: first, the laser radar point cloud data is projected and coordinate-regulated to obtain sparse depth constraints; at the same time, the pixel parallax in the depth camera image data is converted into dense depth constraints; then, in the training process of the DS-NeRF model, the sparse depth constraints and the dense depth constraints are added to the depth supervision loss function and the volume rendering loss function to generate a joint optimization target; The DS-NeRF model is updated by performing a streaming incremental training in a continuous input process of a synchronous environment data set, the new synchronous environment data set is calculated in combination with the training parameters of the previous stage, and an updated three-dimensional point cloud rendering result is iteratively output, and a three-dimensional point cloud rendering model with depth supervision is gradually optimized; A two-dimensional grid map and a robot trajectory are generated by running a SLAM algorithm based on the synchronous environment data set; The two-dimensional grid map and the three-dimensional point cloud rendering model with depth supervision are fused to generate a three-dimensional environment rendering model, and dynamic display output of the three-dimensional environment rendering model is completed on the client side.

4. The robot environment perception method based on three-dimensional point cloud rendering according to claim 1, characterized in that, The generation of the initial navigation path includes: In the three-dimensional environment rendering model on the client side, a target waypoint set by user interaction is received, and a current position of the robot is obtained based on the robot trajectory; The passable area in the three-dimensional environment rendering model is discretized to generate a node set, and connections between the node set are established based on spatial adjacency and assigned with generation values; The current position of the robot is taken as a starting point, and the target waypoint is taken as an end point, and a search is performed on the node set and the connections based on the A* algorithm to generate an initial navigation path from the current position of the robot to the target waypoint; The initial navigation path is written into a dynamic display layer of the three-dimensional environment rendering model on the client side, and the initial navigation path is dynamically displayed in the three-dimensional environment rendering model.

5. The robot environment perception method based on three-dimensional point cloud rendering according to claim 1, characterized in that, The generation of the navigation path data includes: In the navigation process, the current position of the robot is obtained based on the synchronous environment data set, and the nearest path segment of the initial navigation path is located in the three-dimensional environment rendering model; Based on the synchronous environment data set, the positions of obstacles and the passable area in the three-dimensional environment rendering model are updated, and an occupancy check result of the initial navigation path is generated; A deviation value is calculated based on the spatial relationship between the current position of the robot and the initial navigation path in the three-dimensional environment rendering model, and the deviation value is compared with a threshold value; When the deviation value is greater than the threshold value and the occupancy check result indicates that the initial navigation path is blocked by an obstacle, a search is performed on the node set and the connections to generate a new navigation path in the three-dimensional environment rendering model; The display of the initial navigation path is replaced by the new navigation path in the three-dimensional environment rendering model, and the new navigation path is associated with the target waypoint and the current position of the robot, and the navigation path data is generated.

6. The robot environment perception method based on three-dimensional point cloud rendering according to claim 1, characterized in that, The generation of the environment semantic representation includes: The spatial point set in the three-dimensional environment rendering model is classified, and obstacle distribution information and free area information are generated according to the occupancy state of the three-dimensional environment rendering model; Based on the navigation path data, connectivity screening is performed in the free area information, a free area subset connected with the navigation path data is extracted as passable area information, and elements intersecting with the obstacle distribution information and having a distance from the obstacle boundary less than a preset safety distance are removed from the passable area information; The obstacle distribution information, the free area information and the passable area information are mapped to the navigation path data according to a unified coordinate index structure to generate the environment semantic representation.

7. The robot environment perception method based on three-dimensional point cloud rendering according to claim 1, characterized in that, The generation of the task execution state includes: A task orchestration graph is created on the client side, and a three-dimensional environment rendering model, navigation path data, and an environment semantic representation are loaded in the task orchestration graph as input references; The task orchestration graph is created on the client side by selecting trigger nodes, content nodes, action nodes, and waiting nodes and placing them in a visual canvas by dragging, and establishing an execution relationship by connecting lines. The user completes event binding, action selection, and delay setting in the attribute panel. The operation result is real-time echoed and synchronized to the three-dimensional environment rendering model; Trigger nodes, content nodes, action nodes, and waiting nodes are added in the task orchestration graph, and each node is assigned a unique identifier and associated with the three-dimensional environment rendering model, navigation path data, and environment semantic representation; The trigger node is bound to an event in the environment semantic representation, and the event in the environment semantic representation includes reaching a target waypoint, updating a navigation path, and entering a specified area; The execution relationship of the nodes is set in the task orchestration graph. After the trigger node is activated, the content node and the action node are executed in sequence. After the action node is completed, the waiting node is entered. After the waiting node is ended, the standby entrance of the task orchestration graph is returned; The task orchestration graph is run on the client side and subscribes to event updates of the environment semantic representation and changes of the navigation path data. The node execution is driven according to the event binding and the interactive display in the three-dimensional environment rendering model is updated; The subscription indicates that the client side declares to receive real-time changes of events in the environment semantic representation and updates of the navigation path data; A task execution state is generated, which includes the current trigger event and the node identifier being executed.

8. The robot environment perception method based on three-dimensional point cloud rendering according to claim 1, characterized in that, The generation of the environment perception result includes: The three-dimensional environment rendering model, navigation path data, environment semantic representation, and task execution state are loaded on the client side, and the timestamp and version number consistency check is completed; The obstacle distribution information in the environment semantic representation is aligned and associated with the three-dimensional environment rendering model and the navigation path data under a unified coordinate index structure; The three-dimensional environment rendering model is displayed and refreshed on the client side, the navigation path data is superimposed and displayed, and the area is highlighted with the obstacle distribution information; An environment perception result is generated, which includes the three-dimensional environment rendering model, obstacle distribution information, and navigation path data, and is associated with the task execution state; The environment perception result is published to the server and the robot side, and the saving and query registration of the environment perception result are completed on the client side.

Images