Autonomous navigation method based on laser and vision fusion and related equipment

Abstract

The invention provides an autonomous navigation method and related equipment based on laser and vision fusion, and the method comprises the following steps: carrying out multi-sensor space-time joint synchronization processing on laser point cloud data, binocular image data and a clock signal, and generating a space-time aligned multi-modal data stream; the multi-modal data stream is processed through cross-modal feature pyramid and attention guidance fusion, and dense environment characterization and a dynamic object space-time mask are generated; determining a multi-layer semantic map corresponding to the dense environment representation through layered factor graph optimization; fusing the multilayer semantic map with the neural radiation field to generate a mixed map of an explicit semantic map and an implicit radiation field; and determining a motion track of the target equipment through space-time joint constraint on the mixed map and the space-time mask of the dynamic object. Through the implementation of the scheme of the invention, the real-time distinguishing between the dynamic obstacle and the static structure can be realized in a dynamic complex environment, so that the obstacle avoidance robustness of the motion trail is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of data fusion technology, and in particular to an autonomous navigation method based on laser and vision fusion and related equipment. Background Art

[0002] With the rapid development of robotics and autonomous driving technology, the accuracy and stability of autonomous navigation systems have become increasingly important. Traditional autonomous navigation methods mainly rely on single sensor data, such as simple lidar or visual cameras, which often show obvious limitations in complex and changing environments.

[0003] In complex dynamic environments, existing autonomous navigation technologies face key challenges such as insufficient spatiotemporal alignment accuracy of multi-sensor data and difficulty ensuring the real-time and consistency of semantic maps under the interference of dynamic objects. Traditional methods rely on single-modality sensors (such as lidar or vision) to build environmental models, resulting in a serious split between geometric information and semantic understanding in scenarios such as sudden changes in lighting and frequent movement of dynamic obstacles, and explicit maps are difficult to represent implicit environmental properties (such as material reflection characteristics and lighting distribution). In addition, the mixed interference of dynamic objects and static structures makes it impossible for the motion planning module to effectively distinguish between traversable areas and instantaneous risk areas. Especially in high-frequency dynamic scenarios such as warehousing and logistics, human-machine collaboration, etc., existing technologies often cause trajectory jitter or even collisions due to delayed map updates or semantic ambiguity. Summary of the invention

[0004] The present application provides an autonomous navigation method and related equipment based on laser and vision fusion, which are used to solve the problem that related technologies cannot effectively perform route planning in complex dynamic environments.

[0005] The first aspect of the present application provides an autonomous navigation method based on laser and vision fusion, and the autonomous navigation method based on laser and vision fusion includes: Obtain laser point cloud data from the laser radar, binocular image data from the binocular camera, and IMU clock signal; Performing multi-sensor spatiotemporal joint synchronization processing on the laser point cloud data, the binocular image data, and the IMU clock signal to generate a spatiotemporal aligned multimodal data stream; Processing the multimodal data stream by cross-modal feature pyramid and attention-guided fusion to generate dense environment representation with semantic labels and spatiotemporal masks of dynamic objects; Determining a multi-layer semantic map corresponding to the dense environment representation through hierarchical factor graph optimization; fusing the multi-layer semantic map with the neural radiation field to generate a hybrid map of the explicit semantic map and the implicit radiation field; The motion trajectory of the target device is determined by the spatiotemporal joint constraints on the hybrid map and the spatiotemporal mask of the dynamic object.

[0006] Optionally, in a first implementation of the first aspect of the present application, the step of performing multi-sensor spatiotemporal joint synchronous processing on the laser point cloud data, the binocular image data, and the IMU clock signal to generate a spatiotemporal aligned multimodal data stream includes: Build a hardware trigger network based on the precise clock protocol; Generate a sensor data pool with timestamp alignment of the laser radar, the binocular camera and the IMU clock signal through the hardware trigger network; Performing interpolation processing on the unsynchronized laser point cloud data and the binocular image data in the sensor data pool to generate a spatiotemporally continuous laser point cloud sequence and image frame sequence; Determine an extrinsic parameter matrix corresponding to the laser point cloud sequence and the image frame sequence based on deep reinforcement learning; A multimodal data stream aligned in time and space is generated through the spatiotemporal joint synchronization of the sensor data pool and the extrinsic parameter matrix.

[0007] Optionally, in a second implementation of the first aspect of the present application, the step of processing the multimodal data stream by fusing a cross-modal feature pyramid with attention guidance to generate a dense environment representation with semantic labels and a spatiotemporal mask of a dynamic object includes: Generate edge feature map and height feature encoding map by extracting geometric features of spatiotemporally aligned laser point cloud data; Generate semantic feature maps and instance mask maps through semantic segmentation of spatiotemporally aligned binocular image data; Determining a cross-modal feature tensor corresponding to the edge feature map and the semantic feature map through cross-modal attention-guided feature fusion; Generate a dynamic object spatiotemporal mask by detecting dynamic objects in the cross-modal feature tensor; A dense environment representation with semantic labels is generated according to the multi-scale feature pyramid fusion of the highly encoded feature map and the instance mask map.

[0008] Optionally, in a third implementation of the first aspect of the present application, the step of determining a multi-layer semantic map corresponding to the dense environment representation through hierarchical factor graph optimization includes: The laser point cloud ICP matching residual, visual reprojection error and IMU pre-integration constraint represented by the dense environment are jointly modeled as factor nodes, and a corresponding local factor graph is generated; The spatiotemporal density distribution of the local factor graph divides the global map corresponding to the dense environment representation into a plurality of sub-graph units; Performing incremental semantic processing on the plurality of sub-graph units based on loop closure detection, and generating corresponding sub-graph semantic maps; The geometric and semantic information are weightedly fused according to the confidence of the overlapping areas of all the sub-graph semantic maps to generate a globally consistent multi-layer semantic map.

[0009] Optionally, in a fourth implementation of the first aspect of the present application, the step of fusing the multi-layer semantic map with the neural radiation field to generate a hybrid map of the explicit semantic map and the implicit radiation field includes: Mapping semantic labels and geometric features at different levels in the multi-layer semantic map into a sparse voxel hash table to generate an explicit semantic voxel field; Eliminating implicit voxel regions in the neural radiation field that conflict with the multi-layer semantic map according to the geometric boundary of the explicit semantic voxel field to generate a geometrically constrained radiation field; Performing residual learning on the color rendering branch of the geometrically constrained radiation field according to the semantic label of the explicit semantic voxel field to generate an implicit radiation field that integrates explicit semantics; The implicit radiation field is subjected to spatiotemporal mask filtering processing according to the spatiotemporal mask of the dynamic object to generate a hybrid map of the explicit semantic map and the implicit radiation field.

[0010] Optionally, in a fifth implementation of the first aspect of the present application, the step of determining the motion trajectory of the target device by applying a spatiotemporal joint constraint to the hybrid map and the spatiotemporal mask of the dynamic object includes: generating a spatiotemporal risk field according to the semantic label and the radiation field density distribution of the implicit radiation field; Determine the feasible area boundary of the starting point and the target point of the target device according to the spatiotemporal risk field and the spatiotemporal mask of the dynamic object; Generate an initial motion trajectory within the boundary of the feasible region through multi-objective trajectory optimization; Adjusting the timestamps and spatial coordinates of the trajectory points corresponding to the initial motion trajectory in real time based on the spatiotemporal occupancy probability distribution of the spatiotemporal mask of the dynamic object; The motion trajectory of the target device is updated in real time according to the timestamp and the spatial coordinates adjusted in real time.

[0011] Optionally, in a sixth implementation of the first aspect of the present application, the method further includes: Comparing the real-time sensor data of the hybrid map with the density distribution of the historical implicit radiation field to determine the area of ​​environmental structure change; Performing a local semantic map update on the explicit semantic map according to the environmental structure change area; Perform radiation field reprojection optimization on the updated local semantic map and implicit radiation field; Correcting the density distribution of the optimized radiation field by back-propagation of the spatiotemporal mask of the dynamic object; An environment-adaptive hybrid map is generated based on the updated explicit semantic map and implicit radiation field.

[0012] The second aspect of the present application provides an autonomous navigation device based on laser and vision fusion, and the autonomous navigation device based on laser and vision fusion includes: An acquisition module is used to acquire laser point cloud data from the laser radar, binocular image data from the binocular camera, and IMU clock signal; A first generating module is used to perform multi-sensor spatiotemporal joint synchronous processing on the laser point cloud data, the binocular image data and the IMU clock signal to generate a spatiotemporal aligned multimodal data stream; A second generation module is used to process the multimodal data stream through cross-modal feature pyramid and attention-guided fusion to generate dense environment representation with semantic labels and spatiotemporal masks of dynamic objects; A first determination module, configured to determine a multi-layer semantic map corresponding to the dense environment representation through hierarchical factor graph optimization; A third generation module is used to fuse the multi-layer semantic map with the neural radiation field to generate a hybrid map of the explicit semantic map and the implicit radiation field; The second determination module is used to determine the motion trajectory of the target device by applying spatiotemporal joint constraints to the hybrid map and the spatiotemporal mask of the dynamic object.

[0013] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the processor is used to execute a computer program stored in the memory, and when the processor executes the computer program, it implements each step of the autonomous navigation method based on laser and vision fusion provided in the first aspect of the embodiment of the present application.

[0014] The fourth aspect of the embodiments of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step of the autonomous navigation method based on laser and vision fusion provided in the first aspect of the embodiments of the present application is implemented.

[0015] In summary, according to the autonomous navigation method and related equipment based on laser and vision fusion provided by the present application scheme, the laser point cloud data of the laser radar, the binocular image data of the binocular camera and the IMU clock signal are obtained; the laser point cloud data, the binocular image data and the IMU clock signal are subjected to multi-sensor spatiotemporal joint synchronous processing to generate a spatiotemporal aligned multimodal data stream; the multimodal data stream is processed by cross-modal feature pyramid and attention-guided fusion to generate a dense environment representation with semantic labels and a spatiotemporal mask of dynamic objects; a multi-layer semantic map corresponding to the dense environment representation is determined by hierarchical factor graph optimization; the multi-layer semantic map is fused with the neural radiation field to generate a hybrid map of the explicit semantic map and the implicit radiation field; the motion trajectory of the target device is determined by the spatiotemporal joint constraints of the hybrid map and the spatiotemporal mask of the dynamic object. Through the implementation of the solution of this application, it is possible to achieve high-precision real-time distinction between dynamic obstacles and static structures in a dynamic and complex environment through the fusion of explicit semantic maps and implicit radiation fields and the joint constraints of spatiotemporal masks of dynamic objects, thereby significantly improving the obstacle avoidance robustness of the motion trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a process flow of an autonomous navigation method based on laser and vision fusion provided in an embodiment of the present application; Figure 2 A schematic diagram of a program module of an autonomous navigation device based on laser and vision fusion provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0018] In order to solve the problem that related technologies cannot effectively perform route planning in complex dynamic environments, the present application embodiment provides an autonomous navigation method based on laser and vision fusion, such as Figure 1 The flowchart of the autonomous navigation method based on laser and vision fusion provided in this embodiment includes the following steps: Step 110: Obtain laser point cloud data from the laser radar, binocular image data from the binocular camera, and an IMU clock signal.

[0019] Specifically, in this embodiment, the laser point cloud data of the laser radar, the binocular image data of the binocular camera, and the IMU clock signal are obtained in order to collect multimodal information of the environment and the equipment. The laser radar generates point cloud data by emitting lasers and receiving reflected light, providing high-precision depth information of the environment; the binocular camera obtains a two-dimensional image through the difference in the perspectives of the two cameras, and calculates the parallax between the images to infer the three-dimensional spatial information; the IMU clock signal provides timestamp information to help synchronize the data of different sensors and ensure the temporal and spatial consistency between multi-sensor data.

[0020] Step 120: Perform multi-sensor spatiotemporal joint synchronous processing on the laser point cloud data, binocular image data, and IMU clock signal to generate a spatiotemporal aligned multimodal data stream.

[0021] Specifically, in this embodiment, the laser point cloud data, binocular image data and IMU clock signal are processed by multi-sensor spatiotemporal joint synchronization, the purpose of which is to ensure the spatiotemporal consistency between the data by accurately matching the data timestamps of different sensors. Through this synchronization processing, the data collected by the laser, vision and inertial measurement unit sensors can be aligned in time and space, thereby realizing data fusion from different sources and ensuring the accuracy and consistency of subsequent processing.

[0022] In an optional real-time method of the present embodiment, the laser point cloud data, binocular image data and IMU clock signal are subjected to multi-sensor spatiotemporal joint synchronization processing to generate a spatiotemporal aligned multimodal data stream, including: building a hardware trigger network based on a precise clock protocol; generating a sensor data pool with timestamp alignment of laser radar, binocular camera and IMU clock signals through the hardware trigger network; interpolating the unsynchronized laser point cloud data and binocular image data in the sensor data pool to generate a spatiotemporal continuous laser point cloud sequence and image frame sequence; determining the extrinsic parameter matrix corresponding to the laser point cloud sequence and the image frame sequence based on deep reinforcement learning; and generating a spatiotemporal aligned multimodal data stream through spatiotemporal joint synchronization of the sensor data pool and the extrinsic parameter matrix.

[0023] Specifically, in this embodiment, in order to realize the hardware-triggered network construction based on the precise clock protocol, it is first necessary to introduce a precise clock synchronization protocol, using the IEEE 1588 Precision Time Protocol (PTP). The PTP protocol transmits time information between different devices through a master-slave clock mechanism to ensure that the sensor clocks can remain synchronized and achieve microsecond or even sub-microsecond synchronization accuracy. In specific implementation, a PTP clock source is installed between sensors, and the sensor devices are connected through a network to synchronize their clocks regularly according to the PTP protocol. Whenever a sensor collects data, a timestamp is attached to record the current system time. Through this precise synchronization, the problem of clock drift between different sensors can be effectively solved, so that the data of sensors such as lidar, binocular cameras, and IMU can be processed under the same time reference.

[0024] Based on the above hardware trigger network, a temporally and spatially aligned sensor data pool can be generated through the synchronization of sensor data acquisition and precise clock protocols. The data of each sensor will be timestamped when it is collected, indicating the exact time of data collection. Due to the different sampling frequencies of LiDAR, binocular camera, and IMU, data may be collected from different sensors in the same time window, resulting in data misalignment. In this case, the generated sensor data pool can ensure that the data of each sensor has a timestamp for subsequent alignment and fusion.

[0025] However, despite the foundation of spatiotemporal alignment, some sensor data are still out of sync due to the sampling frequencies and data transmission delays of different sensors, especially laser point cloud data and image frame data. At this time, it is necessary to interpolate the unsynchronized laser point cloud data and image data. For each frame of unsynchronized laser point cloud data, the laser data of the adjacent frames can be used for linear interpolation to infer the point cloud data of the middle frame. Similarly, for binocular image data, through the matching features between images, the image data of the previous and next frames are interpolated to generate a spatiotemporally continuous image sequence and laser point cloud sequence.

[0026] The extrinsic parameter matrix corresponding to the laser point cloud sequence and the image frame sequence is determined by deep reinforcement learning. The extrinsic parameter matrix includes the rotation matrix and the translation vector, which describe the relative position and posture between the sensors. In order to solve the problem of dynamic adjustment of the extrinsic parameters, a DRL-based agent is designed. In this agent, the action space is the 6 degrees of freedom (6-DoF) of the extrinsic parameters, including the adjustment of rotation and translation. By defining the reward function as the laser-visual feature matching residual, the agent can continuously optimize the extrinsic parameter matrix through interaction with the environment to maximize the feature matching between the laser point cloud and the image. For example, assuming there is a laser point cloud P and an image frame I, the matching error between them can be expressed as: , in, and Represent the corresponding feature points on the laser point cloud and the image, respectively. Represents the 3D points of the corresponding positions extracted from the image frame. By optimizing this matching error, the agent gradually learns the optimal extrinsic parameter matrix. The optimization algorithm in reinforcement learning is used to continuously adjust the extrinsic parameter matrix in this process to improve the calibration accuracy.

[0027] After the optimization of the external parameter matrix, these external parameter matrices are applied to the sensor data pool to achieve spatiotemporal joint synchronization. Specifically, the laser point cloud data and image data will be transformed according to the external parameter matrix and converted to the same coordinate system to ensure the spatial alignment of the data. At the same time, combined with the timestamp information in the sensor data pool, the temporal consistency of the data from different sensors is ensured. Through this spatiotemporal joint synchronization, a multimodal data stream is generated, which includes precisely aligned laser point clouds, image frames, and IMU data.

[0028] Step 130: Process the multimodal data stream by cross-modal feature pyramid and attention-guided fusion to generate dense environment representation with semantic labels and spatiotemporal masks of dynamic objects.

[0029] Specifically, in this embodiment, the multimodal data stream is processed by cross-modal feature pyramid and attention-guided fusion, with the aim of extracting and integrating key information from each sensor data. The feature pyramid structure can extract deep features at different scales, while the attention mechanism dynamically adjusts the fusion process according to the importance of the data to highlight the information in key areas. The dense environment representation and dynamic object spatiotemporal mask generated by this process can effectively represent the environment structure and the spatiotemporal position of dynamic objects, providing an accurate basis for subsequent spatial modeling and path planning.

[0030] In an optional real-time method of the present embodiment, a multimodal data stream is processed by a cross-modal feature pyramid and attention-guided fusion to generate a dense environment representation with semantic labels and a spatiotemporal mask of a dynamic object, including: generating an edge feature map and a height feature encoding map by extracting geometric features of spatiotemporally aligned laser point cloud data; generating a semantic feature map and an instance mask map by semantic segmentation of spatiotemporally aligned binocular image data; determining a cross-modal feature tensor corresponding to the edge feature map and the semantic feature map by feature fusion guided by cross-modal attention; generating a spatiotemporal mask of a dynamic object by dynamic object detection of the cross-modal feature tensor; and generating a dense environment representation with semantic labels based on multi-scale feature pyramid fusion of the height encoding feature map and the instance mask map.

[0031] Specifically, in this embodiment, to achieve the geometric feature extraction of spatiotemporal aligned laser point cloud data, it is first necessary to extract important geometric features from the point cloud data, mainly including edge and height change information. The point cloud data generated by the laser radar can reveal the three-dimensional geometric structure of the scene. The edge feature represents the outline of the object or the place where it changes sharply, which can help the system identify the significant structure or boundary in the scene. The height feature encoding map captures the changes in the terrain in the scene by analyzing the height information of the point cloud, such as identifying the relative height of buildings, roads or obstacles. In the processing of binocular image data, the semantic segmentation technology is first used to decompose the image into different semantic areas, such as roads, buildings or pedestrians. Semantic segmentation classifies each pixel in the image through a convolutional neural network (CNN), so that each pixel is assigned to a specific category. For example, the value of a pixel in the image is I(x,y), and the pixel belongs to category c. The goal of semantic segmentation is to output a label map L(x,y)=c through the network, where c represents the category of the pixel. In this way, a semantic feature map can be generated, which maps each pixel to a specific semantic label. In addition, the instance mask map further generates a mask for each object in the image based on semantic segmentation. For example, for multiple objects in the same category, the instance mask map can distinguish the area of ​​each object and generate a specific object mask by calculating the overlap or separation between different objects.

[0032] Afterwards, the feature fusion method guided by cross-modal attention is used to fuse the edge feature map extracted from the laser point cloud with the semantic feature map extracted from the image. The cross-modal attention mechanism uses the self-attention mechanism to align the feature map of the laser point cloud with the feature map of the image at each scale, and adjusts the weight of the feature by calculating the correlation between different modalities, so that the information of important areas is enhanced. For example, assuming that the edge feature map and the semantic feature map are E and S respectively, a weighted fusion feature tensor F can be calculated through the attention mechanism, and the formula is: , in, and is the learned weight matrix, is an activation function (such as ReLU). Through this fusion, edge features and semantic features can complement each other, thereby improving the expressive power of fused features. By performing dynamic object detection on the cross-modal feature tensor, dynamic objects in the scene can be identified. Dynamic objects are objects that change position between multiple time frames. By analyzing the spatiotemporal information in the cross-modal feature tensor, dynamic objects can be effectively detected and distinguished from static backgrounds. Dynamic object detection can use a temporal convolutional neural network (TCN) to capture temporal information. By processing the features of each time step, a spatiotemporal mask of a dynamic object is generated. For example, the mask M(t) generated by the TCN model represents the dynamic state of an object at a certain moment, where M(t)=1 means that the object is dynamic and M(t)=0 means that the object is static.

[0033] Finally, the multi-scale feature pyramid fusion combining the height feature encoding map and the instance mask map can generate a dense environment representation with semantic labels. Multi-scale feature pyramid networks (FPN) help the system capture detailed information of the environment at different resolutions by extracting and fusing feature maps at different scales. Assuming that the feature maps extracted from multiple scales s1, s2, ..., sn are Fs1, Fs2, ..., Fsn respectively, these features are fused through the pyramid structure to generate the final environment representation. : , in, is the weight of each scale feature, and the final generated It contains global information and local details of the environment, and can provide accurate environmental representation for subsequent path planning, navigation, and other perception tasks. This representation not only includes semantic information in the image (such as roads, buildings, etc.), but also combines the geometric information of the laser point cloud (such as the edges and heights of objects), so that the system can accurately understand and reconstruct the structure of the environment. Semantic labeling refers to assigning a specific meaning or category to each element or area in the environment, which is achieved through technologies such as semantic segmentation, object detection, and classification.

[0034] Step 140: Determine a multi-layer semantic map corresponding to the dense environment representation through hierarchical factor graph optimization.

[0035] Specifically, in this embodiment, a multi-layer semantic map corresponding to the dense environment representation is determined by hierarchical factor graph optimization, and the semantic information of different levels can be integrated and optimized by using the factor graph model to generate a semantic map with a hierarchical structure. This process combines the constraints of different perceptual information, improves the accuracy and reliability of the environment representation, and enables the multi-layer semantic map to better reflect the dynamic changes and complex structure of the environment.

[0036] In an optional real-time method of the present embodiment, the step of determining a multi-layer semantic map corresponding to a dense environment representation through hierarchical factor graph optimization includes: jointly modeling the laser point cloud ICP matching residual, visual reprojection error and IMU pre-integration constraint of the dense environment representation as factor nodes, and generating a corresponding local factor graph; the spatiotemporal density distribution of the local factor graph divides the global map corresponding to the dense environment representation into a number of sub-graph units; incrementally semantically processing the sub-graph units based on loop closure detection, and generating corresponding sub-graph semantic maps; weightedly fusing geometric and semantic information according to the confidence of the overlapping areas of all sub-graph semantic maps to generate a globally consistent multi-layer semantic map.

[0037] Specifically, in this embodiment, the ICP (Iterative Closest Point) matching residual of the laser point cloud estimates the relative position of adjacent laser scans by minimizing the spatial distance between the point clouds; the visual reprojection error constrains the camera pose in the SLAM (Simultaneous Localization and Mapping) system by the difference between the projection of the feature points of the image on the image plane and the actual observation; the IMU pre-integration constraint is based on the acceleration and angular velocity data provided by the inertial measurement unit, and is integrated in time series to estimate the pose change during the motion process. In order to achieve the effective fusion of these constraints, they need to be modeled as factor nodes in the factor graph to form a unified optimization framework. The factor graph is a graph structure used to represent various constraints, in which nodes represent variables and edges represent constraints. In this factor graph, each factor corresponds to a constraint, and the edges connect the variables to be optimized. By minimizing the cost function of all factors in the factor graph, the optimal variable value can be obtained. In this scheme, the ICP matching residual of the laser point cloud, the visual reprojection error, and the IMU pre-integration constraint will be added as factors to the factor graph. Each factor will have a corresponding cost function, which can be optimized by the least squares method. After constructing the factor graph, the next step is to divide the global map according to the spatiotemporal density distribution. The spatiotemporal density distribution reflects the change law of sensor data (including laser point cloud, image and IMU data) in time and space. By analyzing the distribution of these data, the entire environment can be divided into several sub-graph units. Each sub-graph unit represents the environmental characteristics of a local area. By analyzing the density of different sub-graphs, the appropriate sub-graph size can be selected according to the needs to ensure that each sub-graph can contain sufficient semantic information and geometric structure, while avoiding computational redundancy caused by over-subdivision. The sub-graph division process is based on loop detection, which is a technology used to determine whether to return to a previously visited place. It matches the current sensor observation with the historical observation. In the process of constructing subgraphs, loop detection can determine whether two subgraphs belong to the same environment area by calculating the overlapping area between them. If a loop is detected, incremental semantic processing can be triggered to update the semantic information in the subgraph. The incremental semantic processing method integrates new observations with historical observations, and gradually optimizes the semantic expression of each subgraph in the process of continuously accumulating data. Each subgraph is optimized through a local factor graph, and finally generates a semantic map related to its location and time.

[0038] In the process of generating the sub-image semantic map, the fusion of geometric information and semantic information is taken into consideration. The geometric information comes from the structural data of the laser point cloud, while the semantic information comes from the semantic segmentation result of the image. To this end, a weighted fusion mechanism needs to be designed to combine the geometric information and semantic information in each sub-image. When fusion is performed, weighting can be performed according to the confidence of the overlapping area between sub-images to ensure that the semantic information and geometric information between multiple sub-images can be accurately aligned and fused in the overlapping area. Assume that the overlapping area of ​​two sub-images G1 and G2 is ， is the confidence of the overlapping area, then their fusion cost function can be expressed as: , in, is the cost of geometric information, is the cost of semantic information, the cost of geometric information mainly reflects the matching error of spatial structure or shape, and the cost of semantic information mainly reflects the matching error of object category, label or semantic identification in image or other sensor data. Through weighted fusion, it can ensure that the geometric information and semantic information of each sub-image in the overlapping area are consistent, and finally generate a globally consistent multi-layer semantic map.

[0039] Step 150: Fusing the multi-layer semantic map with the neural radiation field to generate a hybrid map of the explicit semantic map and the implicit radiation field.

[0040] Specifically, in this embodiment, in order to break through the limitations of traditional explicit maps, the system introduces neural radiation fields (an implicit rendering model) for hybrid modeling. The voxel structure (three-dimensional grid unit) of the explicit semantic map serves as a spatial constraint of the neural radiation field, limiting its reconstruction range to improve computational efficiency; the neural radiation field models implicit properties (such as material reflectivity, lighting and shadows) through ray tracing to fill in the missing details of the explicit map. The dynamic object spatiotemporal mask filters the radiation field data of the moving area at this stage to avoid dynamic interference affecting the static environment modeling, and finally forms an explicit-implicit complementary hybrid map: the explicit part supports real-time path planning, and the implicit part enhances environmental understanding.

[0041] In an optional implementation of the present embodiment, the step of fusing the multi-layer semantic map with the neural radiation field to generate a hybrid map of the explicit semantic map and the implicit radiation field includes: mapping the semantic labels and geometric features of different levels in the multi-layer semantic map into a sparse voxel hash table to generate an explicit semantic voxel field; eliminating the implicit voxel areas in the neural radiation field that conflict with the multi-layer semantic map according to the geometric boundaries of the explicit semantic voxel field to generate a geometrically constrained radiation field; performing residual learning on the color rendering branch of the geometrically constrained radiation field according to the semantic label of the explicit semantic voxel field to generate an implicit radiation field that fuses explicit semantics; performing spatiotemporal mask filtering on the implicit radiation field according to the spatiotemporal mask of the dynamic object to generate a hybrid map of the explicit semantic map and the implicit radiation field.

[0042] Specifically, in this embodiment, the semantic labels and geometric features of different levels in the multi-layer semantic map are mapped to a sparse voxel hash table to generate an explicit semantic voxel field. A voxel represents a small volume unit in space and is used to build a 3D map. In a sparse voxel hash table, only areas where objects exist are explicitly stored, while blank areas do not occupy storage space, which can effectively reduce computational and storage overhead. The explicit semantic voxel field can assign a unique semantic label and geometric attribute to each voxel in space by combining geometric features and semantic labels from laser point clouds and images. This mapping process is implemented through a voxelization algorithm, in which the features of each voxel reflect not only its geometric information (such as depth, boundaries) but also semantic information (such as roads, buildings, obstacles, etc.).

[0043] Neural radiance fields use an implicit representation method to model lighting, texture, and structural features in 3D space. It estimates color and density from different perspectives by training neural networks. In this process, implicit voxels refer to the spatial regions learned by the neural network. The physical information they contain (such as color and density) is not stored explicitly, but is inferred through the radiation field. Since the neural radiance field is not directly subject to geometric constraints, the generated implicit voxels may be inconsistent with the semantic labels and geometric features in the multi-layer semantic map. For example, in an urban scene, the neural radiance field may generate some areas that cannot be matched with roads or buildings. In order to ensure that the generated map Figure 1The geometric boundaries of the explicit semantic voxel field will be used to compare and eliminate these conflicting areas, so that the physical information in the radiation field is consistent with the geometric features of the actual environment. On this basis, the color rendering branch of the geometrically constrained radiation field is further optimized by learning the semantic labels of the explicit semantic voxel field. Residual learning can reduce rendering errors and generate implicit radiation fields that incorporate explicit semantics. By training the neural network, the output of color rendering is more consistent with the label information in the explicit semantic voxel field, which can improve the accuracy of the final rendering results, especially in object recognition and texture reconstruction in complex scenes.

[0044] Finally, spatiotemporal filtering is performed based on the spatiotemporal mask of dynamic objects to further optimize the representation of the implicit radiation field. This process can help remove the interference of dynamic objects on the modeling of the static environment and ensure that the final hybrid map contains only static geometric structures. The spatiotemporal mask of dynamic objects is to detect moving objects such as pedestrians and vehicles by analyzing the laser point cloud, image frames and IMU (inertial measurement unit) data during the multimodal data fusion process. In this step, the spatiotemporal mask is used to filter out the influence of these dynamic objects to obtain a more stable and accurate representation of the static environment. The generated hybrid map is a combination of explicit semantic maps and implicit radiation fields. It not only retains the geometric structure and object classification information of the environment, but also achieves high-quality three-dimensional rendering through the implicit radiation field.

[0045] Step 160: Determine the motion trajectory of the target device by applying spatiotemporal joint constraints to the hybrid map and the spatiotemporal mask of the dynamic object.

[0046] Specifically, in this embodiment, the motion trajectory of the target device is determined by the spatiotemporal joint constraints of the hybrid map and the spatiotemporal mask of the dynamic object, in order to ensure more accurate positioning and path planning of the device. By constraining the spatiotemporal mask of the dynamic object with the environmental map, the motion trajectory of the target device can be tracked in real time, avoiding the interference of dynamic objects while maintaining high-precision cognition of the environment. This constraint helps to improve the stability and reliability of the autonomous navigation system and ensure that the device can achieve accurate positioning and navigation in complex environments.

[0047] In an optional implementation of the present embodiment, the step of determining the motion trajectory of the target device by means of the spatiotemporal joint constraints on the hybrid map and the spatiotemporal mask of the dynamic object includes: generating a spatiotemporal risk field according to the semantic label and the radiation field density distribution of the implicit radiation field; determining the feasible area boundary of the starting point and the target point of the target device according to the spatiotemporal risk field and the spatiotemporal mask of the dynamic object; generating an initial motion trajectory within the feasible area boundary through multi-target trajectory optimization; adjusting the timestamps and spatial coordinates of the trajectory points corresponding to the initial motion trajectory in real time based on the spatiotemporal occupancy probability distribution of the spatiotemporal mask of the dynamic object; and updating the motion trajectory of the target device in real time according to the timestamps and spatial coordinates adjusted in real time.

[0048] Specifically, in this embodiment, when constructing the spatiotemporal risk field, the radiation field density distribution of the semantic label and the implicit radiation field must first be combined. This process mainly estimates the risk level of each area in the environment by analyzing the geometric features and semantic information of the objects in the environment. By combining the semantic label with the radiation field density distribution of the implicit radiation field, the spatiotemporal risk distribution of each voxel or spatial area can be obtained. This spatiotemporal risk field reflects the potential risk of different areas in the environment to the target device at a certain moment or over a period of time. For example, dynamic objects may make some areas more dangerous, while static obstacles will cause the area to have a high density of collision risks. According to the generated spatiotemporal risk field and the spatiotemporal mask of the dynamic object, the boundary of the feasible area of ​​the target device can be further determined. The spatiotemporal mask of the dynamic object is obtained by detecting and calibrating the dynamic objects in the environment. These dynamic objects may have an impact on the path planning of the target device. By combining with the spatiotemporal risk field, it can be determined which areas are feasible at a specific moment, that is, areas with no risk or low risk. It is understandable that the starting point and target point of the target device (including but not limited to self-driving cars, drones, mobile robots, AGVs, etc.) also need to be mapped to the spatiotemporal risk field to determine a suitable feasible area boundary, which is an area composed of multiple sub-areas that contains all spaces that can be safely passed. After determining the boundary of the feasible area, the initial motion trajectory is generated through multi-objective trajectory optimization, and the trajectory planning of the target device is optimized using the spatial information within the boundary of the feasible area and the spatiotemporal mask of the dynamic object. Multi-objective trajectory optimization considers the balance of multiple objectives, such as the shortest path, the minimum risk path, the obstacle avoidance path, etc., and ensures that the target device can avoid collisions and complete the task within a given time window. Optimization algorithms such as the A* algorithm, the Dijkstra algorithm, or optimization methods based on deep reinforcement learning can be used to generate preliminary motion trajectories.

[0049] Based on the spatiotemporal occupancy probability distribution of the spatiotemporal mask of the dynamic object, the initial motion trajectory will be adjusted in real time. In this process, the spatiotemporal occupancy probability distribution represents the probability that a dynamic object occupies a given time point and spatial position. This probability distribution is generated by continuously monitoring the motion trajectory of the dynamic object and can be dynamically updated. By analyzing the spatiotemporal occupancy probability distribution, the target device can adjust its trajectory in time to avoid collision with the dynamic object. When adjusting the trajectory, the timestamp and spatial coordinates adjusted in real time will be adjusted according to the position of the dynamic object to ensure that the target device can respond to environmental changes in real time. The real-time updated motion trajectory is based on the continuous adjustment of timestamps and spatial coordinates. The timestamp represents the state of the target device at different points in time, while the spatial coordinates represent the position of the device in three-dimensional space. By continuously optimizing the timestamps and spatial coordinates in the trajectory, the target device can ensure that it can move in the best way throughout the path. At each trajectory update, the device recalculates the possible paths based on the current spatiotemporal occupancy probability distribution to ensure that the path is always the safest. This motion trajectory update method based on real-time adjustment not only ensures that the device can avoid collisions with dynamic objects, but also adapts to complex environmental changes and achieves efficient and safe path planning.

[0050] It should be noted that in order to ensure that the target device can dynamically respond to environmental changes, trajectory planning not only considers the current spatial position, but also takes time factors into consideration. Therefore, a space-time graph can be used to represent each time step and spatial position in the path. The trajectory planning of the target device can evaluate each space-time coordinate point, calculate the risk of collision with dynamic objects, and dynamically adjust the path. For example, trajectory optimization can be performed by combining the following objective function: , in, Expressed as the objective function of trajectory optimization, It's in time t The spatial position of the target device at the moment, is the spatiotemporal occupancy probability of the corresponding position, is an indicator function indicating whether the position is safe (i.e., there is no dynamic object occupying the position). is the travel distance of the target device between adjacent time steps, The safety weight determines the importance of safety on the path. At the moment t If it is occupied by dynamic objects, the safety index The trajectory will be penalized, and the higher the safety, the better the target device's ability to avoid obstacles. is the path length weight, indicating that the target device hopes to complete the target in the shortest possible time. The larger the weight, the more likely the device is to choose a shorter path. To weight the collision risk, the path selection is controlled to prevent the target device from approaching the location of dynamic objects.

[0051] When the target device encounters a dynamic object during real-time operation, the trajectory will be adjusted in real time according to the new environmental state. In this process, the real-time update of timestamps and spatial coordinates is very critical. To this end, a spatiotemporal dynamic optimization algorithm, such as a Kalman filter or a particle filter, is used to update the trajectory of the target device in real time. Through these methods, the state of the target device at the next time step can be estimated, and the optimal position of the target device can be recalculated based on the latest position and occupancy probability of the dynamic object. The position and velocity estimates of the target device can be updated through the following Kalman filter update formula: , in, is the state of the device at time k (such as position, speed), is the state transition matrix, is the control input (such as acceleration), is the process noise, is the control input matrix, which is used to convert the control input Transformed into an impact on the system state. By estimating and updating the motion state of the device in real time, accurate position prediction of the target device in a dynamic environment can be achieved.

[0052] In an optional implementation of the present embodiment, the real-time sensor data of the hybrid map is compared with the density distribution of the historical implicit radiation field to determine the area of ​​environmental structure change; the explicit semantic map is locally updated according to the area of ​​environmental structure change; the updated local semantic map and implicit radiation field are optimized by radiation field reprojection; the density distribution of the optimized radiation field is corrected by back propagation of the spatiotemporal mask of dynamic objects; and an environment-adaptive hybrid map is generated based on the updated explicit semantic map and implicit radiation field.

[0053] Specifically, in this embodiment, in order to determine the area where the environmental structure has changed, it is necessary to compare the density distribution of the real-time sensor data with the historical implicit radiation field. The implicit radiation field (such as the neural radiation field NeRF) models the radiation field of the scene through a multi-layer MLP (multi-layer perceptron) network. These models provide information about illumination, color and transparency for each spatial point. By comparing the real-time laser point cloud or image data (provided by sensors such as lidar and cameras) with the density data in the historical implicit radiation field, the area that has changed compared to the original environment can be identified. After determining the area where the environmental structure has changed, the explicit semantic map needs to be locally updated. The explicit semantic map is constructed by extracting semantic information from sensor data (such as images or laser point clouds), including information such as the category and boundary of the object. Local semantic map update means that only the area affected by the environmental change is modified, rather than the entire map, which can greatly improve the computational efficiency. In the local update process, by comparing the real-time data with the historical data, the objects that have changed or disappeared can be corrected, such as adding new obstacles or deleting objects that no longer exist. The relationship between the updated local semantic map and the implicit radiation field also needs to be further optimized, especially the reprojection optimization of the radiation field. Since the explicit semantic map and the implicit radiation field are generated by different modeling methods, they may have geometric or semantic inconsistencies. Through reprojection optimization, the spatial information of the explicit semantic map can be more accurately aligned with the voxel position in the implicit radiation field, and the parameters of the implicit radiation field can be adjusted to better match the updated structure and object position in the environment. After the radiation field reprojection optimization, the density distribution of the optimized radiation field needs to be further corrected through the back propagation of the dynamic object spatiotemporal mask. The spatiotemporal mask is generated based on the detection results of dynamic objects and is used to mark objects in the scene that do not belong to the static environment, such as moving vehicles or walking people. Through back propagation, the influence of these dynamic objects will be eliminated, thereby avoiding misleading the static environment model (such as the implicit radiation field). The core idea of ​​back propagation is to adjust the parameters in the model by comparing the error between the predicted radiation field and the real data, so that the radiation field better matches the result of the spatiotemporal mask of the dynamic object, thereby reducing the influence of the dynamic object on the environment model. Finally, an adaptive hybrid map can be generated based on the updated explicit semantic map and implicit radiation field. This hybrid map not only retains the clear structural information provided by the explicit semantic labels, but also integrates the high-precision details of the implicit radiation field. The adaptive hybrid map can automatically update as the environment changes and make adjustments based on dynamic changes in the environment. By combining the advantages of explicit semantic maps and implicit radiation fields, the hybrid map can provide a more comprehensive and accurate representation of the environment in complex scenes.

[0054] It should be noted that when optimizing through reprojection, the calculation formula can be expressed as: , Where E is the total optimization error, represents a point in space, and Respectively represent the projection positions in the map and data, and Represent the density value, is a weight that balances the geometric error and the density error. With this formula, the alignment between the implicit radiation field and the explicit semantic map can be optimized by adjusting the model parameters to minimize the error.

[0055] According to the autonomous navigation method based on laser and vision fusion provided by the present application, the laser point cloud data of the laser radar, the binocular image data of the binocular camera and the IMU clock signal are obtained; the laser point cloud data, binocular image data and IMU clock signal are processed by multi-sensor spatiotemporal joint synchronization to generate a multimodal data stream aligned in time and space; the multimodal data stream is processed by cross-modal feature pyramid and attention-guided fusion to generate a dense environment representation with semantic labels and a dynamic object spatiotemporal mask; the multi-layer semantic map corresponding to the dense environment representation is determined by hierarchical factor graph optimization; the multi-layer semantic map is fused with the neural radiation field to generate a hybrid map of the explicit semantic map and the implicit radiation field; the motion trajectory of the target device is determined by the spatiotemporal joint constraints of the hybrid map and the spatiotemporal mask of the dynamic object. Through the implementation of the present application, in a dynamic and complex environment, through the fusion of the explicit semantic map and the implicit radiation field and the joint constraints of the spatiotemporal mask of the dynamic object, the high-precision real-time distinction between dynamic obstacles and static structures can be achieved, thereby significantly improving the obstacle avoidance robustness of the motion trajectory.

[0056] Figure 2 An autonomous navigation device based on laser and vision fusion is provided in an embodiment of the present application. The autonomous navigation device based on laser and vision fusion can be used to implement the autonomous navigation method based on laser and vision fusion in the above-mentioned embodiment. Figure 2 As shown, the autonomous navigation device based on laser and vision fusion mainly includes: An acquisition module 10 is used to acquire laser point cloud data of a laser radar, binocular image data of a binocular camera, and an IMU clock signal; The first generation module 20 is used to perform multi-sensor spatiotemporal joint synchronization processing on the laser point cloud data, the binocular image data and the IMU clock signal to generate a spatiotemporal aligned multi-modal data stream; A second generation module 30 is used to process the multimodal data stream by cross-modal feature pyramid and attention-guided fusion to generate dense environment representation with semantic labels and spatiotemporal masks of dynamic objects; A first determination module 40, for determining a multi-layer semantic map corresponding to the dense environment representation through hierarchical factor graph optimization; A third generation module 50 is used to fuse the multi-layer semantic map with the neural radiation field to generate a hybrid map of the explicit semantic map and the implicit radiation field; The second determination module 60 is used to determine the motion trajectory of the target device by applying spatiotemporal joint constraints to the hybrid map and the spatiotemporal mask of the dynamic object.

[0057] In an optional real-time mode of the present embodiment, the first generation module is specifically used to: build a hardware trigger network based on a precise clock protocol; generate a sensor data pool with timestamp-aligned laser radar, binocular camera and IMU clock signals through the hardware trigger network; interpolate the unsynchronized laser point cloud data and binocular image data in the sensor data pool to generate a spatiotemporally continuous laser point cloud sequence and image frame sequence; determine the extrinsic parameter matrix corresponding to the laser point cloud sequence and the image frame sequence based on deep reinforcement learning; and generate a spatiotemporally aligned multimodal data stream through the spatiotemporal joint synchronization of the sensor data pool and the extrinsic parameter matrix.

[0058] In an optional real-time method of the present embodiment, the second generation module is specifically used to: generate an edge feature map and a height feature encoding map by extracting geometric features of spatiotemporally aligned laser point cloud data; generate a semantic feature map and an instance mask map by semantic segmentation of spatiotemporally aligned binocular image data; determine a cross-modal feature tensor corresponding to the edge feature map and the semantic feature map by feature fusion guided by cross-modal attention; generate a spatiotemporal mask of a dynamic object by dynamic object detection of the cross-modal feature tensor; generate a dense environment representation with semantic labels based on the multi-scale feature pyramid fusion of the height encoding feature map and the instance mask map.

[0059] In an optional real-time method of the present embodiment, the first determination module is specifically used to: jointly model the ICP matching residual of the laser point cloud, the visual reprojection error and the IMU pre-integration constraint of the dense environment representation as factor nodes, and generate a corresponding local factor graph; the spatiotemporal density distribution of the local factor graph divides the global map corresponding to the dense environment representation into a number of sub-graph units; based on loop closure detection, incremental semantic processing is performed on the sub-graph units, and a corresponding sub-graph semantic map is generated; geometric and semantic information are weightedly fused according to the confidence of the overlapping areas of all sub-graph semantic maps to generate a globally consistent multi-layer semantic map.

[0060] In an optional real-time method of the present embodiment, the third generation module is specifically used to: map the semantic labels and geometric features of different levels in the multi-layer semantic map into a sparse voxel hash table to generate an explicit semantic voxel field; eliminate the implicit voxel areas in the neural radiation field that conflict with the multi-layer semantic map according to the geometric boundaries of the explicit semantic voxel field to generate a geometrically constrained radiation field; perform residual learning on the color rendering branch of the geometrically constrained radiation field according to the semantic labels of the explicit semantic voxel field to generate an implicit radiation field that integrates explicit semantics; perform spatiotemporal mask filtering on the implicit radiation field according to the spatiotemporal mask of the dynamic object to generate a hybrid map of the explicit semantic map and the implicit radiation field.

[0061] In an optional real-time method of the present embodiment, the second determination module is specifically used to: generate a spatiotemporal risk field based on semantic tags and the radiation field density distribution of the implicit radiation field; determine the feasible area boundary of the starting point and target point of the target device based on the spatiotemporal risk field and the spatiotemporal mask of the dynamic object; generate an initial motion trajectory within the feasible area boundary through multi-target trajectory optimization; adjust the timestamps and spatial coordinates of the trajectory points corresponding to the initial motion trajectory in real time based on the spatiotemporal occupancy probability distribution of the spatiotemporal mask of the dynamic object; and update the motion trajectory of the target device in real time based on the timestamps and spatial coordinates adjusted in real time.

[0062] In an optional real-time method of the present embodiment, the third generation module is also used to: compare the real-time sensor data of the hybrid map with the density distribution of the historical implicit radiation field to determine the area of ​​environmental structure change; update the local semantic map of the explicit semantic map according to the area of ​​environmental structure change; perform radiation field reprojection optimization on the updated local semantic map and implicit radiation field; correct the density distribution of the optimized radiation field through back propagation of the dynamic object spatiotemporal mask; and generate an environment-adaptive hybrid map based on the updated explicit semantic map and implicit radiation field.

[0063] According to the autonomous navigation device based on laser and vision fusion provided by the present application, the laser point cloud data of the laser radar, the binocular image data of the binocular camera and the IMU clock signal are obtained; the laser point cloud data, the binocular image data and the IMU clock signal are processed by multi-sensor spatiotemporal joint synchronization to generate a multimodal data stream aligned in time and space; the multimodal data stream is processed by cross-modal feature pyramid and attention-guided fusion to generate a dense environment representation with semantic labels and a spatiotemporal mask of dynamic objects; the multi-layer semantic map corresponding to the dense environment representation is determined by hierarchical factor graph optimization; the multi-layer semantic map is fused with the neural radiation field to generate a hybrid map of the explicit semantic map and the implicit radiation field; the motion trajectory of the target device is determined by the spatiotemporal joint constraints of the hybrid map and the spatiotemporal mask of the dynamic object. Through the implementation of the present application, in a dynamic and complex environment, through the fusion of the explicit semantic map and the implicit radiation field and the joint constraints of the spatiotemporal mask of the dynamic object, the high-precision real-time distinction between dynamic obstacles and static structures can be achieved, thereby significantly improving the obstacle avoidance robustness of the motion trajectory.

[0064] According to the application plan provided Figure 3 An electronic device provided in an embodiment of the present application. The electronic device can be used to implement the autonomous navigation method based on laser and vision fusion in the aforementioned embodiment, mainly including: The memory 301, the processor 302, and the computer program 303 stored in the memory 301 and executable on the processor 302, the memory 301 and the processor 302 are connected by communication. When the processor 302 executes the computer program 303, the autonomous navigation method based on laser and vision fusion in the above-mentioned embodiment is implemented. The number of processors can be one or more.

[0065] The memory 301 may be a high-speed random access memory (RAM) memory, or a non-volatile memory (non-volatile memory), such as a disk memory. The memory 301 is used to store executable program codes, and the processor 302 is coupled to the memory 301 .

[0066] Furthermore, the present application also provides a computer-readable storage medium, which may be provided in the electronic device in the above embodiments. Figure 3 Memory in the illustrated embodiment.

[0067] The computer readable storage medium stores a computer program, and when the program is executed by the processor, the autonomous navigation method based on laser and vision fusion in the aforementioned embodiment is implemented. Furthermore, the computer storable medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk, and other media that can store program codes.

[0068] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0069] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0070] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An autonomous navigation method based on laser and vision fusion, characterized in that: include: Obtain laser point cloud data from the laser radar, binocular image data from the binocular camera, and IMU clock signal; Performing multi-sensor spatiotemporal joint synchronization processing on the laser point cloud data, the binocular image data, and the IMU clock signal to generate a spatiotemporal aligned multimodal data stream; Processing the multimodal data stream by cross-modal feature pyramid and attention-guided fusion to generate dense environment representation with semantic labels and spatiotemporal masks of dynamic objects; determining a multi-layer semantic map corresponding to the dense environment representation through hierarchical factor graph optimization; fusing the multi-layer semantic map with the neural radiation field to generate a hybrid map of the explicit semantic map and the implicit radiation field; The motion trajectory of the target device is determined by the spatiotemporal joint constraints on the hybrid map and the spatiotemporal mask of the dynamic object.

2. The autonomous navigation method based on laser and vision fusion according to claim 1 is characterized in that: The step of performing multi-sensor spatiotemporal joint synchronous processing on the laser point cloud data, the binocular image data and the IMU clock signal to generate a spatiotemporal aligned multimodal data stream includes: Build a hardware trigger network based on the precise clock protocol; Generate a sensor data pool with timestamp alignment of the laser radar, the binocular camera and the IMU clock signal through the hardware trigger network; Performing interpolation processing on the unsynchronized laser point cloud data and the binocular image data in the sensor data pool to generate a spatiotemporally continuous laser point cloud sequence and image frame sequence; Determine an extrinsic parameter matrix corresponding to the laser point cloud sequence and the image frame sequence based on deep reinforcement learning; A multimodal data stream aligned in time and space is generated through the spatiotemporal joint synchronization of the sensor data pool and the extrinsic parameter matrix.

3. The autonomous navigation method based on laser and vision fusion according to claim 2 is characterized in that: The step of processing the multimodal data stream by cross-modal feature pyramid and attention-guided fusion to generate dense environment representation with semantic labels and spatiotemporal masks of dynamic objects includes: Generate edge feature map and height feature encoding map by extracting geometric features of spatiotemporally aligned laser point cloud data; Generate semantic feature maps and instance mask maps through semantic segmentation of spatiotemporally aligned binocular image data; Determining a cross-modal feature tensor corresponding to the edge feature map and the semantic feature map through cross-modal attention-guided feature fusion; Generate a dynamic object spatiotemporal mask by detecting dynamic objects in the cross-modal feature tensor; A dense environment representation with semantic labels is generated according to the multi-scale feature pyramid fusion of the highly encoded feature map and the instance mask map.

4. The autonomous navigation method based on laser and vision fusion according to claim 1 is characterized in that: The step of determining a multi-layer semantic map corresponding to the dense environment representation by hierarchical factor graph optimization comprises: The laser point cloud ICP matching residual, visual reprojection error and IMU pre-integration constraint represented by the dense environment are jointly modeled as factor nodes, and a corresponding local factor graph is generated; The spatiotemporal density distribution of the local factor graph divides the global map corresponding to the dense environment representation into a plurality of sub-graph units; Performing incremental semantic processing on the plurality of sub-graph units based on loop closure detection, and generating corresponding sub-graph semantic maps; The geometric and semantic information are weightedly fused according to the confidence of the overlapping areas of all the sub-graph semantic maps to generate a globally consistent multi-layer semantic map.

5. The autonomous navigation method based on laser and vision fusion according to claim 4 is characterized in that: The step of fusing the multi-layer semantic map with the neural radiation field to generate a hybrid map of the explicit semantic map and the implicit radiation field comprises: Mapping semantic labels and geometric features at different levels in the multi-layer semantic map into a sparse voxel hash table to generate an explicit semantic voxel field; Eliminating implicit voxel regions in the neural radiation field that conflict with the multi-layer semantic map according to the geometric boundary of the explicit semantic voxel field to generate a geometrically constrained radiation field; Performing residual learning on the color rendering branch of the geometrically constrained radiation field according to the semantic label of the explicit semantic voxel field to generate an implicit radiation field that integrates explicit semantics; The implicit radiation field is subjected to spatiotemporal mask filtering processing according to the spatiotemporal mask of the dynamic object to generate a hybrid map of the explicit semantic map and the implicit radiation field.

6. The autonomous navigation method based on laser and vision fusion according to claim 5 is characterized in that: The step of determining the motion trajectory of the target device by applying spatiotemporal joint constraints to the hybrid map and the spatiotemporal mask of the dynamic object comprises: generating a spatiotemporal risk field according to the semantic label and the radiation field density distribution of the implicit radiation field; Determine the feasible area boundary of the starting point and the target point of the target device according to the spatiotemporal risk field and the spatiotemporal mask of the dynamic object; Generate an initial motion trajectory within the boundary of the feasible region through multi-objective trajectory optimization; Adjusting the timestamps and spatial coordinates of the trajectory points corresponding to the initial motion trajectory in real time based on the spatiotemporal occupancy probability distribution of the spatiotemporal mask of the dynamic object; The motion trajectory of the target device is updated in real time according to the timestamp and the spatial coordinates adjusted in real time.

7. The autonomous navigation method based on laser and vision fusion according to claim 5 is characterized in that: The method further comprises: Comparing the real-time sensor data of the hybrid map with the density distribution of the historical implicit radiation field to determine the area of ​​environmental structure change; Performing a local semantic map update on the explicit semantic map according to the environmental structure change area; Perform radiation field reprojection optimization on the updated local semantic map and implicit radiation field; Correcting the density distribution of the optimized radiation field by back-propagation of the spatiotemporal mask of the dynamic object; An environment-adaptive hybrid map is generated based on the updated explicit semantic map and implicit radiation field.

8. An autonomous navigation device based on laser and vision fusion, characterized in that: The autonomous navigation device based on laser and vision fusion includes: An acquisition module is used to acquire laser point cloud data from the laser radar, binocular image data from the binocular camera, and IMU clock signal; A first generating module is used to perform multi-sensor spatiotemporal joint synchronous processing on the laser point cloud data, the binocular image data and the IMU clock signal to generate a spatiotemporal aligned multimodal data stream; A second generation module is used to process the multimodal data stream through cross-modal feature pyramid and attention-guided fusion to generate dense environment representation with semantic labels and spatiotemporal masks of dynamic objects; A first determination module, configured to determine a multi-layer semantic map corresponding to the dense environment representation through hierarchical factor graph optimization; A third generation module is used to fuse the multi-layer semantic map with the neural radiation field to generate a hybrid map of the explicit semantic map and the implicit radiation field; The second determination module is used to determine the motion trajectory of the target device by applying spatiotemporal joint constraints to the hybrid map and the spatiotemporal mask of the dynamic object.

9. An electronic device, characterized in that: The device comprises a memory and a processor, wherein: The processor is used to execute the computer program stored in the memory; When the processor executes the computer program, the steps of the autonomous navigation method based on laser and vision fusion described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the autonomous navigation method based on laser and vision fusion described in any one of claims 1 to 7 are implemented.

Images