A Laser Vision Fusion Navigation and Positioning Method, Device, Equipment and Storage Medium

Abstract

This application discloses a laser vision fusion navigation and positioning method, device, equipment, and storage medium, including: obtaining point cloud data and image data of the current area, and achieving calibration through data alignment; fusing and transforming the aligned data to generate a fusion map containing geometric structure and semantic information; based on the fusion map and a preset neural network model, identifying the passable area, obstacle category, and location information from the fusion map based on the preset neural network model; fusing the above multi-target recognition results to generate a navigation and positioning map, providing environmental model support for the navigation system. This application achieves multi-dimensional perception of the environment through the complementary advantages of laser and visual data, model training, and multi-source information integration, improving the accuracy and reliability of navigation and positioning in complex scenarios.

Description

Technical Field

[0001] The present application relates to the field of positioning and navigation technology, and in particular to a laser vision fusion navigation and positioning method, device, equipment and storage medium. Background Art

[0002] With the rapid development of autonomous driving and intelligent navigation technologies, high-precision environmental perception has become a core requirement for reliable navigation and positioning. Current autonomous driving and intelligent navigation technologies are implemented through pure laser or pure vision solutions. However, single-sensor solutions generally have defects: for example, although lidar can provide accurate three-dimensional coordinates, it lacks semantic information; while visual sensors have rich texture and semantic recognition capabilities, they are easily affected by environmental factors such as lighting and occlusion, resulting in incomplete obstacle recognition and fuzzy demarcation of traversable areas, making it difficult to meet the navigation and positioning accuracy requirements in complex scenarios. Although existing fusion solutions attempt to combine multimodal data, they suffer from problems such as low data alignment efficiency, insufficient feature fusion, and insufficient model generalization capabilities, resulting in large errors in navigation map construction and poor system robustness.

[0003] Therefore, how to improve the accuracy and reliability of navigation and positioning in complex scenarios is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide a laser vision fusion navigation and positioning method, device, equipment, and storage medium, which can improve the accuracy and reliability of navigation and positioning in complex scenarios. The embodiments of the present application provide a laser vision fusion navigation and positioning method, device, equipment, and storage medium as follows:

[0005] Acquire point cloud data and image data of the current area, and perform data alignment on the point cloud data and the image data;

[0006] Performing fusion conversion processing on the aligned point cloud data and the image data to obtain a fused map;

[0007] Based on the fused map and a preset neural network model, a plurality of target recognition results are obtained, the plurality of target recognition results including passable areas, obstacle categories, and location information, the preset neural network model being obtained by training an initial neural network model based on sample data collected in historical navigation and positioning scenarios, the sample data including a plurality of historical fused maps collected in the historical navigation and positioning scenarios, and a plurality of passable areas, a plurality of historical obstacle categories, and a plurality of location information in the historical navigation and positioning scenarios;

[0008] The multiple target recognition results are fused according to formula (1) to obtain a navigation positioning map, which is used for navigation positioning;

[0009] (1)

[0010] in, Positioning map for navigation, The number of recognition results for the target, is the weight coefficient of the i-th target recognition result, is the i-th target recognition result, and Ci is the confidence of the i-th target recognition result.

[0011] In some embodiments, the multiple target recognition results obtained based on the fusion map and the preset neural network model include:

[0012] Normalizing the fused map to obtain a normalized fused map;

[0013] Performing feature extraction on the normalized fusion map to obtain a multi-scale feature map;

[0014] Multi-task branch processing is performed on the multi-scale feature map to obtain passable areas, obstacle categories and location information.

[0015] In some embodiments, the multi-task branch includes a traversable area segmentation branch, an obstacle category identification branch, and a location information extraction branch. The multi-task branch processing of the multi-scale feature map to obtain the traversable area, obstacle category, and location information includes:

[0016] Upsampling the multi-scale feature map through the passable area segmentation branch, inputting the up-sampled multi-scale feature map into a preset probability prediction function for calculation to obtain a passable area;

[0017] Performing mapping processing on the multi-scale feature map through the obstacle category recognition branch to obtain a probability score for each obstacle category in the multi-scale feature map, and determining the obstacle category with the highest probability score as the current obstacle category;

[0018] The multi-scale feature map is subjected to information extraction processing by a position information extraction branch to obtain position information, where the position information includes at least one of spatial coordinates, relative position relationship, and positioning reliability.

[0019] In some embodiments, acquiring point cloud data and image data of the current area and aligning the point cloud data and the image data includes:

[0020] Obtaining a timestamp of the point cloud data of the current region, a timestamp of the image data, and an extrinsic parameter matrix between the point cloud data and the image data, wherein the extrinsic parameter matrix includes a rotation parameter and a translation vector;

[0021] aligning the timestamps of the point cloud data and the timestamps of the image data according to a time interpolation method;

[0022] The point cloud data and the image data are rotationally aligned according to the rotation parameter, and the point cloud data and the image data are translationally aligned according to the translation vector.

[0023] In some embodiments, the fused map includes a dynamic area and a static area, and after performing fusion conversion processing on the aligned point cloud data and the image data to obtain the fused map, the method further includes:

[0024] Segmenting the fused map according to a preset region segmentation method to obtain a segmented fused map, wherein the segmented fused map includes a plurality of different regions;

[0025] Performing semantic recognition on the segmented fused map according to a visual semantic algorithm to obtain semantic categories of a plurality of different areas in the fused map, wherein the semantic categories include dynamic objects and static objects;

[0026] Calculating motion vectors of point cloud data in a plurality of different regions in the segmented fusion map, and marking regions whose semantic categories are dynamic objects and whose motion vectors are greater than a preset motion threshold as dynamic regions;

[0027] The point cloud data and image data corresponding to the dynamic area are eliminated to obtain a processed fusion map.

[0028] In some embodiments, fusing the multiple target recognition results to obtain a navigation and positioning map includes:

[0029] Calculating the multiple target recognition results according to an evidence theory algorithm to obtain multiple result confidences of different target recognition results in the multiple target recognition results;

[0030] Perform probability distribution calculation on multiple confidence levels of different target recognition results to obtain the recognition accuracy of different target recognition results;

[0031] The target category of the target recognition result corresponding to the current recognition accuracy is determined according to the recognition accuracy, and a navigation positioning map is generated according to the target category.

[0032] In some embodiments, before normalizing the fused map to obtain the normalized fused map, the method further includes:

[0033] The fused map is subjected to bilateral filtering and denoising processing to obtain a processed fused map.

[0034] The embodiment of the present application provides a laser vision fusion navigation and positioning device, comprising:

[0035] An acquisition module, configured to acquire point cloud data and image data of a current area, and perform data alignment on the point cloud data and the image data;

[0036] a processing module, configured to perform fusion conversion processing on the aligned point cloud data and the image data to obtain a fused map;

[0037] an identification module, configured to obtain a plurality of target recognition results based on the fused map and a preset neural network model, the plurality of target recognition results including passable areas, obstacle categories, and location information, the preset neural network model being obtained by training an initial neural network model based on sample data collected in historical navigation and positioning scenarios, the sample data including a plurality of historical fused maps collected in the historical navigation and positioning scenarios, and a plurality of passable areas, a plurality of historical obstacle categories, and a plurality of location information in the historical navigation and positioning scenarios;

[0038] A fusion module is used to fuse the multiple target recognition results according to formula (1) to obtain a navigation positioning map, and the navigation positioning map is used for navigation positioning;

[0039] (1)

[0040] in, Positioning map for navigation, The number of recognition results for the target, is the weight coefficient of the i-th target recognition result, is the i-th target recognition result, and Ci is the confidence of the i-th target recognition result.

[0041] The computer device provided in an embodiment of the present application includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the method described in the embodiment of the present application is implemented.

[0042] The computer-readable storage medium provided in the embodiment of the present application stores a computer program thereon, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0043] The embodiments of the present application provide a laser vision fusion navigation and positioning method, device, equipment, and storage medium, including: obtaining point cloud data and image data of the current area, and achieving calibration through data alignment; performing fusion conversion processing on the aligned data to generate a fusion map containing geometric structure and semantic information; based on the fusion map and a preset neural network model, identifying the passable area, obstacle category, and location information from the fusion map based on the preset neural network model; fusing the above multi-target recognition results to generate a navigation and positioning map, providing environmental model support for the navigation system. In this way, navigation and positioning are performed using the obtained navigation and positioning map, which improves the accuracy and reliability of navigation and positioning in complex scenarios and solves the technical problems raised in the background technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A schematic diagram of the implementation flow of a laser vision fusion navigation and positioning method provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of the implementation flow of another laser vision fusion navigation and positioning method provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of the implementation flow of another laser vision fusion navigation and positioning method provided in an embodiment of the present application;

[0048] Figure 4 A laser vision fusion navigation and positioning device is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the specific technical solutions of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0051] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0052] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are used to distinguish similar or different objects, and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0053] Figure 1 This is a schematic diagram of the implementation process of a laser vision fusion navigation and positioning method provided in an embodiment of the present application. Figure 1 As shown, the method may include the following steps 101 to 104:

[0054] Step 101: Acquire point cloud data and image data of the current area, and perform data alignment on the point cloud data and the image data.

[0055] In an embodiment of the present application, a lidar device can be used to collect point cloud data of the current area. The lidar emits a laser beam at a certain frequency, and calculates the distance between the object and the sensor by measuring the time it takes for the laser beam to be reflected back, thereby obtaining the three-dimensional coordinate information of the object surface and forming point cloud data.

[0056] At the same time, visual sensors such as cameras collect image data of the current area. This image data contains texture, color, and semantic information about the scene. Because there may be a time difference between the LiDAR and camera data acquisition, the two data need to be synchronized. This can be achieved through hardware triggering to ensure that both collect data at the same time; or through software algorithms, data can be interpolated or matched based on timestamps to align the point cloud data and image data in time.

[0057] The LiDAR and camera are installed at different positions and angles, so spatial calibration is required. This can be done using tools such as calibration plates. Through feature matching and coordinate transformation, the relative position and posture relationship between the two can be determined, and the point cloud data and image data can be projected into the same coordinate system.

[0058] Step 102: Perform fusion conversion processing on the aligned point cloud data and image data to obtain a fused map.

[0059] In the embodiment of the present application, geometric features are extracted from point cloud data, such as density, curvature, surface normal, etc. Machine learning-based algorithms, such as principal component analysis, can be used to calculate surface normal and curvature information of the point cloud.

[0060] Extract semantic features from image data, such as object categories, edges, textures, etc. Deep learning models, such as convolutional neural networks, can be used to perform semantic segmentation and object detection on images to extract useful feature information.

[0061] The extracted point cloud geometric features and image semantic features are fused. This can be done by either feature concatenation, where the two features are concatenated in the feature dimension, or by weighted fusion, where different weights are assigned based on the importance of the features before fusion.

[0062] The fused features are processed to generate a fused map. The fused map can be in the form of a raster map, an octree map, etc., where each grid or node contains the geometric information of the point cloud and the semantic information of the image.

[0063] Step 103: Obtain multiple target recognition results based on the fusion map and the preset neural network model.

[0064] In an embodiment of the present application, the current fusion map is input into a trained preset neural network model, and the model outputs multiple target recognition results through forward propagation calculation.

[0065] For the passable area, the model output can be a pixel-level classification probability map, which is converted into a binary passable area mask through threshold processing.

[0066] For obstacle categories, the model outputs the probability that each area belongs to different obstacle categories, and selects the category with the highest probability as the obstacle category of the area.

[0067] For location information, the model can directly output the three-dimensional coordinates or pixel coordinates of obstacles or passable areas.

[0068] The training process for the pre-set neural network model involves collecting sample data from historical navigation and positioning scenarios, including multiple historical fused maps and corresponding annotated data on traversable areas, obstacle categories, and locations. The initial neural network model is then constructed, using structures such as convolutional neural networks and recurrent neural networks.

[0069] The initial neural network model is trained using sample data, and the backpropagation algorithm and optimizer (such as stochastic gradient descent) are used to continuously adjust the model parameters to minimize the error between the model's output and the labeled data.

[0070] Step 104 , fusing multiple target recognition results to obtain a navigation and positioning map, which is used for navigation and positioning.

[0071] In the embodiment of the present application, the recognition results of the passable area, obstacle category and location information are fused. Methods such as voting and weighted average can be used to comprehensively consider the confidence of different results to obtain the final fusion result.

[0072] Post-process the fusion results, such as removing noise, filling holes, smoothing boundaries, etc., to improve the accuracy and reliability of the results.

[0073] According to the fusion processing results, a navigation positioning map is generated. Specifically, the fusion processing is performed according to formula (1) to obtain the navigation positioning map.

[0074] (1)

[0075] in, A navigation and positioning map is a two-dimensional or three-dimensional map data structure that integrates multiple target recognition results. Its specific form can be a raster map, feature map, etc. It is used by the navigation and positioning system for path planning, obstacle avoidance and other operations. is the number of target recognition results. Different target recognition results may come from the output of the neural network model for different features, or target recognition results at different times. is the weight coefficient of the i-th target recognition result, which reflects the importance of the target recognition result in the final fusion. is the i-th target recognition result, which can be a vector or a matrix. The specific form depends on the content of the target recognition. Ci is the confidence level of the i-th target recognition result, which ranges from 0 to 1 and is used to measure the reliability of the target recognition result.

[0076] The navigation positioning map can be a two-dimensional map or a three-dimensional map, which contains information such as the passable area, the location and category of obstacles, etc.

[0077] Apply navigation and positioning maps to navigation and positioning systems to provide path planning and obstacle avoidance guidance for robots, self-driving vehicles, etc.

[0078] By fusing lidar point cloud data with camera image data, the present embodiment leverages the strengths of both, improving the accuracy and reliability of environmental perception. Utilizing a pre-set neural network model for target recognition, it is possible to quickly and accurately determine traversable areas, obstacle categories, and location information. By fusing multiple target recognition results, the resulting navigation and positioning map is more accurate, providing strong support for navigation and positioning.

[0079] In the above Figure 1Based on the implementation flow diagram of a laser vision fusion navigation and positioning method shown in FIG, the present application also provides a implementation flow diagram of a laser vision fusion navigation and positioning method, such as Figure 2 The method may include the following steps 201 to 203:

[0080] Step 201 : normalize the fused map to obtain a normalized fused map.

[0081] In this embodiment of the present application, the collected fused map contains LiDAR point cloud data and camera image data. First, the fused map is linearly normalized, normalizing pixel values to the [0, 1] range. For example, for the height information in the LiDAR point cloud data, its minimum and maximum values are found and processed according to the linear normalization formula.

[0082] Step 202 : performing feature extraction on the normalized fusion map to obtain a multi-scale feature map.

[0083] In this embodiment, a convolutional neural network is used to extract features from the normalized fused map. Different convolutional layers output feature maps of different scales. For example, shallow convolutional layers produce high-resolution feature maps to capture detailed information, while deep convolutional layers produce low-resolution feature maps rich in semantic information. Simultaneously, a feature pyramid network is constructed to fuse feature maps of different scales to produce multi-scale feature maps.

[0084] Step 203: Perform multi-task branch processing on the multi-scale feature map to obtain passable areas, obstacle categories, and location information.

[0085] In this embodiment of the present application, the multi-scale feature map is upsampled through a deconvolution layer to the same size as the original fused map. A convolution layer and a sigmoid function are then used to output a probability map of each pixel belonging to a traversable area. A threshold is set to 0.5, and pixels with a probability greater than 0.5 are marked as traversable areas, while pixels with a probability less than 0.5 are marked as impassable areas, resulting in a binary traversable area mask.

[0086] The multi-scale feature map is input into the fully connected layer, which then passes through the Softmax activation function to output the probability that each region belongs to a different obstacle category. For example, it identifies obstacle categories such as vehicles and pillars and selects the category with the highest probability as the obstacle category for that region.

[0087] The multi-scale feature maps are processed using convolutional and regression layers to output the three-dimensional coordinate information of obstacles. For example, by identifying a vehicle, the X, Y, and Z coordinates of the vehicle in the parking lot coordinate system are output.

[0088] The embodiments of the present application eliminate the scale differences of different sensor data in the fused map through normalization processing, unify data distribution, improve the stability and convergence efficiency of neural network training, and enhance the adaptability of the model to various scenarios; with the help of convolutional neural networks and other methods, multi-scale feature maps are extracted, taking into account both fine-grained geometric details and high-level semantic information, solving the problem of incomplete recognition of single-scale features in complex environments, and improving the comprehensiveness of target recognition; multi-task branch processing enables each target recognition task to focus on exclusive feature optimization, while reducing computational redundancy and achieving feature complementarity, accurately dividing the traversable area, reducing the obstacle misclassification rate, improving the accuracy of location information, enhancing the efficiency and robustness of model reasoning, and adapting to complex dynamic scenarios.

[0089] In the above Figure 2 Based on the implementation flow diagram of a laser vision fusion navigation and positioning method shown in FIG, the present application also provides a implementation flow diagram of a laser vision fusion navigation and positioning method, such as Figure 3 The method may include the following steps 301 to 303:

[0090] In step 301, the multi-scale feature map is upsampled by the passable area segmentation branch, and the upsampled multi-scale feature map is input into a preset probability prediction function for calculation to obtain a passable area.

[0091] In the embodiments of the present application, deconvolution layers (such as transposed convolution) or interpolation algorithms (such as bilinear interpolation) are used to upsample multi-scale feature maps, gradually restoring the feature map resolution to 1 / 2 of the original fused map or the same size. For example, for a deep feature map of size H / 4×W / 4×C, two deconvolutions (with a kernel size of 2×2 and a stride of 2) are used to upsample it to H×W×C / 4, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map, preserving spatial details.

[0092] The upsampled feature map is input into the preset probability prediction function (such as Sigmoid function or Softmax function), and the probability value of each pixel belonging to the passable area is output. .

[0093] Set the probability threshold T (such as 0.5), >T pixels are marked as passable areas and a binary mask is generated .

[0094] In step 302 , the obstacle category recognition branch performs mapping processing on the multi-scale feature map to obtain a probability score for each obstacle category in the multi-scale feature map, and determines the obstacle category with the highest probability score as the current obstacle category.

[0095] In the embodiment of the present application, the multi-scale feature map is mapped to the category dimension through global average pooling or a fully connected layer to obtain the probability score vector of each obstacle category. , where N is the number of preset obstacle categories (such as pedestrians, vehicles, cones, etc.).

[0096] The Softmax function is used to normalize the score vector so that , where zi is the original score output by the branch network.

[0097] The category with the highest likelihood score is selected as the current obstacle category, and the score vector is retained for location confidence evaluation.

[0098] Step 303: Perform information extraction processing on the multi-scale feature map through the position information extraction branch to obtain position information, where the position information includes at least one of spatial coordinates, relative position relationship, and positioning reliability.

[0099] In this embodiment, a regression network directly outputs the 3D spatial coordinates (x, y, z) or 2D pixel coordinates (u, v) of obstacles. For point cloud fusion features, the PointNet++ network is used to extract local point cloud coordinates, and the fully connected layers are used to regress the 3D coordinates in the global coordinate system. For visual features, the 2D coordinates of the image detection frame are converted to 3D world coordinates using camera extrinsics.

[0100] Taking the sensor coordinate system as the origin, the distance, azimuth and height of the obstacle relative to the sensor are calculated to form a relative position vector.

[0101] The uncertainty of the position estimate is expressed by the predicted variance or probability distribution of the output coordinates, which is used by the subsequent navigation decision module to adjust the path planning weights.

[0102] The embodiment of the present application retains spatial details through a dedicated upsampling branch and combines it with pixel-level probability prediction to avoid the boundary blurring problem caused by excessive downsampling in traditional single-task models, making the edge division between the passable area and the obstacle clearer while outputting spatial coordinates, relative position and positioning confidence. It provides the navigation system with hierarchical positioning information and precise coordinates for global path planning. The relative position supports real-time obstacle avoidance, and the positioning confidence assists the decision-making system to dynamically adjust the control strategy, thereby improving the overall positioning reliability and navigation safety.

[0103] In some embodiments, point cloud data and image data of the current area are obtained, and data alignment is performed on the point cloud data and image data, including: obtaining the timestamp of the point cloud data of the current area, the timestamp of the image data, and the extrinsic parameter matrix between the point cloud data and the image data, the extrinsic parameter matrix including rotation parameters and translation vectors.

[0104] In the embodiments of this application, the LiDAR and camera simultaneously record their respective timestamp information when collecting data to identify the time the data was collected. For example, the LiDAR collects point cloud data at a frequency of 10Hz and generates corresponding timestamps, while the camera collects image data at a frequency of 30Hz and records synchronized timestamps.

[0105] The relative position relationship between the lidar and the camera is determined by the joint calibration method, and the extrinsic parameter matrix is obtained. The extrinsic parameter matrix is used to describe the relative position relationship between the lidar coordinate system and the camera coordinate system, and is specifically expressed as a homogeneous transformation matrix: ,in, It is a 3X3 rotation matrix, which consists of the rotation parameters around the three axes of the camera coordinate system. It is used to describe the spatial rotation of the lidar coordinate system relative to the camera coordinate system. It is specifically expressed as: ,in, is the rotation angle (pitch angle) of the lidar coordinate system around the x-axis of the camera coordinate system, in radians. is the rotation angle (roll angle) around the y-axis, in radians, is the rotation angle around the z-axis (yaw angle), in radians, They are the basic rotation matrices around the three axes, following the right-hand screw rule, is a 3×1 translation vector, which represents the three-dimensional coordinate offset of the origin of the lidar coordinate system in the camera coordinate system. The specific meaning is: is the horizontal offset distance (x-axis) of the lidar in the camera coordinate system, in meters; is the vertical (y-axis) offset distance in meters; The offset distance in the depth direction (z axis) is in meters.

[0106] The extrinsic parameter matrix contains rotation parameters and translation vectors. The rotation parameters are used to describe the rotation angle of the lidar coordinate system relative to the camera coordinate system (such as rotation around the x-axis, y-axis, and z-axis); the translation vector is used to describe the three-dimensional coordinate offset of the origin of the lidar coordinate system in the camera coordinate system (such as the distance in the horizontal, vertical, and depth directions).

[0107] Specifically, calibration tools (such as a checkerboard calibration plate combined with dedicated software) can be used to collect multiple sets of point cloud and image data from different perspectives, and accurate extrinsic parameter matrices can be calculated.

[0108] Specifically, the timestamps of the point cloud data and the timestamps of the image data are aligned according to a time interpolation method.

[0109] In the embodiment of the present application, when the timestamps of the laser point cloud and the image are inconsistent, a time interpolation method is used for synchronization. For example, for the timestamp of a certain image data, two adjacent timestamps are searched in the point cloud data, which are respectively earlier and later than the current image timestamp.

[0110] According to the time difference between the two adjacent point cloud data and the corresponding point cloud data, the interpolated point cloud data corresponding to the current image timestamp is calculated by linear interpolation or curve interpolation, so that the point cloud and the image are matched in time.

[0111] To ensure data validity, a time synchronization threshold (e.g., 50 milliseconds) is set, and only point cloud and image pairs with a time difference within the threshold are processed, and invalid data pairs with excessive time deviation are eliminated.

[0112] Specifically, the point cloud data and the image data are rotationally aligned according to the rotation parameter, and the point cloud data and the image data are translationally aligned according to the translation vector.

[0113] In the embodiments of this application, the rotation parameters in the extrinsic matrix are used to adjust the spatial angle of the laser point cloud data. Specifically, the three-dimensional coordinates of each laser point are spatially transformed according to the rotation parameters to eliminate directional deviations caused by differences in sensor installation angles, so that the spatial orientation of the point cloud data remains consistent with the image data.

[0114] Based on the translation vector in the extrinsic matrix, the rotated point cloud data is compensated for positional offset. The origin of the laser point cloud coordinate system is adjusted to the origin of the camera coordinate system, so that the two types of data are in the same reference coordinate system in space, ensuring that the 3D coordinates of the point cloud and the 2D pixels of the image can accurately correspond.

[0115] The embodiment of the present application controls the timestamp error between the point cloud and the image within an extremely small range through the time interpolation method, effectively solving the problem of object position misalignment caused by different sensor sampling frequencies in dynamic scenes, ensuring the temporal consistency of the data, and accurately calibrating the extrinsic parameter matrix and the rotation and translation transformation, so that the position error of the laser point cloud projected onto the image plane is significantly reduced, ensuring that the three-dimensional coordinates of the point cloud accurately correspond to the image semantic information, and providing a high-precision spatial reference for subsequent fusion map generation.

[0116] In some embodiments, the fused map includes dynamic areas and static areas. After the aligned point cloud data and image data are fused and converted to obtain the fused map, the fused map is further divided according to a preset area division method to obtain a segmented fused map, and the segmented fused map includes multiple different areas.

[0117] In some embodiments, the fused map generated by the aligned point cloud data and image data is divided into multiple sub-areas according to the spatial division rules. Specifically, any of the following methods can be used:

[0118] Raster segmentation: Divide the fused map into a uniform grid (such as a 0.5m x 0.5m grid). Each grid contains point cloud data (such as density and height) and image semantic features (such as color and texture) within the area.

[0119] Superpixel segmentation: Based on the color and texture similarity of the image, the continuous area in the fused map is divided into superpixel blocks. Each superpixel block corresponds to a semantically coherent area in the environment (such as the road surface, wall, or vehicle surface).

[0120] Segmentation output: multiple sub-regions containing geometric information (point cloud coordinates) and semantic features (image pixel values) are obtained, providing basic units for subsequent processing.

[0121] Specifically, semantic recognition is performed on the segmented fused map according to a visual semantic algorithm to obtain semantic categories of multiple different areas in the fused map, where the semantic categories include dynamic objects and static objects.

[0122] In some embodiments, a deep learning model or a traditional image algorithm is used to perform semantic analysis on each segmented sub-region to determine its category.

[0123] Dynamic object categories mainly include pedestrians, vehicles, cyclists and other objects with autonomous movement capabilities;

[0124] The static object category mainly includes fixed environmental elements such as buildings, roads, trees, and traffic signs.

[0125] Assign semantic labels to each sub-region (such as "vehicle", "pedestrian", "road surface", "wall") and establish a correspondence between semantic categories and regions.

[0126] Specifically, motion vectors of point cloud data in multiple different areas of the segmented fusion map are calculated, and areas with a semantic category of dynamic objects and a motion vector greater than a preset motion threshold are marked as dynamic areas.

[0127] In some embodiments, the motion information of each region is obtained by any of the following methods:

[0128] Point cloud timing analysis: Compare the changes in point cloud coordinates of the same area in two adjacent fused map frames and calculate the average displacement as the motion vector (e.g., displacement in the x, y, and z directions).

[0129] Visual optical flow method: Calculate the optical flow field of the sub-region based on the image sequence, obtain the pixel-level motion direction and speed, and convert it into the motion vector of the entire region.

[0130] A preset motion threshold (e.g., 0.1 meters per frame) is set. Regions with semantic categories of dynamic objects (e.g., "vehicle" or "pedestrian") and motion vectors greater than the threshold are marked as dynamic regions. For example, if a region is identified as a "vehicle" and its displacement exceeds 0.1 meters per frame, it is considered a dynamically moving vehicle region.

[0131] Specifically, the point cloud data and image data corresponding to the dynamic area are eliminated to obtain a processed fusion map.

[0132] In some embodiments, point cloud data (such as vehicle point cloud) and image data (such as vehicle pixel areas) marked as dynamic areas are removed from the fused map, and only area data with semantic categories of static objects (such as "road surface" and "wall") or dynamic objects but whose motion vectors do not exceed a threshold (such as stationary vehicles) are retained.

[0133] Generate a processed fusion map, which only contains information about static environmental structures and stationary objects, providing stable input data for subsequent target recognition and navigation planning.

[0134] This embodiment of the application uses a dual screening process of semantic recognition and motion analysis to precisely remove regions corresponding to dynamic objects, preventing them from interfering with static maps. This allows the fused map to better align with the fixed structure of the actual environment, improving the long-term reliability of the map. Regional segmentation and semantic recognition combine to transform global dynamic detection into local analysis of subregions, reducing computational complexity. Motion vector calculation is based on the temporal correlation between point clouds and images, avoiding the time-consuming problem of traditional 3D point cloud full-scene matching.

[0135] In some embodiments, multiple target recognition results are fused to obtain a navigation positioning map, including: calculating the multiple target recognition results according to an evidence theory algorithm to obtain multiple result confidence levels of different target recognition results in the multiple target recognition results.

[0136] In the embodiment of the present application, the evidence theory algorithm can comprehensively consider the reliability and uncertainty of different target recognition results. Multiple target recognition results are input into the algorithm as evidence.

[0137] By analyzing the characteristics and credibility of each target recognition result through the relevant computational steps of the evidence theory algorithm, multiple confidence levels are obtained for different target recognition results. These confidence levels reflect the reliability of each recognition result. For example, for a target recognized by a lidar and a camera, the evidence theory algorithm can evaluate the confidence level of the lidar recognition result and the confidence level of the camera recognition result.

[0138] Specifically, a probability distribution calculation is performed on multiple result confidences of different target recognition results to obtain the recognition accuracy rates of different target recognition results.

[0139] In the embodiment of the present application, probabilities are assigned to different target recognition results based on the obtained multiple result confidences, which can more reasonably measure the contribution of each recognition result to the final decision.

[0140] The probability distribution results are used to calculate the recognition accuracy of different target recognition results. The recognition accuracy rate can intuitively reflect the accuracy of each target recognition result. For example, a higher recognition accuracy rate for a target recognition result indicates that the result is more reliable.

[0141] Specifically, the target category of the target recognition result corresponding to the current recognition accuracy is determined according to the recognition accuracy, and a navigation positioning map is generated according to the target category.

[0142] In the embodiment of the present application, the target category of the target recognition result corresponding to the current recognition accuracy is determined based on the calculated recognition accuracy. The target category with the highest recognition accuracy is selected as the final target category. For example, if the recognition accuracy of the lidar identifying the target as "car" is higher than the recognition accuracy of the camera identifying the target as "truck", then the target category is ultimately determined to be "car".

[0143] Based on the determined target category and other relevant information (such as the target's location and size), a navigation and positioning map is generated. The navigation and positioning map accurately reflects the category and distribution of targets in the environment, providing a precise navigation basis for the navigation system.

[0144] The embodiments of this application utilize an evidence-based algorithm to comprehensively process multiple target recognition results, fully accounting for the uncertainty of different recognition results. This allows for more accurate calculation of recognition accuracy, thereby improving target recognition accuracy and reducing the likelihood of misjudgment. The navigation and positioning map generated based on accurate target recognition results can more realistically reflect environmental information, improving map reliability and usability. This helps the navigation system make more reasonable and accurate navigation decisions.

[0145] In some embodiments, before normalizing the fused map to obtain the normalized fused map, the method further includes: performing bilateral filtering denoising on the fused map to obtain the processed fused map.

[0146] In the embodiment of the present application, the fused map includes the three-dimensional coordinate data of the laser point cloud and the two-dimensional image data of the camera. The types of noise that may be present include:

[0147] Gaussian noise: manifests as random offsets in point cloud coordinates or random perturbations in image pixel values (such as lidar measurement errors and camera sensor thermal noise).

[0148] Salt and pepper noise: manifests as isolated outliers (such as distant mismeasured points in a point cloud) or black and white noise in an image;

[0149] Mixed noise: The superposition of two or more types of noise in a complex environment causes local data anomalies in the fused map.

[0150] According to the resolution and noise intensity of the fused map, the spatial radius of the bilateral filter (such as 3×3, 5×5 pixels) and the pixel value difference threshold (such as the grayscale value difference does not exceed 20) are set.

[0151] For each pixel in the fused map, the spatial distance weight and pixel value similarity weight of all pixels in its neighborhood are calculated, and the weighted average is used as the new value of the pixel.

[0152] If the fused map contains multi-channel data (such as the X, Y, and Z coordinates of a point cloud and the RGB color values of an image), bilateral filtering is performed on each channel to ensure that the noise in each dimension of data is effectively suppressed.

[0153] The embodiment of the present application can remove noise in a targeted manner through bilateral filtering while preserving important structural features in the fused map, avoiding the edge blurring problem caused by traditional filtering methods. The data distribution of the fused map after denoising is more stable during normalization.

[0154] Although this application provides method operation steps such as embodiments or flowcharts, more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in this embodiment is only one way of executing the steps among many, and does not represent the only execution order. When an actual device or client product executes, the method can be executed sequentially according to the embodiment or the accompanying drawings, or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0155] like Figure 4 As shown, the embodiment of the present application also provides a laser vision fusion navigation and positioning device 400. The device includes:

[0156] An acquisition module 401 is used to acquire point cloud data and image data of the current area and perform data alignment on the point cloud data and the image data;

[0157] The processing module 402 is used to perform fusion conversion processing on the aligned point cloud data and image data to obtain a fused map;

[0158] Identification module 403 is configured to obtain multiple target recognition results based on the fused map and a preset neural network model, the multiple target recognition results including traversable areas, obstacle categories, and location information. The preset neural network model is obtained by training an initial neural network model based on sample data collected in historical navigation and positioning scenarios. The sample data includes multiple historical fused maps collected in historical navigation and positioning scenarios, as well as multiple traversable areas, multiple historical obstacle categories, and multiple location information in the historical navigation and positioning scenarios.

[0159] The fusion module 404 is used to fuse multiple target recognition results according to formula (1) to obtain a navigation positioning map, which is used for navigation positioning;

[0160] (1)

[0161] in, Positioning map for navigation, is the number of target recognition results, is the weight coefficient of the i-th target recognition result, is the i-th target recognition result, and Ci is the confidence of the i-th target recognition result.

[0162] In some embodiments, the processing module 402 is further configured to perform normalization processing on the fused map to obtain a normalized fused map;

[0163] The processing module 402 is further used to perform feature extraction on the normalized fusion map to obtain a multi-scale feature map;

[0164] The processing module 402 is further configured to perform multi-task branch processing on the multi-scale feature map to obtain passable areas, obstacle categories, and location information.

[0165] In some embodiments, the processing module 402 is further configured to perform upsampling processing on the multi-scale feature map through the traversable area segmentation branch, and input the up-sampled multi-scale feature map into a preset probability prediction function for calculation to obtain a traversable area;

[0166] The processing module 402 is further configured to perform mapping processing on the multi-scale feature map through the obstacle category identification branch to obtain a probability score for each obstacle category in the multi-scale feature map, and determine the obstacle category with the highest probability score as the current obstacle category;

[0167] The processing module 402 is further configured to perform information extraction processing on the multi-scale feature map through a position information extraction branch to obtain position information, where the position information includes at least one of spatial coordinates, relative position relationship, and positioning reliability.

[0168] In some embodiments, the acquisition module 401 is further configured to acquire a timestamp of the point cloud data of the current region, a timestamp of the image data, and an extrinsic parameter matrix between the point cloud data and the image data, where the extrinsic parameter matrix includes a rotation parameter and a translation vector.

[0169] The processing module 402 is further configured to align the timestamps of the point cloud data and the timestamps of the image data according to a time interpolation method;

[0170] The processing module 402 is further configured to perform rotational alignment on the point cloud data and the image data according to the rotation parameter, and perform translational alignment on the point cloud data and the image data according to the translation vector.

[0171] In some embodiments, the processing module 402 is further configured to segment the fused map according to a preset region segmentation method to obtain a segmented fused map, where the segmented fused map includes a plurality of different regions.

[0172] The processing module 402 is further configured to perform semantic recognition on the segmented fused map according to a visual semantic algorithm to obtain semantic categories of multiple different regions in the fused map, where the semantic categories include dynamic objects and static objects.

[0173] The processing module 402 is further configured to calculate motion vectors of the point cloud data in a plurality of different regions in the segmented fused map, and mark regions with a semantic category of dynamic objects and a motion vector greater than a preset motion threshold as dynamic regions;

[0174] The processing module 402 is further configured to remove the point cloud data and image data corresponding to the dynamic area to obtain a processed fused map.

[0175] In some embodiments, the recognition module 403 is further configured to calculate the multiple target recognition results according to the evidence theory algorithm to obtain multiple result confidences of different target recognition results in the multiple target recognition results;

[0176] The recognition module 403 is further configured to perform probability distribution calculation on multiple result confidences of different target recognition results to obtain recognition accuracy rates of different target recognition results;

[0177] The recognition module 403 is further configured to determine the target category of the target recognition result corresponding to the current recognition accuracy rate according to the recognition accuracy rate, and generate a navigation positioning map according to the target category.

[0178] In some embodiments, the processing module 402 is further configured to perform bilateral filtering and denoising on the fused map to obtain a processed fused map.

[0179] Some modules in the apparatus described herein may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0180] The devices or modules described in the above application embodiments can be implemented by computer chips or physical devices, or by products with certain functions. For ease of description, the above devices are described separately by function in various modules. When implementing the embodiments of this application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0181] The methods, devices, or modules described herein can be implemented in the form of computer-readable program code. The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the memory control logic. Those skilled in the art will also appreciate that, in addition to implementing the controller in pure computer-readable program code, the controller can also be implemented in the form of logic gates, switches, an application-specific integrated circuit, a programmable logic controller, an embedded microcontroller, etc. by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the means for implementing various functions may be considered to be both a software module for implementing the method and a structure within a hardware component.

[0182] An embodiment of the present application further provides a device comprising: a processor; a memory for storing processor-executable instructions; and when the processor executes the executable instructions, the method described in the embodiment of the present application is implemented.

[0183] The embodiments of the present application also provide a non-volatile computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, the method described in the embodiments of the present application is implemented.

[0184] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist independently, or two or more modules may be integrated into one module.

[0185] The above-mentioned storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD), or memory card. Such memory can be used to store computer program instructions.

[0186] Through the description of the above implementation methods, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, or can be embodied through the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application or certain parts of the embodiments.

[0187] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0188] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A laser vision fusion navigation and positioning method, characterized in that: The method comprises: Acquire point cloud data and image data of the current area, and perform data alignment on the point cloud data and the image data; Performing fusion conversion processing on the aligned point cloud data and the image data to obtain a fused map; Based on the fused map and a preset neural network model, a plurality of target recognition results are obtained, the plurality of target recognition results including passable areas, obstacle categories, and location information, the preset neural network model being obtained by training an initial neural network model based on sample data collected in historical navigation and positioning scenarios, the sample data including a plurality of historical fused maps collected in the historical navigation and positioning scenarios, and a plurality of passable areas, a plurality of historical obstacle categories, and a plurality of location information in the historical navigation and positioning scenarios; The multiple target recognition results are fused according to formula (1) to obtain a navigation positioning map, which is used for navigation positioning; （1） in, Positioning map for navigation, The number of recognition results for the target, is the weight coefficient of the i-th target recognition result, is the i-th target recognition result, Ci is the confidence of the i-th target recognition result; According to the fusion map and the preset neural network model, multiple target recognition results are obtained, including: Normalizing the fused map to obtain a normalized fused map; Performing feature extraction on the normalized fusion map to obtain a multi-scale feature map; Performing multi-task branch processing on the multi-scale feature map to obtain passable areas, obstacle categories, and location information; The multi-task branch includes a traversable area segmentation branch, an obstacle category recognition branch, and a position information extraction branch; The multi-task branch processing is performed on the multi-scale feature map to obtain the passable area, obstacle category and location information, including: Upsampling the multi-scale feature map through the passable area segmentation branch, inputting the up-sampled multi-scale feature map into a preset probability prediction function for calculation to obtain a passable area; Performing mapping processing on the multi-scale feature map through the obstacle category recognition branch to obtain a probability score for each obstacle category in the multi-scale feature map, and determining the obstacle category with the highest probability score as the current obstacle category; The multi-scale feature map is subjected to information extraction processing by a position information extraction branch to obtain position information, where the position information includes at least one of spatial coordinates, relative position relationship, and positioning reliability.

2. The method according to claim 1, characterized in that The acquiring point cloud data and image data of the current area and performing data alignment on the point cloud data and the image data includes: Obtaining a timestamp of the point cloud data of the current region, a timestamp of the image data, and an extrinsic parameter matrix between the point cloud data and the image data, wherein the extrinsic parameter matrix includes a rotation parameter and a translation vector; aligning the timestamps of the point cloud data and the timestamps of the image data according to a time interpolation method; The point cloud data and the image data are rotationally aligned according to the rotation parameter, and the point cloud data and the image data are translationally aligned according to the translation vector.

3. The method according to claim 1, characterized in that The fused map includes a dynamic area and a static area; After the aligned point cloud data and the image data are fused and converted to obtain a fused map, the method further includes: Segmenting the fused map according to a preset region segmentation method to obtain a segmented fused map, wherein the segmented fused map includes a plurality of different regions; Performing semantic recognition on the segmented fused map according to a visual semantic algorithm to obtain semantic categories of a plurality of different areas in the fused map, wherein the semantic categories include dynamic objects and static objects; Calculating motion vectors of point cloud data in a plurality of different regions in the segmented fusion map, and marking regions whose semantic categories are dynamic objects and whose motion vectors are greater than a preset motion threshold as dynamic regions; The point cloud data and image data corresponding to the dynamic area are eliminated to obtain a processed fusion map.

4. The method according to claim 1, wherein The fusing of the plurality of target recognition results to obtain a navigation positioning map includes: Calculating the multiple target recognition results according to an evidence theory algorithm to obtain multiple result confidences of different target recognition results in the multiple target recognition results; Perform probability distribution calculation on multiple confidence levels of different target recognition results to obtain the recognition accuracy of different target recognition results; The target category of the target recognition result corresponding to the current recognition accuracy is determined according to the recognition accuracy, and a navigation positioning map is generated according to the target category.

5. The method according to claim 1, wherein Before normalizing the fused map to obtain the normalized fused map, the method further includes: The fused map is subjected to bilateral filtering and denoising processing to obtain a processed fused map.

6. A laser vision fusion navigation and positioning device, characterized in that: include: An acquisition module, configured to acquire point cloud data and image data of a current area, and perform data alignment on the point cloud data and the image data; a processing module, configured to perform fusion conversion processing on the aligned point cloud data and the image data to obtain a fused map; an identification module, configured to obtain a plurality of target recognition results based on the fused map and a preset neural network model, the plurality of target recognition results including passable areas, obstacle categories, and location information, the preset neural network model being obtained by training an initial neural network model based on sample data collected in historical navigation and positioning scenarios, the sample data including a plurality of historical fused maps collected in the historical navigation and positioning scenarios, and a plurality of passable areas, a plurality of historical obstacle categories, and a plurality of location information in the historical navigation and positioning scenarios; A fusion module is used to fuse the multiple target recognition results according to formula (1) to obtain a navigation positioning map, and the navigation positioning map is used for navigation positioning; （1） in, Positioning map for navigation, The number of recognition results for the target, is the weight coefficient of the i-th target recognition result, is the i-th target recognition result, Ci is the confidence of the i-th target recognition result; The processing module is further configured to obtain multiple target recognition results based on the fusion map and a preset neural network model, wherein: Normalizing the fused map to obtain a normalized fused map; Performing feature extraction on the normalized fusion map to obtain a multi-scale feature map; Performing multi-task branch processing on the multi-scale feature map to obtain passable areas, obstacle categories, and location information; The multi-task branch includes a traversable area segmentation branch, an obstacle category recognition branch, and a position information extraction branch; The processing module is further configured to perform multi-task branch processing on the multi-scale feature map to obtain passable areas, obstacle categories, and location information, wherein: Upsampling the multi-scale feature map through the passable area segmentation branch, inputting the up-sampled multi-scale feature map into a preset probability prediction function for calculation to obtain a passable area; Performing mapping processing on the multi-scale feature map through the obstacle category recognition branch to obtain a probability score for each obstacle category in the multi-scale feature map, and determining the obstacle category with the highest probability score as the current obstacle category; The multi-scale feature map is subjected to information extraction processing by a position information extraction branch to obtain position information, where the position information includes at least one of spatial coordinates, relative position relationship, and positioning reliability.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images