Dynamic scene laser mapping and positioning method based on semantic information visual enhancement

Abstract

The invention discloses a dynamic scene laser mapping and positioning method based on semantic information visual enhancement, and belongs to the field of navigation positioning. Comprising the steps that observation data of multiple sensors are acquired, and the multiple sensors comprise a laser radar and a visual sensor; establishing a coordinate conversion relation of the visual sensor relative to the laser radar according to the observation data; according to the coordinate conversion relation, mapping the visual semantic information to a laser radar coordinate system to obtain semantic enhancement data; according to the semantic enhancement data, pose estimation of the robot is updated under a particle filtering framework; constructing a composite map containing semantic information and geometric information according to the updated pose estimation; and executing a navigation task according to the composite map. Through deep fusion of visual semantic information and laser radar data, the positioning precision and navigation efficiency of the robot in a dynamic scene are effectively improved, and reliable technical support is provided for intelligent upgrading of the manufacturing industry.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot intelligent perception and autonomous navigation, and in particular relates to a dynamic scene laser mapping and positioning method based on semantic information visual enhancement. Background Art

[0002] As the manufacturing industry transforms toward intelligent automation, mobile robots are widely used in production and logistics. However, existing navigation technologies struggle to meet the demands of complex industrial environments. Traditional laser SLAM technology suffers from two major drawbacks: First, its two-dimensional sensing characteristics make it prone to overlooking high obstacles like overhead conveyor belts and hanging material racks, leading to robot collisions. Second, it cannot effectively distinguish between dynamic and static objects, making map drift and positioning errors common in scenarios like AGV cluster operations and frequent material handling.

[0003] While visual SLAM technology can provide semantic information, it is severely affected by interference from the industrial environment. Strong light and dust in the workshop can reduce the accuracy of visual sensors, leading to feature matching errors. Furthermore, existing algorithms struggle to adapt to dynamic scenarios involving humans and machines, and the movement of workers and equipment can render map updates ineffective.

[0004] Furthermore, traditional navigation systems rely heavily on geometric maps and lack understanding of environmental semantics, such as "hazardous areas" and "dedicated passages." For example, in semiconductor factories, robots unable to recognize "clean zone" rules can cause contamination. On automotive production lines, robots lacking semantic perception can easily cause logistics congestion. Therefore, the development of SLAM and navigation technologies that integrate semantic information and adapt to dynamic scenarios is urgently needed to advance automation in the manufacturing industry. Summary of the Invention

[0005] To address the limited perception dimensions of single-line LiDAR and the high cost of multi-line LiDAR, this paper proposes a dynamic scene laser mapping and positioning method based on semantic information visual enhancement. By equipping a robot with a binocular camera and a single-line LiDAR, it achieves three-dimensional spatial information acquisition and precise obstacle location.

[0006] To achieve the above objectives, the present invention provides a dynamic scene laser mapping and positioning method based on semantic information visual enhancement, comprising:

[0007] Acquiring observation data from multiple sensors, the multiple sensors including a lidar and a visual sensor;

[0008] Establishing a coordinate transformation relationship between the visual sensor and the laser radar based on the observation data;

[0009] According to the coordinate transformation relationship, the visual semantic information is mapped to the lidar coordinate system to obtain semantically enhanced data;

[0010] updating the pose estimation of the robot under a particle filter framework according to the semantic augmented data;

[0011] constructing a composite map containing semantic information and geometric information according to the updated pose estimation;

[0012] performing a navigation task according to the composite map.

[0013] Preferably, the process of obtaining the observation data of the multi-sensor comprises:

[0014] obtaining environment point cloud data through a laser radar;

[0015] obtaining image data through a visual sensor;

[0016] wherein the image data comprises a depth map and an RGB map.

[0017] Preferably, the process of establishing the coordinate conversion relationship of the visual sensor relative to the laser radar according to the observation data comprises:

[0018] extracting feature points in a common observation region based on the image data and the point cloud data;

[0019] solving a rotation matrix and a translation vector according to the corresponding relationship of the feature points in the two coordinate systems;

[0020] establishing a coordinate conversion relationship between the visual sensor coordinate system and the laser radar coordinate system according to the rotation matrix and the translation vector.

[0021] Preferably, the process of mapping the visual semantic information to the laser radar coordinate system to obtain the semantic augmented data comprises:

[0022] performing target detection according to the image data to obtain semantic labels and their pixel positions in the image;

[0023] converting the pixel positions to three-dimensional coordinates according to the depth map and the camera intrinsic matrix;

[0024] projecting the three-dimensional coordinates to the laser radar coordinate system according to the coordinate conversion relationship to generate semantic augmented data.

[0025] Preferably, the process of updating the pose estimation of the robot under a particle filter framework according to the semantic augmented data comprises:

[0026] identifying a dynamic obstacle region according to the semantic augmented data;

[0027] eliminating the laser point cloud data corresponding to the dynamic obstacle region from the observation data to obtain static observation data;

[0028] According to the static observation data, the particle weight and the pose estimation in the particle filter are updated.

[0029] Preferably, according to the updated pose estimation, the process of constructing the composite map containing semantic information and geometric information comprises:

[0030] According to the static observation data, a geometric grid layer is constructed;

[0031] According to the semantic enhancement data, a semantic grid layer is constructed; the semantic grid layer comprises semantic labels and confidence levels;

[0032] The geometric grid layer and the semantic grid layer are spatially aligned and superimposed to generate a composite map.

[0033] Preferably, according to the composite map, the process of performing a navigation task comprises:

[0034] According to the semantic labels in the composite map, passable areas and forbidden areas are set;

[0035] According to the passable areas, a global path is planned;

[0036] During the movement of the robot, if a dynamic obstacle is detected, then the local path is updated according to the composite map.

[0037] Preferably, the target detection adopts a Mask-RCNN target detection model;

[0038] The output of the Mask-RCNN target detection model comprises the category, mask and bounding box of the target.

[0039] Preferably, in the process of updating the pose estimation of the robot under the particle filter framework, the particle filter adopts an improved Gmapping algorithm;

[0040] The improvement process of the improved Gmapping algorithm comprises:

[0041] A scoring function of semantic enhancement;

[0042] Nonlinear optimization is used to solve the pose parameters;

[0043] Semantic constraints are introduced in the pose search space.

[0044] Preferably, the resolution of the composite map is 5cm*5cm, and each grid unit stores an occupancy probability, a semantic label and a confidence level.

[0045] Compared with the prior art, the present application has the following advantages and technical effects:

[0046] The application significantly enhances the adaptability of the particle filtering algorithm to the dynamic environment by fusing visual semantic information and laser data, effectively solves the positioning deviation problem caused by the blind area of single-line laser radar perception, generates enhanced point cloud data containing geometric contour and semantic attributes, enables the robot to identify high obstacles and distinguish static / dynamic targets, and provides more comprehensive environmental information for navigation decision-making, the deep fusion of semantic map and geometric map gives the robot semantic understanding ability to the industrial scene, can automatically identify functional areas such as "equipment maintenance area" and "material temporary storage area", and improves the intelligent level of task execution; the combination of low-cost binocular camera and single-line laser radar greatly reduces the hardware cost while ensuring high performance, and has significant economic and engineering application value. BRIEF DESCRIPTION OF DRAWINGS

[0047] The drawings constituting a part of this application are used to provide further understanding of the application, the illustrative embodiments of the application and the description thereof are used to explain the application, and do not constitute improper limitation on the application. In the drawings:

[0048] Figure 1 The figure is a schematic diagram of the installation layout and calibration process of the binocular camera and single-line laser radar on the AGV of the embodiment of the application;

[0049] Figure 2 The figure is a schematic diagram of the fusion process of visual semantic three-dimensional information and laser point cloud data of the embodiment of the application;

[0050] Figure 3 The figure is a schematic diagram of the combination of the semantic layer and the laser geometric layer into a composite map and the AGV navigation scene of the embodiment of the application. DETAILED DESCRIPTION

[0051] It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict. The application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0052] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0053] Embodiment one

[0054] The embodiment specifically relates to a multi-sensor fusion semantic mapping and precise positioning navigation method applied to an industrial manufacturing dynamic scene, and aims to solve the problems of low positioning precision and poor navigation efficiency of traditional navigation technology under the interference of dynamic obstacles. It is particularly suitable for flexible production line material handling, intelligent warehouse logistics scheduling and other intelligent manufacturing scenes with strict requirements for environmental perception and dynamic decision-making.

[0055] The embodiment provides a dynamic scene laser mapping and positioning method based on semantic information visual enhancement, comprising the following steps:

[0056] Obtaining observation data of a plurality of sensors, wherein the plurality of sensors comprise a laser radar and a visual sensor;

[0057] According to the observation data, a coordinate conversion relationship of the visual sensor relative to the laser radar is established;

[0058] According to the coordinate conversion relationship, visual semantic information is mapped to a laser radar coordinate system to obtain semantic enhancement data;

[0059] According to the semantic enhancement data, a pose estimation of a robot is updated under a particle filtering framework;

[0060] According to the updated pose estimation, a composite map containing semantic information and geometric information is constructed;

[0061] According to the composite map, a navigation task is performed.

[0062] Further, the process of obtaining observation data of a plurality of sensors comprises the following steps:

[0063] Obtaining environment point cloud data through the laser radar;

[0064] Obtaining image data through the visual sensor;

[0065] The image data comprises a depth map and an RGB map.

[0066] Further, the process of establishing a coordinate conversion relationship of the visual sensor relative to the laser radar according to the observation data comprises the following steps:

[0067] Based on the image data and the point cloud data, feature points in a common observation region are extracted;

[0068] According to the corresponding relationship of the feature points in the two coordinate systems, a rotation matrix and a translation vector are solved;

[0069] According to the rotation matrix and the translation vector, a coordinate conversion relationship between the visual sensor coordinate system and the laser radar coordinate system is established.

[0070] Further, the method of the embodiment comprises four parts of multi-sensor coordinate system calibration, target detection and three-dimensional information acquisition, visual semantic enhancement particle filtering positioning and map construction, and intelligent navigation strategy. Specifically, the following steps are included:

[0071] Calibration of the visual and laser radar coordinate systems to determine a rotation matrix R and a translation vector d of the camera relative to the laser radar;

[0072] Object detection and 3D coordinate estimation based on the Mask-RCNN target detection model, calculating the length, width, and height parameters of the object in the camera coordinate system through the depth map;

[0073] Fusion of visual semantic information and laser data to improve the Gmapping particle filter algorithm, including dynamic obstacle recognition and removal, and semantic enhancement weight update model;

[0074] Construct a multi-level semantic map, including a geometric raster layer and a semantic raster layer, where the semantic raster layer contains category labels and confidence information.

[0075] Furthermore, the visual and lidar coordinate system calibration steps include:

[0076] Set up a regular checkerboard calibration board and obtain the observation data of the camera and lidar on the same calibration board;

[0077] Based on the ground plane constraints and the correspondence between feature points, an optimization objective function is constructed;

[0078] The Levenberg-Marquardt algorithm is used to find the optimal rotation matrix R and translation vector d.

[0079] Specifically, to achieve accurate fusion of visual and lidar data, the camera and lidar are first calibrated together. The workflow mainly includes the following steps:

[0080] The binocular camera and single-line lidar are rigidly fixed to the robot platform to ensure that their relative positions remain unchanged.

[0081] Within the sensor's field of view, multiple checkerboard calibration plates with known spatial positions are arranged to obtain camera images and lidar point cloud data respectively.

[0082] Based on the coordinate correspondence of the feature points of the calibration plate in the two coordinate systems, the least squares method is used to solve the rotation matrix R and translation vector d of the camera relative to the lidar, and the transformation relationship in the unified world coordinate system is established. , laying the foundation for subsequent data fusion.

[0083] More specifically, if Figure 1 The figure shows the joint calibration process for a visual camera and lidar. The lidar and camera on a mobile robot are fixed in position, but their relative pose (R, d) is unknown and needs to be determined through calibration. Prepare multiple 25×25 mm black and white checkerboard calibration plates and place them vertically in the common field of view of the lidar and camera. Ensure that the ground in the calibration area is flat, the ground plane is parallel to the xy plane of the laser coordinate system, the calibration plates are not coplanar, and at least n (recommended n ≥ 10) spatial feature points are formed. Keep the robot stationary during calibration to avoid vibration interference.

[0084] The laser radar coordinate system is used as the reference coordinate system (world coordinate system), denoted as L. The camera coordinate system is denoted as C, and the parameter to be calibrated is the translation vector of the camera relative to the laser coordinate system. And the rotation matrix R. Control the laser radar to scan the calibration plate in space 360 ​​degrees horizontally to obtain point cloud data. Based on the RANSAC algorithm, extract line segment features from the point cloud, identify the corner points of the calibration plate, and generate a point set in the laser coordinate system:

[0085]

[0086] Where, Represents the three-dimensional coordinates of the i-th corner point in the lidar coordinate system.

[0087] The camera is triggered synchronously to capture the image and depth of the calibration plate. The pixel coordinates of the corner points of the calibration plate are extracted using the FAST corner detection algorithm: , Represents the pixel coordinates of the i-th corner point on the RGB image.

[0088] Through the camera intrinsic parameter matrix K and depth value , convert the pixel coordinates into three-dimensional coordinates in the camera coordinate system:

[0089]

[0090] Where, Represents the three-dimensional coordinates of the i-th corner point in the camera coordinate system.

[0091] The conversion formula is:

[0092]

[0093] Where K represents the camera intrinsic parameter matrix, Indicates the depth value.

[0094] The following is a process for optimizing the calibration parameters (R, d):

[0095] Construct two constraint equations. Ground constraint: The plane where the calibration plate is located is parallel to the xy plane of the laser coordinate system, that is, the plane normal vector constraint: .in, , is the plane normal vector in the camera coordinate system. Point correspondence constraint: point in the camera coordinate system After transformation, the point in the laser coordinate system coincide: , and Represent the rotation matrix and translation vector from the camera coordinate system to the laser coordinate system respectively.

[0096] Determine the optimization objective function. Use weighted least squares method to construct the objective function:

[0097]

[0098] in: is the weight corresponding to the point, which can be dynamically adjusted based on the measurement accuracy of the lidar and camera. In this embodiment, the value is 0.8. represents the i-th corner point in the laser coordinate system, Indicates the camera coordinate system The corresponding corner points. is the plane constraint weight coefficient, which is used to balance the strength of geometric constraints. In this embodiment, the value is 0.2.

[0099] Solution algorithm: First, the iterative closest point (ICP) algorithm is used to estimate the initial value. Then, the Levenberg-Marquardt algorithm is used to perform nonlinear optimization of the objective function to solve for the optimal rotation matrix R and translation vector d. Finally, singular value decomposition (SVD) is used to ensure that R is an orthogonal matrix that satisfies the rotation matrix constraints.

[0100] Furthermore, the process of mapping the visual semantic information to the LiDAR coordinate system to obtain semantically enhanced data includes:

[0101] Perform object detection based on image data to obtain semantic labels and their pixel locations in the image;

[0102] Convert pixel positions into three-dimensional coordinates based on the depth map and camera intrinsic parameter matrix;

[0103] According to the coordinate transformation relationship, the three-dimensional coordinates are projected into the lidar coordinate system to generate semantically enhanced data.

[0104] Furthermore, the specific process of object detection and 3D coordinate estimation includes:

[0105] Use the Mask-RCNN model to perform instance segmentation on images and identify static obstacles, dynamic obstacles, and background in industrial scenes;

[0106] Combine the depth map and the camera intrinsic parameter matrix to calculate the 3D bounding box parameters of each object, including the center point and length, width and height;

[0107] Project the object onto a horizontal plane to determine its footprint and geometric outline in two dimensions.

[0108] More specifically, when acquiring object detection and 3D information, the Mask-RCNN object detection model performs transfer learning based on the COCO dataset and, in combination with the requirements of industrial manufacturing scenarios, adds data from specific categories such as machine tools, shelves, and AGVs. The mask is used to capture the object's contour features. This is combined with the depth map captured by the binocular camera to calculate its length, width, and height, and then projected onto a horizontal plane to generate 2D semantic information that can be fused with the laser grid map.

[0109] Furthermore, based on the semantic enhancement data, the process of updating the robot's pose estimation in the particle filter framework includes:

[0110] Identify dynamic obstacle areas based on semantically enhanced data;

[0111] Eliminate the laser point cloud data corresponding to the dynamic obstacle area from the observation data to obtain static observation data;

[0112] Based on static observation data, the particle weights and pose estimates in the particle filter are updated.

[0113] Furthermore, the specific process of fusing visual semantic information with laser data includes:

[0114] The laser point cloud is divided into static and dynamic areas based on semantic labels. The dynamic area includes pedestrians, mobile robots, etc.

[0115] In the particle filter framework, using a clean static laser point cloud Updated maps;

[0116] Optimize the initial pose estimation through semantic matching to reduce the accumulated error of the odometry.

[0117] Specifically, when performing particle filtering for visual semantic enhancement, target information recognized by the camera is converted into laser point cloud-like data, supplementing the vertical blind spots of single-line lidar and enhancing the representation of environmental features. For dynamic obstacles, semantic information is used to quickly identify and mark the corresponding laser point cloud areas, and this data is removed during the particle filtering process to prevent dynamic objects from interfering with static map construction.

[0118] More specifically, if Figure 2 As shown in the figure, the process of fusing visual semantic information with lidar data is shown. The workflow mainly includes the following steps: instance segmentation based on Mask-RCNN; 3D information extraction and semantic layer construction; visual and lidar data fusion.

[0119] Further optimization of the technical solution, instance segmentation based on Mask-RCNN involves using industrial manufacturing scene dataset to perform transfer learning on the Mask-RCNN model, and the dataset covers typical static obstacles (such as machine tools, shelves, workbenches), dynamic obstacles (such as workers, mobile robots, AGVs), and background elements (walls, floors, ceilings). During the training process, the adaptability of the model to the complex environment of the industrial scene is improved by adjusting the backbone network parameters and optimizing the loss function (such as increasing the weight of small target detection). After training, the lightweight model is deployed to the robot edge computing unit to ensure real-time inference performance.

[0120] Further optimization of the technical solution, three-dimensional information extraction and semantic layer construction involves target three-dimensional parameter calculation and semantic layer construction. Based on the camera depth map, combined with the camera intrinsic matrix K and the calibration parameters (R, d), the three-dimensional coordinates of the segmented target in the camera coordinate system are calculated. By extracting the target contour points, the length, width, height dimensions and center point position are determined, forming a three-dimensional bounding box description, such as shown in Figure 2 The target three-dimensional information is projected onto the horizontal plane to generate a two-dimensional semantic grid map, and each grid stores the semantic label (such as "shelf" "pedestrian") of the corresponding area, the confidence and the occupancy state.

[0121] Further optimization of the technical solution, fusion of visual and lidar data involves fusion of visual and lidar data and coordinate conversion. The semantic point cloud in the camera coordinate system (i.e. the two-dimensional semantic grid layer) is converted to the lidar coordinate system through the rotation matrix R and the translation vector d obtained by calibration. This process ensures that the visual and lidar data are aligned under the same spatial reference. Static information fusion: Project the semantic point cloud of static obstacles onto the lidar x-y plane and superimpose it with the lidar point cloud as supplementary data to participate in the filtering process and grid map construction process. Dynamic information isolation: Label the lidar point cloud in the dynamic obstacle area and exclude this part of data during map updating to avoid map distortion caused by target movement.

[0122] Further, according to the updated pose estimation, the process of constructing a composite map containing semantic information and geometric information includes:

[0123] Construct a geometric grid layer according to static observation data;

[0124] Construct a semantic grid layer according to semantic enhancement data; the semantic grid layer includes semantic labels and confidence;

[0125] Spatially align and superimpose the geometric grid layer and the semantic grid layer to generate a composite map.

[0126] Furthermore, when constructing a semantic map, a two-layer map structure is established, consisting of a traditional laser raster layer (recording environmental geometric information) and a semantic raster layer (storing semantic attributes such as target category and confidence). Visual semantic information is projected onto the raster map, and each grid is assigned a semantic label, achieving a multi-level representation of environmental information.

[0127] Furthermore, the steps of constructing a multi-level semantic map include:

[0128] Geometric grid layer: A traditional occupancy grid map constructed based on the laser point cloud, expressing the geometric structure of the environment;

[0129] Semantic raster layer: Each raster contains a semantic label (machine tool, shelf, channel) and a confidence value, generated by projecting visual semantic information;

[0130] Map fusion mechanism: overlay the semantic raster layer with the geometric raster layer to form a semantically enhanced composite map.

[0131] Furthermore, the present invention proposes a mapping scheme that deeply integrates semantic information with laser grid maps. By constructing a composite structure of semantic occupancy grid map layers and laser geometric grid map layers, a multi-level expression of environmental information is achieved, such as Figure 3 The workflow mainly includes the following steps:

[0132] Semantic occupancy grid definition. Figure 1 The semantic occupancy grid map is constructed with a consistent spatial resolution (5x5cm) and a semantic label and confidence value is assigned to each grid. The form is: p grid occupancy probability (range is 0-1, 0.5 means unknown); label is the semantic category label (such as "workbench", "conveyor belt", "employee"); confidence is the semantic classification confidence.

[0133] Semantic occupancy grid map generation process: Project the visual semantic segmentation results (output of the Mask-RCNN object detection model) onto the horizontal plane and combine them with the robot pose Perform coordinate transformation; divide the semantic object area into grid cells, and store the semantic category with the maximum probability and confidence score in each grid.

[0134] Map fusion strategy. Establish a two-layer map structure ,in is the semantic layer, For the laser geometry layer. Through the logical merge operation Generate a fused navigation map, in which: the laser grid map provides accurate geometric contour information; the semantic grid supplements high-level semantic knowledge such as obstacle attributes and regional functions.

[0135] Navigation and positioning optimization in dynamic scenarios. Integrate semantic information into the AMCL particle filter framework, optimize particle weight calculation through semantic constraints (such as known static obstacle positions), and improve positioning accuracy. During the path search process, use the generated fusion navigation map Generate a more reasonable navigation path, primarily utilizing alternative path information in the semantic map (such as backup channels and temporary passages) to avoid entering dangerous areas and achieve real-time global path replanning based on the A-star algorithm. When a dynamic obstacle is detected, local obstacle avoidance based on the DWA (Dynamic Window Algorithm) is triggered to quickly adjust the robot's movement direction.

[0136] Furthermore, according to the composite map, the process of performing the navigation task includes:

[0137] According to the semantic labels in the composite map, the passable areas and prohibited areas are set;

[0138] Plan the global path based on the traversable area;

[0139] During the robot's motion, if a dynamic obstacle is detected, the local path is updated according to the composite map.

[0140] Furthermore, the resolution of the composite map is 5cm×5cm, and each grid cell stores the occupancy probability, semantic label, and confidence.

[0141] Furthermore, during the navigation phase, semantic maps are used for path planning and dynamic obstacle avoidance, including: defining traffic rules based on semantic labels, such as prohibiting entry into the machine tool operating area and giving priority to the use of safe passages; when a dynamic obstacle is detected, its movement trend is predicted in combination with semantic information to generate a safe obstacle avoidance path.

[0142] Specifically, the intelligent navigation strategy includes three steps: path planning, real-time positioning, and dynamic obstacle avoidance. Based on the fused semantically enhanced map, the A-star algorithm is used to plan a global path, incorporating semantic labels (such as "dangerous area" and "dedicated channel") to generate a safe and efficient navigation route. The AMCL (Adaptive Monte Carlo Localization) algorithm, combined with semantic constraints, optimizes particle weights to achieve high-precision real-time positioning of the robot in dynamic environments. When a dynamic obstacle is detected, a local obstacle avoidance strategy based on the DWA (Dynamic Windowing Approach) is triggered. This strategy uses semantic information to predict the obstacle's motion trend and adjusts the robot's trajectory in real time.

[0143] Furthermore, the Mask-RCNN target detection model is used for target detection;

[0144] The output of the Mask-RCNN object detection model includes the category, mask, and bounding box of the object.

[0145] Furthermore, in the process of updating the robot’s pose estimation under the particle filter framework, the particle filter adopts an improved Gmapping algorithm;

[0146] The improvement process of the improved Gmapping algorithm includes:

[0147] Semantically enhanced scoring function;

[0148] Solve the pose parameters by nonlinear optimization;

[0149] Introducing semantic constraints in the pose search space.

[0150] Furthermore, based on the Gmapping particle filter algorithm, this invention optimizes the map construction and pose estimation process by introducing visual semantic information to address the dynamic characteristics of manufacturing scenarios. Specific improvements focus on three core areas: dynamic obstacle handling, initial pose optimization, and importance weight updating. Specifically, estimation accuracy and efficiency are improved through the following methods:

[0151] In a particle filtering framework, dynamic obstacles are identified and filtered from laser point cloud data using visual semantic information. First, using Mask-RCNN instance segmentation results, the projection areas of detected dynamic obstacles (such as mobile robots and workers) in the lidar coordinate system are marked. Then, point sets corresponding to dynamic obstacles are removed from the laser point cloud data to prevent interference with static map construction. Finally, the remaining static laser point cloud is fused with visual semantic information (such as static obstacle outlines and locations) to form enhanced observation data.

[0152] Optimize the initial pose through semantic matching positioning. Extract the center of mass coordinates of the visually segmented target and calculate the translation vector d through target center of mass matching, replacing the initial pose estimate provided by the traditional odometry. This strategy uses semantic information constraints to reduce the cumulative error of the initial pose caused by the odometry. The pose prediction is updated as follows:

[0153]

[0154] in, represents the i-th sampling particle at the current time t, is the i-th sampled particle at the previous moment (t-1), d represents the translation of the particle from moment (t-1) to moment t, Represents a state transfer operation, defined as: , Represents the translation of the world coordinate system in the x-axis direction, Indicates the translation along the y-axis of the world coordinate system.

[0155] After processing, the updated map is obtained and the current observation Then, the particle filter algorithm is performed based on the fused point cloud data. The update equation is as follows:

[0156]

[0157] in, represents the map of the i-th particle at time (t-1), represents the i-th sampling particle at the current time t, is the i-th sampling particle at the current moment after maximum likelihood probability estimation.

[0158] Incorporating visual semantic information improves the speed and accuracy of particle filter estimation. In formula (6), particle The solution process is as follows:

[0159] Based on the vscao algorithm, a visual semantic constraint reconstruction scoring function is introduced .

[0160]

[0161] Where S(R,d) represents the reconstruction scoring function of the rotation matrix R and the translation vector d, M represents the number of point clouds, represents the weight of the j-th point cloud, and , and are the normal vectors of the reference point cloud and the observation point cloud respectively; this function comprehensively considers the matching degree between the laser point cloud and the semantic information, and the weights Dynamically adjusted based on data source (laser / vision) and confidence level. Measured by normal consistency , strengthen the geometric constraints of point cloud alignment, set the threshold Avoid over-punishing small deviations.

[0162] particle By the rotation parameter and translation parameters (x, y), belonging to the special Euclidean group , using nonlinear least squares methods (such as the Levenberg-Marquardt algorithm) to solve the scoring function The minimum value of , to obtain the optimal particle ,Right now:

[0163]

[0164] In the particle filter weight update process, visual semantic information is introduced to build a dynamic abnormal data detection and elimination mechanism. First, the spatial distance between each laser point cloud and the semantic segmentation target is calculated. If the distance exceeds the threshold, Points with a distance of 0.3m (recommended) are considered abnormal. Then, through inter-frame semantic object tracking, laser points with sudden position changes are filtered out. Finally, laser points in the dynamic area are clustered by density to eliminate isolated point sets.

[0165] Based on dynamic data culling, the particle weight update formula is improved:

[0166]

[0167] in, represents the pure observation set after removing dynamic outlier data, represents the updated weight of the i-th particle, represents the weight of the i-th particle before updating, represents the conditional probability of particle weight update, represents the i-th particle waiting for weight update, Represents particles Corresponding map.

[0168] Furthermore, semantic information is used to identify specific areas in industrial scenarios, including: dangerous areas (such as areas around running machine tools); temporary passage areas (such as forklift passages and material handling areas); and work areas (such as assembly tables, inspection stations, and other functional areas).

[0169] This embodiment discloses a method for indoor navigation and positioning of mobile robots suitable for dynamic scenes in the manufacturing industry, which realizes precise navigation through multi-sensor collaboration and semantic map construction. First, the camera and laser sensor are calibrated using regular objects such as a chessboard to determine the relative pose parameters of the two and complete the coordinate system alignment; secondly, the Mask-RCNN target detection model is pre-trained based on a self-made industrial scene data set, and integrated with the laser SLAM algorithm and deployed on the robot platform; then, the target recognition result is projected onto the horizontal plane and fused with the grid map generated by the laser SLAM to construct an enhanced map containing semantic information such as static obstacles, dynamic obstacles and background; finally, during the navigation process, when encountering dynamic obstacles, the obstacles are quickly located with the help of visual object detection, and the improved Gmapping particle filter algorithm and the A-star path planning algorithm are combined to realize real-time map updates and dynamic path planning. This embodiment effectively improves the positioning accuracy and navigation efficiency of the robot in dynamic scenes through the deep fusion of visual semantic information and lidar data, providing reliable technical support for the intelligent upgrade of the manufacturing industry.

[0170] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A dynamic scene laser mapping and positioning method based on semantic information visual enhancement, characterized by: include: Acquiring observation data from multiple sensors, the multiple sensors including a lidar and a visual sensor; Establishing a coordinate transformation relationship between the visual sensor and the laser radar based on the observation data; According to the coordinate transformation relationship, the visual semantic information is mapped to the lidar coordinate system to obtain semantically enhanced data; Based on the semantically enhanced data, updating the robot's pose estimation in a particle filtering framework; Based on the updated pose estimate, a composite map containing semantic and geometric information is constructed; A navigation task is performed according to the composite map.

2. The method according to claim 1, characterized in that The process of obtaining multi-sensor observation data includes: Obtain environmental point cloud data through lidar; Acquire image data through visual sensors; The image data includes a depth map and an RGB map.

3. The method according to claim 1, characterized in that The process of establishing a coordinate transformation relationship between the visual sensor and the laser radar based on the observation data includes: Extract feature points in the common observation area based on image data and point cloud data; According to the correspondence between the feature points in the two coordinate systems, a rotation matrix and a translation vector are solved; A coordinate transformation relationship between the visual sensor coordinate system and the laser radar coordinate system is established according to the rotation matrix and the translation vector.

4. The method according to claim 1, wherein The process of mapping visual semantic information to the LiDAR coordinate system to obtain semantically enhanced data includes: Perform object detection based on image data to obtain semantic labels and their pixel locations in the image; Convert the pixel position into three-dimensional coordinates according to the depth map and the camera intrinsic parameter matrix; According to the coordinate transformation relationship, the three-dimensional coordinates are projected into a lidar coordinate system to generate semantic enhancement data.

5. The method according to claim 1, wherein Based on the semantic enhancement data, the process of updating the robot's pose estimation in the particle filter framework includes: identifying dynamic obstacle areas based on the semantic enhancement data; Eliminating the laser point cloud data corresponding to the dynamic obstacle area from the observation data to obtain static observation data; Based on the static observation data, the particle weights and pose estimation in the particle filter are updated.

6. The method according to claim 1, characterized in that Based on the updated pose estimate, the process of constructing a composite map containing semantic and geometric information includes: Construct geometric raster layers based on static observation data; Constructing a semantic raster layer according to the semantic enhancement data; the semantic raster layer includes semantic labels and confidence levels; The geometric grid layer and the semantic grid layer are spatially aligned and superimposed to generate a composite map.

7. The method according to claim 1, characterized in that The process of performing a navigation task according to the composite map includes: According to the semantic labels in the composite map, a passable area and a prohibited area are set; Planning a global path based on the traversable area; During the robot's motion, if a dynamic obstacle is detected, the local path is updated according to the composite map.

8. The method according to claim 4, characterized in that The target detection adopts the Mask-RCNN target detection model; The output of the Mask-RCNN target detection model includes the target category, mask, and bounding box.

9. The method according to claim 1, characterized in that In the process of updating the robot’s pose estimation under the particle filter framework, the particle filter adopts the improved Gmapping algorithm; The improvement process of the improved Gmapping algorithm includes: Semantically enhanced scoring function; Solve the pose parameters by nonlinear optimization; Introducing semantic constraints in the pose search space.

10. The method according to claim 1, characterized in that The resolution of the composite map is 5 cm×5 cm, and each grid cell stores occupancy probability, semantic label and confidence.

Images