Dynamic scene laser mapping and localization method based on semantic information visual enhancement

Abstract

The application discloses a kind of dynamic scene laser mapping and positioning method based on semantic information visual enhancement, belong to navigation positioning field.It includes: obtaining the observation data of multiple sensors, multiple sensors include laser radar and visual sensor;According to observation data, the coordinate conversion relationship of visual sensor relative to laser radar is established;According to the coordinate conversion relationship, the visual semantic information is mapped to laser radar coordinate system, and semantic enhancement data is obtained;According to semantic enhancement data, the pose estimation of robot is updated under the framework of particle filtering;According to the updated pose estimation, the composite map containing semantic information and geometric information is constructed;According to composite map, navigation task is executed.The application effectively improves the positioning accuracy and navigation efficiency of robot in dynamic scene through the deep fusion of visual semantic information and laser radar data, and provides reliable technical support for the intelligent upgrading of manufacturing industry.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot intelligent perception and autonomous navigation, and particularly relates to a dynamic scene laser mapping and positioning method based on semantic information visual enhancement. BACKGROUND

[0002] In the intelligent transformation process of manufacturing industry, mobile robots are widely used in production, logistics and other links, but the existing navigation technology is difficult to meet the needs of complex industrial environments. Traditional laser SLAM technology has two defects: one is that two-dimensional perception characteristics make it easy to ignore high obstacles such as overhead conveyor belts and suspended material racks, resulting in robot collision; the other is that it cannot effectively distinguish dynamic and static objects, and in AGV cluster operation and frequent material handling scenarios, map drift and positioning error are prone to occur.

[0003] Although the visual SLAM technology can provide semantic information, it is seriously disturbed by the industrial environment. Strong light, dust and other factors in the workshop can reduce the accuracy of the visual sensor, leading to feature matching errors; at the same time, the existing algorithm is difficult to adapt to dynamic scenes with mixed workers and equipment, and the movement of workers and equipment will cause the map update to fail.

[0004] In addition, traditional navigation systems mostly rely on geometric maps and lack understanding of environmental semantic information such as "dangerous areas" and "dedicated paths". For example, in a semiconductor workshop, robots cannot identify "clean area" rules, which may cause pollution; in an automobile production line, robots without semantic perception are prone to cause logistics congestion. Therefore, it is urgent to develop SLAM and navigation technology that integrates semantic information and adapts to dynamic scenes to promote the automation level of manufacturing industry. SUMMARY

[0005] In view of the problem of limited perception dimension of single-line laser radar and high cost of multi-line laser radar, the application provides a dynamic scene laser mapping and positioning method based on semantic information visual enhancement. By mounting binocular cameras and single-line laser radars on the robot, three-dimensional space information acquisition and obstacle accurate positioning are realized.

[0006] To achieve the above purpose, the application provides a dynamic scene laser mapping and positioning method based on semantic information visual enhancement, which comprises:

[0007] Obtaining observation data of a plurality of sensors, the plurality of sensors comprising a laser radar and a visual sensor;

[0008] According to the observation data, a coordinate conversion relationship of the visual sensor relative to the laser radar is established;

[0009] According to the coordinate conversion relationship, the visual semantic information is mapped to the laser radar coordinate system to obtain semantic enhancement data;

[0010] updating the pose estimation of the robot under a particle filter framework according to the semantic augmented data;

[0011] constructing a composite map containing semantic information and geometric information according to the updated pose estimation;

[0012] performing a navigation task according to the composite map.

[0013] Preferably, the process of obtaining the observation data of the multi-sensor comprises:

[0014] obtaining environment point cloud data through a laser radar;

[0015] obtaining image data through a visual sensor;

[0016] wherein the image data comprises a depth map and an RGB map.

[0017] Preferably, the process of establishing the coordinate conversion relationship of the visual sensor relative to the laser radar according to the observation data comprises:

[0018] extracting feature points in a common observation region based on the image data and the point cloud data;

[0019] solving a rotation matrix and a translation vector according to the corresponding relationship of the feature points in the two coordinate systems;

[0020] establishing a coordinate conversion relationship between the visual sensor coordinate system and the laser radar coordinate system according to the rotation matrix and the translation vector.

[0021] Preferably, the process of mapping the visual semantic information to the laser radar coordinate system to obtain the semantic augmented data comprises:

[0022] performing target detection according to the image data to obtain semantic labels and their pixel positions in the image;

[0023] converting the pixel positions to three-dimensional coordinates according to the depth map and the camera intrinsic matrix;

[0024] projecting the three-dimensional coordinates to the laser radar coordinate system according to the coordinate conversion relationship to generate semantic augmented data.

[0025] Preferably, the process of updating the pose estimation of the robot under a particle filter framework according to the semantic augmented data comprises:

[0026] identifying a dynamic obstacle region according to the semantic augmented data;

[0027] eliminating the laser point cloud data corresponding to the dynamic obstacle region from the observation data to obtain static observation data;

[0028] According to the static observation data, the particle weight and the pose estimation in the particle filter are updated.

[0029] Preferably, according to the updated pose estimation, the process of constructing the composite map containing semantic information and geometric information comprises:

[0030] According to the static observation data, a geometric grid layer is constructed;

[0031] According to the semantic enhancement data, a semantic grid layer is constructed; the semantic grid layer comprises semantic labels and confidence levels;

[0032] The geometric grid layer and the semantic grid layer are spatially aligned and superimposed to generate a composite map.

[0033] Preferably, according to the composite map, the process of performing a navigation task comprises:

[0034] According to the semantic labels in the composite map, passable areas and forbidden areas are set;

[0035] According to the passable areas, a global path is planned;

[0036] During the movement of the robot, if a dynamic obstacle is detected, then the local path is updated according to the composite map.

[0037] Preferably, the target detection adopts a Mask-RCNN target detection model;

[0038] The output of the Mask-RCNN target detection model comprises the category, mask and bounding box of the target.

[0039] Preferably, in the process of updating the pose estimation of the robot under the particle filter framework, the particle filter adopts an improved Gmapping algorithm;

[0040] The improvement process of the improved Gmapping algorithm comprises:

[0041] A scoring function of semantic enhancement;

[0042] Nonlinear optimization is used to solve the pose parameters;

[0043] Semantic constraints are introduced in the pose search space.

[0044] Preferably, the resolution of the composite map is 5cm*5cm, and each grid unit stores an occupancy probability, a semantic label and a confidence level.

[0045] Compared with the prior art, the present application has the following advantages and technical effects:

[0046] The application significantly enhances the adaptability of the particle filtering algorithm to the dynamic environment by fusing visual semantic information and laser data, effectively solves the positioning deviation problem caused by the blind area of single-line laser radar perception, generates enhanced point cloud data containing geometric contour and semantic attributes, enables the robot to identify high obstacles and distinguish static / dynamic targets, and provides more comprehensive environmental information for navigation decision-making, the deep fusion of semantic map and geometric map gives the robot semantic understanding ability to the industrial scene, can automatically identify functional areas such as "equipment maintenance area" and "material temporary storage area", and improves the intelligent level of task execution; the combination of low-cost binocular camera and single-line laser radar greatly reduces the hardware cost while ensuring high performance, and has significant economic and engineering application value. BRIEF DESCRIPTION OF DRAWINGS

[0047] The drawings constituting a part of this application are used to provide further understanding of the application, the illustrative embodiments of the application and the description thereof are used to explain the application, and do not constitute improper limitation on the application. In the drawings:

[0048] Figure 1 The figure is a schematic diagram of the installation layout and calibration process of the binocular camera and single-line laser radar on the AGV of the embodiment of the application;

[0049] Figure 2 The figure is a schematic diagram of the fusion process of visual semantic three-dimensional information and laser point cloud data of the embodiment of the application;

[0050] Figure 3 The figure is a schematic diagram of the combination of the semantic layer and the laser geometric layer into a composite map and the AGV navigation scene of the embodiment of the application. DETAILED DESCRIPTION

[0051] It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict. The application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0052] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0053] Embodiment one

[0054] The embodiment specifically relates to a multi-sensor fusion semantic mapping and precise positioning navigation method applied to an industrial manufacturing dynamic scene, and aims to solve the problems of low positioning precision and poor navigation efficiency of traditional navigation technology under the interference of dynamic obstacles. It is particularly suitable for flexible production line material handling, intelligent warehouse logistics scheduling and other intelligent manufacturing scenes with strict requirements for environmental perception and dynamic decision-making.

[0055] The embodiment provides a dynamic scene laser mapping and positioning method based on semantic information visual enhancement, comprising the following steps:

[0056] Obtaining observation data of a plurality of sensors, wherein the plurality of sensors comprise a laser radar and a visual sensor;

[0057] According to the observation data, a coordinate conversion relationship of the visual sensor relative to the laser radar is established;

[0058] According to the coordinate conversion relationship, visual semantic information is mapped to a laser radar coordinate system to obtain semantic enhancement data;

[0059] According to the semantic enhancement data, a pose estimation of a robot is updated under a particle filtering framework;

[0060] According to the updated pose estimation, a composite map containing semantic information and geometric information is constructed;

[0061] According to the composite map, a navigation task is performed.

[0062] Further, the process of obtaining observation data of a plurality of sensors comprises the following steps:

[0063] Obtaining environment point cloud data through the laser radar;

[0064] Obtaining image data through the visual sensor;

[0065] The image data comprises a depth map and an RGB map.

[0066] Further, the process of establishing a coordinate conversion relationship of the visual sensor relative to the laser radar according to the observation data comprises the following steps:

[0067] Based on the image data and the point cloud data, feature points in a common observation region are extracted;

[0068] According to the corresponding relationship of the feature points in the two coordinate systems, a rotation matrix and a translation vector are solved;

[0069] According to the rotation matrix and the translation vector, a coordinate conversion relationship between the visual sensor coordinate system and the laser radar coordinate system is established.

[0070] Further, the method of the embodiment comprises four parts of multi-sensor coordinate system calibration, target detection and three-dimensional information acquisition, visual semantic enhancement particle filtering positioning and map construction, and intelligent navigation strategy. Specifically, the following steps are included:

[0071] Calibration of the visual and laser radar coordinate systems to determine a rotation matrix R and a translation vector d of the camera relative to the laser radar;

[0072] Object detection and 3D coordinate estimation based on Mask-RCNN target detection model, through depth map calculation object in camera coordinate system under the length, width, height parameters;

[0073] Fusion of visual semantic information and laser data, improved Gmapping particle filter algorithm, including dynamic obstacle identification and elimination, semantic enhanced weight update model;

[0074] Construct a multi-level semantic map, including geometric grid layer and semantic grid layer, and the semantic grid layer contains class label and confidence information.

[0075] Further, the visual and laser radar coordinate system calibration steps include:

[0076] Set up a regular checkerboard calibration board, and obtain observation data of the camera and laser radar on the same calibration board;

[0077] Based on the ground plane constraint and the corresponding relationship of feature points, an optimization objective function is constructed;

[0078] The Levenberg-Marquardt algorithm is used to solve the optimal rotation matrix R and translation vector d.

[0079] Specifically, to realize the accurate fusion of visual and laser radar data, first, the camera and laser radar are jointly calibrated, and the workflow mainly has the following steps:

[0080] The binocular camera and single-line laser radar are rigidly fixed on the robot platform to ensure that the relative position does not change.

[0081] In the field of view of the sensor, a plurality of checkerboard calibration boards with known spatial positions are arranged, and camera images and laser radar point cloud data are obtained respectively.

[0082] Based on the corresponding relationship of feature points on the calibration board in the two coordinate systems, the least squares method is used to solve the rotation matrix R and translation vector d of the camera relative to the laser radar, and the conversion relationship in the unified world coordinate system is established , which lays the foundation for subsequent data fusion.

[0083] More specifically, as Figure 1 shown, the joint calibration process of the visual camera and the laser radar is shown. The laser radar and the camera on the mobile robot are fixed in position, but the relative pose (R, d) between them is unknown and needs to be determined through calibration. A plurality of black and white checkerboard calibration boards with a size of 25x25mm are prepared, and they are placed vertically in the common field of view of the laser radar and the camera. Ensure that the calibration area is flat, the ground plane is parallel to the x-y plane of the laser coordinate system, the calibration boards are not coplanar, and at least n (recommended n≥10) spatial feature points are formed. During the calibration process, the robot body is kept stationary to avoid vibration interference.

[0084] Let the laser radar coordinate system be the reference coordinate system (world coordinate system), denoted as L. The camera coordinate system is denoted as C, and the parameters to be calibrated are the translation vector and the rotation matrix R of the camera relative to the laser coordinate system. The laser radar is controlled to perform a 360° horizontal plane scan on the calibration board in space to obtain point cloud data. Based on the RANSAC algorithm, a line segment feature is extracted from the point cloud, the corner points of the calibration board are identified, and a point set in the laser coordinate system is generated:

[0085]

[0086] In the formula, represents the three-dimensional coordinates of the i-th corner point in the laser radar coordinate system.

[0087] The camera is triggered synchronously to capture the image and depth of the calibration board. The FAST corner detection algorithm is used to extract the pixel coordinates of the corner points of the calibration board: , represents the pixel coordinates of the i-th corner point on the RGB image.

[0088] Through the camera intrinsic matrix K and the depth value , the pixel coordinates are converted into three-dimensional coordinates in the camera coordinate system:

[0089]

[0090] In the formula, represents the three-dimensional coordinates of the i-th corner point in the camera coordinate system.

[0091] The conversion formula is:

[0092]

[0093] In the formula, K represents the camera intrinsic matrix, represents the depth value.

[0094] The following provides a flow of optimizing and solving the calibration parameters (R, d):

[0095] Two constraint equations are constructed. Ground constraint: the plane on which the calibration board is located is parallel to the x-y plane of the laser coordinate system, that is, the plane normal vector constraint: . Wherein, , is the plane normal vector in the camera coordinate system. Point correspondence constraint: the point in the camera coordinate system coincides with the point in the laser coordinate system after transformation: , and represent the rotation matrix and translation vector of the camera coordinate to the laser coordinate system, respectively.

[0096] An optimization objective function is determined. A weighted least squares method is used to construct the objective function:

[0097]

[0098] wherein: is the weight corresponding to the point, which can be dynamically adjusted based on the measurement accuracy of the lidar and the camera, and in this embodiment, the value is 0.8. represents the i-th corner point in the laser coordinate system, represents the corner point corresponding to in the camera coordinate system. is a plane constraint weight coefficient, used to balance the strength of geometric constraints, and in this embodiment, the value is 0.2.

[0099] The solving algorithm. First, an iterative closest point (ICP) algorithm is used for initial value estimation; then a Levenberg-Marquardt algorithm is used to perform nonlinear optimization on the objective function to solve the optimal rotation matrix R and translation vector d; finally, singular value decomposition (SVD) is used to ensure that R is an orthogonal matrix, satisfying the rotation matrix constraint.

[0100] Further, the process of mapping visual semantic information to the lidar coordinate system to obtain semantic enhanced data includes:

[0101] According to the image data, target detection is performed to obtain semantic labels and their pixel positions in the image;

[0102] According to the depth map and the camera intrinsic matrix, the pixel positions are converted into three-dimensional coordinates;

[0103] According to the coordinate conversion relationship, the three-dimensional coordinates are projected into the lidar coordinate system to generate semantic enhanced data.

[0104] Further, the specific process of object detection and three-dimensional coordinate estimation includes:

[0105] An instance segmentation is performed on the image using a Mask-RCNN model to identify static obstacles, dynamic obstacles, and backgrounds in the industrial scene;

[0106] Combined with the depth map and the camera intrinsic matrix, the three-dimensional bounding box parameters of each object are calculated, including the center point and the length, width, and height dimensions;

[0107] The object is projected onto the horizontal plane to determine its occupancy area and geometric contour in the two-dimensional plane.

[0108] More specifically, in the process of acquiring target detection and three-dimensional information, based on the COCO dataset, combined with the needs of industrial manufacturing scenes, specific category data such as machine tools, shelves, and AGVs are added, and the Mask-RCNN target detection model is migrated. Through the mask mask, the contour features of the object are obtained, combined with the depth map obtained by the binocular camera and the calculation of its length, width and height size, and projected to the horizontal plane, a two-dimensional semantic information that can be fused with the laser grid map is generated.

[0109] Further, according to the semantic enhanced data, the process of updating the pose estimation of the robot in the particle filter framework includes:

[0110] According to the semantic enhanced data, the dynamic obstacle region is identified;

[0111] The laser point cloud data corresponding to the dynamic obstacle region is removed from the observation data to obtain static observation data;

[0112] According to the static observation data, the particle weight and pose estimation in the particle filter are updated.

[0113] Further, the specific process of fusing visual semantic information and laser data includes:

[0114] Based on the semantic label, the laser point cloud is divided into static and dynamic regions, and the dynamic region includes pedestrians, mobile robots, etc.

[0115] In the particle filter framework, the pure static laser point cloud is used to update the map.

[0116] The initial pose estimation is optimized through semantic matching to reduce the cumulative error of the odometer.

[0117] Specifically, in the process of visual semantic enhanced particle filter positioning, the target information recognized by the camera is converted into class laser point cloud data, which supplements the sensing blind area of the single-line laser radar in the vertical dimension and enhances the expression of environmental features. For dynamic obstacles, the corresponding laser point cloud region is quickly identified and marked using semantic information, and this part of data is removed in the particle filter process to avoid interference of dynamic targets with static map construction.

[0118] More specifically, as Figure 2 shown, the process of fusing visual semantic information and laser radar data is shown, and the workflow mainly includes the following steps: instance segmentation based on Mask-RCNN; three-dimensional information extraction and semantic layer construction; fusion of visual and laser radar data.

[0119] Further optimization of the technical solution, instance segmentation based on Mask-RCNN involves using industrial manufacturing scene dataset to perform transfer learning on the Mask-RCNN model, and the dataset covers typical static obstacles (such as machine tools, shelves, workbenches), dynamic obstacles (such as workers, mobile robots, AGVs), and background elements (walls, floors, ceilings). During the training process, the adaptability of the model to the complex environment of the industrial scene is improved by adjusting the backbone network parameters and optimizing the loss function (such as increasing the weight of small target detection). After training, the lightweight model is deployed to the robot edge computing unit to ensure real-time inference performance.

[0120] Further optimization of the technical solution, three-dimensional information extraction and semantic layer construction involves target three-dimensional parameter calculation and semantic layer construction. Based on the camera depth map, combined with the camera intrinsic matrix K and the calibration parameters (R, d), the three-dimensional coordinates of the segmented target in the camera coordinate system are calculated. By extracting the target contour points, the length, width, height dimensions and center point position are determined, forming a three-dimensional bounding box description, such as shown in Figure 2 The target three-dimensional information is projected onto the horizontal plane to generate a two-dimensional semantic grid map, and each grid stores the semantic label (such as "shelf" "pedestrian") of the corresponding area, the confidence and the occupancy state.

[0121] Further optimization of the technical solution, fusion of visual and lidar data involves fusion of visual and lidar data and coordinate conversion. The semantic point cloud in the camera coordinate system (i.e. the two-dimensional semantic grid layer) is converted to the lidar coordinate system through the rotation matrix R and the translation vector d obtained by calibration. This process ensures that the visual and lidar data are aligned under the same spatial reference. Static information fusion: Project the semantic point cloud of static obstacles onto the lidar x-y plane and superimpose it with the lidar point cloud as supplementary data to participate in the filtering process and grid map construction process. Dynamic information isolation: Label the lidar point cloud in the dynamic obstacle area and exclude this part of data during map updating to avoid map distortion caused by target movement.

[0122] Further, according to the updated pose estimation, the process of constructing a composite map containing semantic information and geometric information includes:

[0123] Construct a geometric grid layer according to static observation data;

[0124] Construct a semantic grid layer according to semantic enhancement data; the semantic grid layer includes semantic labels and confidence;

[0125] Spatially align and superimpose the geometric grid layer and the semantic grid layer to generate a composite map.

[0126] Furthermore, when constructing the semantic map, a two-layer map structure is established, including a traditional laser raster layer (recording environmental geometric information) and a semantic raster layer (storing semantic attributes such as target category and confidence level). Visual semantic information is projected onto the raster map, and each raster is assigned a semantic label to achieve multi-level expression of environmental information.

[0127] Furthermore, the steps for constructing a multi-level semantic map include:

[0128] Geometric raster layer: A traditional occupancy raster map built based on laser point clouds, representing the geometric structure of the environment;

[0129] Semantic raster layer: Each raster contains semantic labels (machine tool, shelf, aisle) and confidence values, generated by projecting visual semantic information;

[0130] Map fusion mechanism: Overlay semantic raster layers with geometric raster layers to form a semantically enhanced composite map.

[0131] Furthermore, this invention proposes a mapping scheme that deeply integrates semantic information with laser grid maps. By constructing a composite structure of a semantic occupancy grid map layer and a laser geometric grid map layer, it achieves multi-level expression of environmental information, such as... Figure 3 As shown. The workflow mainly includes the following steps:

[0132] Semantic occupancy grid definition. Adopted with laser grid... Figure 1 A spatial resolution of 5x5cm is used to assign semantic labels and confidence values ​​to each raster. Semantic occupancy raster maps are constructed using... The form is: p = grid occupancy probability (ranging from 0 to 1, with 0.5 indicating unknown); label = semantic category label (e.g., "workbench", "conveyor belt", "employee"); confidence = semantic classification confidence.

[0133] Semantic occupancy grid map generation process: Project the visual semantic segmentation results (output of the Mask-RCNN object detection model) onto the horizontal plane, and combine them with the robot pose. Perform coordinate transformation; divide the semantic object region into grid cells, with each grid cell storing the semantic category with the highest probability and the confidence score.

[0134] Map fusion strategy. Establish a two-layer map structure. ,in For the semantic layer, For laser geometry layers. Through logical merging operations. Generate a fused navigation map, in which: the laser grid map provides accurate geometric contour information; and the semantic grid supplements high-level semantic knowledge such as obstacle attributes and area functions.

[0135] Navigation and positioning optimization in dynamic scenarios. The semantic information is integrated into the AMCL particle filter framework, and the particle weight calculation is optimized through semantic constraints (such as known static obstacle positions) to improve positioning accuracy. In the path search process, the generated fusion navigation map Generate a more reasonable navigation path, mainly use the alternative path information (such as the standby channel, temporary passage) in the semantic map, avoid entering the dangerous area, realize the global path real-time re-planning based on A-star algorithm. When detecting dynamic obstacles, trigger local obstacle avoidance based on DWA (dynamic window method), quickly adjust the robot motion direction.

[0136] Further, according to the composite map, the process of performing a navigation task includes:

[0137] According to the semantic label in the composite map, set the passable area and the prohibited area;

[0138] According to the passable area, plan a global path;

[0139] During the robot motion process, if a dynamic obstacle is detected, update the local path according to the composite map.

[0140] Further, the resolution of the composite map is 5cm x 5cm, and each grid cell stores occupancy probability, semantic label and confidence.

[0141] Further, in the navigation stage, the semantic map is used for path planning and dynamic obstacle avoidance, including: defining the traffic rules based on the semantic label, such as prohibiting entering the machine tool working area, and preferentially using the safety channel; When detecting a dynamic obstacle, predict its motion trend combined with semantic information, and generate a safe obstacle avoidance path.

[0142] Specifically, the intelligent navigation strategy includes three steps of path planning, real-time positioning and dynamic obstacle avoidance. Based on the fused semantic enhanced map, A-star algorithm is used to plan a global path, and combined with semantic labels (such as "dangerous area" and "dedicated channel"), a safe and efficient navigation route is generated. Using AMCL (adaptive Monte Carlo localization) algorithm, combined with semantic constraints to optimize particle weight, high-precision real-time positioning of the robot in dynamic environment is realized. When detecting a dynamic obstacle, trigger the local obstacle avoidance strategy based on DWA (dynamic window method), predict the motion trend of the obstacle combined with semantic information, and adjust the robot motion trajectory in real time.

[0143] Further, the target detection uses a Mask-RCNN target detection model;

[0144] The output of the Mask-RCNN target detection model includes the category, mask and bounding box of the target.

[0145] Further, in the process of updating the robot's pose estimation under the particle filter framework, the particle filter adopts an improved Gmapping algorithm;

[0146] The improvement process of the improved Gmapping algorithm includes:

[0147] The semantic enhanced scoring function;

[0148] Nonlinear optimization to solve the pose parameters;

[0149] Introducing semantic constraints in the pose search space.

[0150] Further, based on the Gmapping particle filter algorithm, the application optimizes the map construction and pose estimation process by introducing visual semantic information in view of the characteristics of the dynamic scene in the manufacturing industry. The specific improvements focus on three core environments: dynamic obstacle processing, initial pose optimization, and importance weight update. Specifically, the estimation accuracy and efficiency are improved in the following ways:

[0151] In the particle filter framework, dynamic obstacle identification and filtering are performed on the laser point cloud data through visual semantic information. First, the projection area of the detected dynamic obstacles (such as mobile robots and workers) in the laser radar coordinate system is labeled using Mask-RCNN instance segmentation results. Then, the point set corresponding to the dynamic obstacles is removed from the laser point cloud data to avoid interfering with static map construction. Finally, the remaining static laser point cloud and visual semantic information (such as static obstacle contours and positions) are fused to form enhanced observation data.

[0152] Optimize the initial pose through semantic matching. The centroid coordinates of the visual segmentation targets are extracted, and the translation vector d is calculated through target centroid matching to replace the initial pose estimation provided by the traditional odometer. This strategy uses semantic information constraints to reduce the cumulative error of the initial pose caused by the odometer, and the pose prediction update is:

[0153]

[0154] where, represents the i-th sampling particle at the current time t, is the i-th sampling particle at the previous time (t-1), and d represents the translation of the particle from the (t-1) time to the t time, represents a state transition operation, which is defined as: , represents the translation in the x-axis direction of the world coordinate system, represents the translation in the y-axis direction of the world coordinate system.

[0155] After processing, the updated map and the current observation value Then, the particle filter algorithm is performed based on the fused point cloud data, and the update equation of the particle is as follows:

[0156]

[0157] wherein, represents the map of the i-th particle at the (t-1) time, represents the i-th sampling particle at the current time t, is the i-th sampling particle at the current time after maximum likelihood probability estimation.

[0158] The fusion of visual semantic information improves the speed and accuracy of particle filter estimation. In formula (6), the particle is solved in the following process:

[0159] On the basis of the vscao algorithm, the visual semantic constraint reconstruction score function is introduced .

[0160]

[0161] In the formula, S(R, d) represents the reconstruction score function of the rotation matrix R and the translation vector d, M represents the number of point clouds, represents the weight of the j-th point cloud, and , and are the normal vectors of the reference point cloud and the observed point cloud, respectively; the function comprehensively considers the matching degree of the laser point cloud and the semantic information, and the weight is dynamically adjusted according to the data source (laser / visual) and the confidence. Through the normal consistency measurement , the geometric constraint of point cloud alignment is strengthened, and the threshold is set to avoid excessive punishment of slight deviations.

[0162] The particle is composed of a rotation parameter and a translation parameter (x, y), and belongs to the special Euclidean group . The minimum value of the score function is obtained by using the nonlinear least squares method (such as the Levenberg-Marquardt algorithm), and the optimal particle is obtained, that is:

[0163]

[0164] In the process of updating the weight of the particle filter, the visual semantic information is introduced to construct a dynamic abnormal data detection and elimination mechanism. First, the spatial distance between each laser point cloud and the semantic segmentation target is calculated, and the distance exceeding the threshold ​The points with a distance of 0.3 m (recommended) are considered as abnormal. Then, the laser points with sudden position changes are filtered by inter-frame semantic object tracking. Finally, the laser points in the dynamic area are clustered by density, and the isolated point sets are removed.

[0165] On the basis of dynamic data removal, the particle weight update formula is improved:

[0166]

[0167] wherein, represents a pure observation set after removing dynamic abnormal value data, represents the updated weight of the i th particle, represents the weight of the i th particle before updating, represents the conditional probability of particle weight updating, represents the i th particle waiting for weight updating, represents the particle corresponding map.

[0168] Further, the semantic information is used to identify specific areas in the industrial scene, including: dangerous areas (such as the area around the running machine tool); temporary passage area (such as forklift channel, material handling area); work area (such as assembly table, detection station and other functional areas).

[0169] The embodiment discloses an indoor navigation positioning method of a mobile robot suitable for a manufacturing dynamic scene, and realizes accurate navigation through multi-sensor cooperation and semantic map construction. First, regular objects such as chessboard grids are used to calibrate the camera and laser sensor, determine the relative pose parameters of the two, and complete the coordinate system alignment; second, the Mask-RCNN target detection model is pre-trained based on the self-made industrial scene data set, and is integrated and deployed to the robot platform together with the laser SLAM algorithm; then, the target recognition result is projected to the horizontal plane, and is fused with the grid map generated by the laser SLAM to construct an enhanced map containing semantic information such as static obstacles, dynamic obstacles and background; finally, in the navigation process, when encountering a dynamic obstacle, the obstacle is quickly located with the help of visual object detection, and the improved Gmapping particle filtering algorithm and A-star path planning algorithm are combined to realize real-time updating of the map and dynamic path planning. Through deep fusion of visual semantic information and laser radar data, the positioning accuracy and navigation efficiency of the robot in the dynamic scene are effectively improved, and reliable technical support is provided for intelligent upgrading of the manufacturing industry.

[0170] The above merely provides the preferred embodiments of the present application, and the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed by the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A dynamic scene laser mapping and localization method based on semantic information visual enhancement, characterized in that, The method comprises: acquiring observation data of a plurality of sensors, the plurality of sensors comprising a laser radar and a vision sensor; establishing a coordinate conversion relationship of the vision sensor relative to the laser radar according to the observation data; mapping vision semantic information to a laser radar coordinate system according to the coordinate conversion relationship to obtain semantic enhanced data; updating a pose estimation of a robot under a particle filtering framework according to the semantic enhanced data; constructing a composite map containing semantic information and geometric information according to the updated pose estimation; performing a navigation task according to the composite map; in the process of updating the pose estimation of the robot under the particle filtering framework, the particle filtering adopts an improved Gmapping algorithm; the improvement process of the improved Gmapping algorithm comprises: a semantic enhanced scoring function; nonlinear optimization for solving pose parameters; introducing semantic constraints in a pose search space; the semantic enhanced scoring function is: where S(R, d) denotes the reconstruction score function for rotation matrix R and translation vector d, M denotes the number of point clouds, denotes the weight of the j-th point cloud, and , and are the normal vectors of the reference point cloud and the observed point cloud, respectively. The scoring function comprehensively considers the matching degree of the laser point cloud and the semantic information, and the weight Dynamic adjustment according to data sources and confidence; through normal consistency measurement Strengthen the geometric constraint of point cloud alignment, set threshold Avoid excessive punishment for minor deviations; the nonlinear optimization for solving pose parameters is: The nonlinear least squares method is used to solve the minimum value of the scoring function and obtain the optimal particle that is: the semantic constraints in the pose search space are: in the process of updating the particle filtering weight, a dynamic abnormal data detection and elimination mechanism is introduced by introducing vision semantic information; on the basis of dynamic data elimination, a particle weight updating formula is improved: wherein, represents the pure observation set after removing dynamic outliers, represents the updated weight of the i-th particle, represents the weight of the i-th particle before updating, represents the conditional probability of particle weight updating, represents the i-th particle waiting for weight updating, represents the particle corresponding map.

2. The method of claim 1, wherein the process of acquiring observation data of a plurality of sensors comprises: acquiring environment point cloud data through a laser radar; acquiring image data through a vision sensor; wherein the image data comprises a depth map and an RGB map.

3. The method of claim 1, wherein the process of establishing a coordinate conversion relationship of the vision sensor relative to the laser radar according to the observation data comprises: based on image data and point cloud data, extracting feature points in a common observation area; solving a rotation matrix and a translation vector according to the corresponding relationship of the feature points in the two coordinate systems; establishing a coordinate conversion relationship between the vision sensor coordinate system and the laser radar coordinate system according to the rotation matrix and the translation vector.

4. The method of claim 1, wherein the process of mapping vision semantic information to a laser radar coordinate system to obtain semantic enhanced data comprises: performing target detection according to image data to obtain semantic labels and their pixel positions in the image; converting the pixel positions to three-dimensional coordinates according to a depth map and a camera intrinsic matrix; projecting the three-dimensional coordinates to the laser radar coordinate system according to the coordinate conversion relationship to generate semantic enhanced data.

5. The method of claim 1, wherein the process of updating a pose estimation of a robot under a particle filtering framework according to the semantic enhanced data comprises: identifying a dynamic obstacle area according to the semantic enhanced data; eliminating laser point cloud data corresponding to the dynamic obstacle area from the observation data to obtain static observation data; updating particle weights and pose estimation in the particle filtering according to the static observation data.

6. The method of claim 1, wherein the process of constructing a composite map containing semantic information and geometric information according to the updated pose estimation comprises: According to the static observation data, a geometric grid layer is constructed; According to the semantic enhancement data, a semantic grid layer is constructed; the semantic grid layer includes semantic labels and confidence levels; The geometric grid layer and the semantic grid layer are spatially aligned and superimposed to generate a composite map.

7. The method of claim 1, wherein, According to the composite map, the process of performing a navigation task includes: According to the semantic labels in the composite map, passable areas and prohibited areas are set; According to the passable areas, a global path is planned; During the movement of the robot, if a dynamic obstacle is detected, the local path is updated according to the composite map.

8. The method of claim 4, wherein, The target detection uses a Mask-RCNN target detection model; The output of the Mask-RCNN target detection model includes the category, mask, and bounding box of the target.

9. The method of claim 1, wherein, The resolution of the composite map is 5cm x 5cm, and each grid cell stores an occupancy probability, a semantic label, and a confidence level.

Images