Mobile robot navigation method and system based on deep learning

Abstract

The invention discloses a mobile robot navigation method and system based on deep learning, and the method mainly comprises the steps: S1, collecting a ground image through a vehicle-mounted visual camera of a mobile robot, S2, carrying out the marking of geometric lines and semantic tags of an image containing a ground segmentation line through a processing module, carrying out the training of a deep learning model, and carrying out the navigation of the mobile robot. S3, shooting a ground image in the driving process of the mobile robot, identifying a ground segmentation line in the image by a deep learning model, and outputting a line shape and a semantic tag, S4, converting the identified segmentation line of the line shape into a point cloud, fusing the point cloud with a laser radar point cloud, and matching with a preset map of a map module to position the mobile robot. S5, converting the identified line-shaped segmentation line into a point cloud to pick and place goods, and S6, inputting the identified segmentation line with a semantic tag into a navigation module and a control module, and matching the segmentation line with preset semantic information to carry out mobile robot navigation and control. The existing parting lines are utilized to identify shapes and semantics, navigation positioning is improved, control is accurate, and the method is economical and efficient.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous mobile robot (AMR) navigation, and particularly to a mobile robot navigation method and system based on deep learning. Background Art

[0002] Currently, the positioning technology of robots mainly relies on the following several solutions, but all have significant limitations: (1) Positioning based on preset identifiers: Special identifiers such as two-dimensional codes, RFID tags or reflective tapes are deployed in the environment, and positioning is achieved by identifying these preset marks through cameras or laser sensors. It is necessary to lay special identifiers in advance, with high maintenance costs, and the positions of the identifiers are fixed, making it difficult to adapt to the dynamic adjustment of the factory layout (such as production line changes), and the flexibility is poor; (2) Positioning based on natural textures: Feature matching of natural textures on the ground (such as cement cracks, tile joints) is used to achieve positioning. Natural textures are easily affected by light and wear, with poor feature stability, and cannot distinguish different functional areas, without semantic information; (3) SLAM-based positioning: An environmental map is constructed and real-time positioning is performed through lidar or visual SLAM. It relies on stereo features and is prone to failure in open or low-texture environments. In addition, the computational complexity is high, and 3D point clouds need to be processed in real time, requiring high hardware computing power; (4) Traditional ground line recognition: The Hough transform or edge detection algorithm is used to identify ground lines, but usually it needs to rely on high-contrast and complete dividing lines, and is sensitive to line wear, light changes, and dynamic occlusion, with poor adaptability, and the function is single, only able to detect geometric shapes and unable to understand the semantic meaning of the lines; Therefore, it is necessary to develop a mobile robot navigation method and system based on deep learning to solve the above defects. Summary of the Invention

[0003] Aiming at the technical problems existing in the above-mentioned prior art, the purpose of the present application is to provide a mobile robot navigation method and system based on deep learning.

[0004] To achieve the purpose of the present invention, the present invention proposes the following technical solutions, including steps: S1: The vision camera on the mobile robot collects ground images, and the ground images contain ground dividing lines. S2: The processing module annotates the images containing ground dividing lines with geometric lines and semantic labels, and conducts deep learning model training. S3: During the driving process of the mobile robot, ground images are taken, and the deep learning model identifies the ground dividing lines in the images and outputs the line shapes and semantic labels. S4: Among them, the dividing line of the recognized line shape is converted into a point cloud, input into the navigation module, fused with the lidar point cloud, and matched with the preset map of the map module for mobile robot positioning. S5: Among them, the dividing line of the recognized line shape is converted into a point cloud, input into the docking module, the docking position is recognized, and goods are picked up and placed. S6: Among them, the recognized dividing line with semantic tags is input into the navigation module and the control module, and is matched with the preset semantic information for mobile robot navigation and control.

[0005] Further, before the S3 deep learning model recognizes the ground dividing line in the image, all or part of the following methods are selected to optimize the image according to the required image quality: S231: Perform an inverse perspective transformation on the image to convert the image into a bird's-eye view. S232: Apply geometric constraints to the image to exclude interfering lines. S233: Perform illumination normalization on the image to exclude the influence of illumination. S234: Perform frequency domain enhancement filtering on the image. S235: Virtually complete the image lines, predict the trend of the ground dividing line or complete the missing ground dividing line.

[0006] Preferably, step S3 further includes: S31: In a low illumination environment, use an on-vehicle fill light for environmental fill light or use an infrared camera to capture images.

[0007] Preferably, the preset map in step S4 includes: S41: During the process of scanning the map, the lines extracted from the images scanned by the vision camera are converted into point clouds, and fused with the lidar point cloud to form a preset map.

[0008] Preferably, step S4 further includes: S42: During the driving process of the mobile robot, the map module automatically recognizes the update of the ground dividing line and automatically updates the preset map.

[0009] Preferably, step S4 further includes: S43: After the point cloud of the line shape is fused with the lidar point cloud, it is input into the navigation module according to the weight distribution.

[0010] Preferably, step S6 further includes: S61: The semantic tags include shape, color, and pattern, and the preset semantic information includes corresponding different semantic tags, including area definition, road definition, running speed, warning method, and interaction instructions with devices.

[0011] In addition, the present invention also provides a mobile robot navigation system based on deep learning, which applies the above-mentioned mobile robot positioning method based on deep learning and ground dividing line recognition, and specifically includes: a mobile robot and a ground dividing line. The mobile robot includes a navigation module, a control module, a map module, a processing module, a docking module, a vision camera, and a lidar. The vision camera captures a ground image containing the ground dividing line, the lidar scans environmental point cloud data, the map module generates a preset map and automatically updates the map according to the vision camera and lidar point cloud data, the processing module performs image annotation, deep learning model training, recognizes the ground dividing line, and outputs the line shape and semantic label, the docking module recognizes the docking position and goods, and adjusts the pose of the mobile robot, the navigation module performs mobile robot positioning and path planning, and the control module controls the speed, gives warnings, and conducts device interaction.

[0012] Compared with the prior art, the present invention uses the above-mentioned mobile robot navigation method and system based on deep learning, and has the following main advantages or beneficial effects: (1) Utilize existing ground dividing lines: directly recognize non-preset markings such as paint lines and tape lines already existing in the factory, and incorporate them into the autonomous mobile robot positioning system for integrated positioning. Especially in open and highly dynamically changing environments, the positioning accuracy is improved, and it is applied to realize the positioning of the picking and placing positions and docking for picking and placing goods, with low deployment costs; (2) Deep learning enhances robustness: Solve problems such as low line contrast, breaks, and dynamic occlusion through a semantic segmentation model; (3) Semantic positioning: Combine semantic information such as line color and shape (such as a yellow line indicating a no-go zone) to achieve more intelligent path navigation planning. Description of the Drawings

[0013] Figure 1 The figure shows a schematic flow diagram of the mobile robot navigation method of the present application; Figure 2 The figure shows a schematic diagram of the mobile robot navigation system of the present application; Figure 3 The figure shows a schematic diagram of the mobile robot vision camera recognizing the ground dividing line of the present application.

[0014] In the figure, 1 - mobile robot, 11 - vision camera, 12 - lidar, 13 - navigation module, 14 - control module, 15 - map module, 16 - processing module, 17 - docking module, 2 - ground dividing line, 21 - dividing line of the line shape, 22 - dividing line with semantic label, 23 - docking position. Detailed Embodiments

[0015] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0016] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, components, or modules, assemblies, and / or combinations thereof.

[0017] It should be noted that the terms "including" and "having" in the specification and claims of the present application and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or devices.

[0018] It should be understood that the solution of the present invention can be implemented by a single or a combination of multiple hardware, software, or other devices. In the description of the following embodiments, the methods and steps of the present invention can be implemented by being stored in a storage device including but not limited to a hard disk, a removable storage device, a magnetic disk, an optical disk, etc.

[0019] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0020] Embodiment As Figure 1 、 Figure 2 、 Figure 3 shown, a mobile robot navigation method based on deep learning, the specific steps include: S1: The visual camera 11 on the vehicle of the mobile robot 1 can be a monocular, binocular or depth vision camera. During the movement of the mobile robot 1, it collects ground images and selects the ground images containing the ground dividing line 2. The ground dividing line 2 is generally the existing dividing line in the factory, such as a paint line, a tape line, etc. used to divide different areas and for identification.

[0021] S2: The processing module 16 annotates the geometric lines and semantic labels of the ground segmentation line images collected at different angles and under different lighting conditions, and conducts deep learning model training. In this embodiment, the YOLO deep learning model is used as an example, which has the characteristics of light weight and fast inference speed. In this technical field, other deep learning models can also be used for training and recognition, such as the MobileNetV3+UNet hybrid model, etc., with the same technical effects.

[0022] S3: During the driving process of the mobile robot 1, the ground image is captured, and the deep learning model identifies the ground segmentation line 2 in the image and outputs the line shape and semantic label. The specific steps are as follows: (1) Image preprocessing: Before training the deep learning model on the image, select all or part of the following methods to optimize the image according to the required image quality: S231: Perform inverse perspective transformation on the image to convert the image into a bird's-eye view. Assume that the coordinates of a certain point ( x , y ) in the bird's-eye view are ( x ′, y ′), then the inverse perspective transformation formula is: , Formula 1 Among them, H is a 3×3 homogeneous transformation matrix, which is usually calculated through the intersection coordinates of parallel lines in the known image in the bird's-eye view.

[0023] S232: Apply geometric constraints to the image to exclude interfering lines. Assume that the gradient direction of a certain point ( x , y ) in the image is θ ( x , y ), then the geometric constraint formula is: , Formula 2 Among them, θ min and θ max are thresholds set according to the direction range of the segmentation line in the actual scene.

[0024] S233: Perform dynamic illumination normalization processing on the image to exclude the influence of illumination through the combination of local statistics and global learning: , Formula 3 Among them, I final ( x , y ) is the normalized brightness, I ( x , y) is the brightness of a certain pixel point in the image, μ local ( x , y ) is the mean value of the local area, σ local ( x , y ) is the standard deviation of the local area; μ global The mean value obtained by global learning, σ global The standard deviation obtained by global learning, α is the weighting coefficient, which is used to balance the effects of local and global normalization, and its value range is between 0 and 1.

[0025] S234: Perform frequency-domain enhancement filtering on the image, establish an analytical relationship between the line width and the filtering parameters. Assume that the frequency-domain representation of the image is F ( u , v ), then the frequency-domain enhancement filtering formula is: , Formula Four Among them, F enhanced ( u , v ) is the enhanced frequency-domain representation, F ( u , v ) is the frequency-domain representation of the image, H ( u , v ) is the filtering function, and its analytical relationship with the line width w is: , Formula Five Among them, n is the filtering parameter, which controls the filtering intensity.

[0026] S235: Virtually complete the image lines, predict the ground segmentation line trend or complete the missing ground segmentation line. Taking the LSTM time-series prediction model as an example, assume that the input sequence of LSTM is x t , and the hidden state is h t , then the update formula of LSTM is: , Formula Six Among them: i t : The output of the input gate, f t : The output of the forget gate, o t : The output of the output gate, c t : The cell state, h t : The hidden state, x t: Input at the current time step, h t-1 : Hidden state at the previous time step, c t-1 : Cell state at the previous time step, W xi , W xf , W xc , W xo : Weight matrix of the input, W hi , W hf , W hc , W ho : Weight matrix of the hidden state, b i , b f , b c , b o : Bias vector, σ: Sigmoid activation function, tanh: Hyperbolic tangent activation function, ⊙: Element-wise product.

[0027] It should be noted that in order to improve the recognition accuracy, the present invention uses an LSTM model to predict and complete the lines. In a specific embodiment, due to the limited processing capacity of the controller of the mobile robot itself, when the recognition accuracy does not need to be very high, the LSTM model can also not be used to predict and complete the lines, and the existing image recognition ground dividing lines can be directly used, and the point cloud of the image conversion and the laser point cloud are fused only when the ground dividing lines can be recognized.

[0028] S31: Further, in a low-illumination environment, a vehicle-mounted fill light is used for environmental fill light or an infrared camera is used to capture images to adapt to the illumination changes in different factories and improve the recognition accuracy and stability.

[0029] After preprocessing the above image, the YOLOv11 model is used to train and recognize the image (according to the requirements of specific embodiments, other models can also be used), specifically as follows: (II) Model training (1) The total loss function is: , Formula VII Among them, λ IoU , λ cls , λ obj is a weight parameter used to balance the importance of different tasks.

[0030] Among them: Localization loss: , Among them, IoU: Intersection over Union of the predicted box and the ground truth box, measuring the degree of overlap, ρ 2 (b, b gt ): Square of the Euclidean distance between the center points of the predicted box and the ground truth box, c: Diagonal length of the smallest closed region covering the two boxes, v: Aspect ratio consistency parameter, is the weight coefficient; Classification loss: , where y c : one-hot encoding of the true label, p c : class probability predicted by the model; Confidence loss: , where : true confidence (0 or 1), o: predicted confidence score, λ obj : balance coefficient, usually taken as 1.

[0031] (III) Model Inference Assume the input image is I, then the output of model inference is: , Equation (8) where f is the forward propagation function of the YOLOv11 model, y pred is the predicted position of the segmentation line and the class.

[0032] (IV) Optimization in Mobile Robot Deployment: Assume the state of the robot during movement is x = x , y , θ T , then the state update formula is: , Equation (9) where x t : robot state vector at time step t , usually including position and orientation θ t , for example x t = x t , y t , θ t T ; u t control input vector at time step t , usually including linear velocity v t and angular velocity ω t , for example u t = v t , ω t T ; w t ​​​is the process noise, representing the random disturbances or uncertainties in the system; f(·): the state transition function, which describes the evolution of the robot's state over time.

[0033] S4: Among them, the segmentation line 21 of the recognized line shape is converted into a point cloud and input into the navigation module 13, where it is fused with the lidar point cloud and matched with the preset map of the map module 15 for mobile robot positioning. Taking the extended Kalman filter EKF line fusion of the shape point cloud and the lidar point cloud as an example, in a specific embodiment, algorithms such as fusion based on weighted average, fusion based on probability hypothesis, fusion based on feature extraction, multi-modal fusion, fusion based on deep learning, fusion based on multi-sensor data association, and fusion based on Bayesian filtering can also be used for fusion, which is selected according to the actual usage scenario and has the same technical effect: (1) Assume that the segmentation line point cloud is P line , and the lidar point cloud is P lidar , then the fusion formula is: , Formula Ten Among them, K is the Kalman gain, which is a weight matrix used to adjust the difference between the predicted value and the observed value, and H is the observation matrix, which represents the linear transformation matrix that converts the state vector into the observation space. HP line is the projection of the predicted state in the observation space.

[0034] Among them, the extended Kalman filter fusion also includes the following steps such as state prediction and update: A. State prediction: Prediction based on the system dynamic model, which is used to estimate the state change of the system without observation data: , Among them: : The predicted state at time step t t, x t-1 : The estimated state at time step t- t-1, u t : The control input at time step t t, f(·): the non-linear state transition function.

[0035] B. Prediction covariance: The uncertainty (covariance matrix) of the predicted state: , Among them, : The predicted covariance matrix at time step t, F t : The Jacobian matrix, which represents the linear approximation of the state transition function, Q t : The process noise covariance matrix.

[0036] C. Kalman Gain Calculation: Used to determine the weights of the observed data and predicted data during state update: , where K t : The Kalman gain at time step t, H t The observation matrix at time step t, R t : The observation noise covariance matrix.

[0037] D. State Update: Update the state estimate at the current moment: , where x t : The updated state at time step t , z t : The observation value at time step t , h(·): The non - linear observation function.

[0038] E. Covariance Update: Quantify the accuracy of the state estimate: , where P t : The updated covariance matrix at time step t , I: The identity matrix.

[0039] (2) Localization Result Fusion (Dividing Line + Laser SLAM): Assume the dividing line localization result is x line , and the laser SLAM localization result is x slam , then the fusion formula is: , Formula Eleven where is the weight coefficient, which is adjusted according to actual needs.

[0040] It should be noted that in the specific embodiment, the fused data also includes gyroscopes, wheel speed odometers, etc. The weights are adjusted according to actual applications to further improve the localization accuracy. In this embodiment, taking the fusion of the ground dividing lines recognized by lasers and cameras as an example for illustration is more convenient for those skilled in the art to understand. After fusing the point cloud of the line shape and the laser point cloud in this embodiment, the localization accuracy can be effectively improved, and by using the existing ground dividing lines in the factory, it is easy to deploy and has a low cost.

[0041] Furthermore, the process of generating and updating the preset map includes: S41: During the process of scanning the map, convert the lines extracted from the images scanned by the vision camera 11 into point clouds, and fuse them with the lidar point clouds to form a preset map, S42: During the driving of the mobile robot 1, the map module 15 automatically identifies the update of the ground dividing line and automatically updates the preset map to address the positioning instability caused by highly dynamic environments such as changes in the factory production line layout and ground renovation. S43: After the point cloud of the line shape is fused with the lidar point cloud, it is input into the navigation module according to the weight distribution.

[0042] S5: Among them, the identified dividing line 21 in the shape of a line is converted into a point cloud and input into the docking module 17 to identify the docking position 23. The mobile robot 1 adjusts its own pose according to the docking position for automatic loading and unloading. In the technical field, generally, additional auxiliary facilities such as QR codes, V-shaped / L-shaped feature plates, etc. are used to mark the docking position. In this embodiment, the existing ground dividing line in the factory is used to identify the docking position, which can effectively reduce the deployment difficulty and application cost.

[0043] S6: Among them, the identified dividing line 22 with semantic tags is input into the navigation module 13 and the control module 14, and is matched with the preset semantic information for the navigation and control of the mobile robot 1. The semantic tags include shape, color, and pattern. The preset semantic information includes corresponding different semantic tags, including area definition, road definition, running speed, warning method, interaction instructions with equipment, etc. When the mobile robot 1 recognizes the semantic tags, it executes the corresponding instructions. For example, after reaching a certain area, the mobile robot 1 sounds a warning sound, warns through the color of the light, reduces the running speed, etc., and after recognizing the crosswalk line in the factory, it interacts with the traffic warning light to warn pedestrians to pay attention to avoidance. By combining the line with specific semantics, not only the existing ground dividing line 2 is utilized, but also specific markings can be added in some areas, enabling more intelligent applications of the mobile robot implemented by the method of the present invention.

[0044] In addition, the present invention also provides a mobile robot navigation system based on deep learning, which is applied in the above-mentioned mobile robot positioning method based on deep learning, and specifically includes: a mobile robot 1 and a ground dividing line 2. The ground dividing line 2 is an existing non-preset dividing line in the factory, including paint lines, tapes, etc. The mobile robot 1 includes a navigation module 13, a control module 14, a map module 15, a processing module 16, a docking module 17, a vision camera 11, and a lidar 12. The vision camera 11 captures a ground image containing the ground dividing line 2, and the lidar 12 scans environmental point cloud data. The map module 15 generates a preset map and automatically updates the map according to the point cloud data of the vision camera 11 and the lidar 12. The processing module 16 performs image annotation, deep learning model training, identifies the ground dividing line 2, and outputs the line shape and semantic label. The docking module 17 identifies the docking position and goods, and adjusts the pose of the mobile robot 1 to realize the picking and placing of goods. The navigation module 13 performs the positioning and path planning of the mobile robot 1, and the control module 14 controls the speed, warning, and execution of device interaction instructions, etc.

[0045] The present invention can effectively improve the deficiencies of the existing technology, significantly improve the accuracy of mobile robot positioning and the operation efficiency in a high-dynamic environment, and has great popularization value.

[0046] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A mobile robot navigation method based on deep learning, characterized in that: The steps include: S1: The visual camera on the mobile robot collects ground images. The original ground image contains the ground segmentation line. S2: The processing module annotates the image containing the ground segmentation line with geometric lines and semantic labels, and performs deep learning model training. S3: The mobile robot takes ground images while driving, and the deep learning model recognizes the ground segmentation lines in the image and outputs the line shape and semantic labels. S4: wherein the segmentation line of the recognized line shape is converted into a point cloud, input into the navigation module, merged with the laser radar point cloud, and matched with the preset map of the map module to locate the mobile robot. S5: The segmentation line of the identified line shape is converted into a point cloud, which is input into the docking module to identify the docking position and perform pick-up and delivery. S6: The identified segmentation line with the semantic label is input into the navigation module and the control module, and matched with the preset semantic information to perform navigation and control of the mobile robot.

2. The deep learning-based mobile robot navigation method according to claim 1, characterized in that: Before the deep learning model training of the image in step S2 and the deep learning model recognition of the ground segmentation line in the image in step S3, all or part of the following methods are selected to optimize the image according to the required image quality: S231: Perform inverse perspective transformation on the image to convert the image into a bird's-eye view. S232: Perform geometric constraints on the image to eliminate interference lines. S233: Perform illumination normalization processing on the image to eliminate the influence of illumination. S234: Perform frequency domain enhancement filtering on the image. S235: Virtually complete the image lines, predict the direction of the ground segmentation line or complete the missing ground segmentation line.

3. The deep learning-based mobile robot navigation method according to claim 1, characterized in that: Step S3 further comprises: S31: In low-light environments, use the vehicle-mounted fill light for ambient light or use an infrared camera to capture images.

4. The deep learning-based mobile robot navigation method according to claim 1, characterized in that: The preset map of step S4 includes: S41: During the map scanning process, the line shape of the image scanned by the visual camera is extracted and converted into a point cloud, which is then fused with the laser radar point cloud to form a preset map.

5. The deep learning-based mobile robot navigation method according to claim 1, characterized in that: Step S4 further comprises: S42: During the driving process of the mobile robot, the map module automatically recognizes the update of the ground dividing line and automatically updates the preset map.

6. The deep learning-based mobile robot navigation method according to claim 1, characterized in that: Step S4 further comprises: S43: After the line-shaped point cloud is fused with the lidar point cloud, it is input into the navigation module according to the weight distribution.

7. The deep learning-based mobile robot navigation method according to claim 1, characterized in that: Step S6 further comprises: S61: The semantic label includes shape, color, and pattern, and the preset semantic information includes corresponding different semantic labels, including area definition, road definition, running speed, warning method, and instructions for interacting with the device.

8. A mobile robot navigation system based on deep learning, characterized in that: A mobile robot navigation method based on deep learning according to any one of claims 1 to 7 is applied, comprising a mobile robot and a ground dividing line, wherein the mobile robot comprises a navigation module, a control module, a map module, a processing module, a docking module, a visual camera, and a laser radar. The visual camera captures the ground image including the ground dividing line, the laser radar scans the environmental point cloud data, the map module generates a preset map and automatically updates the map according to the visual camera and the laser radar point cloud data, the processing module performs image annotation, deep learning model training, identifies the ground dividing line, outputs the line shape and semantic labels, the docking module identifies the docking position and cargo, and adjusts the posture of the mobile robot, the navigation module performs mobile robot positioning and path planning, and the control module controls speed, warnings, and device interaction.

Images