Ground obstacle map marking method, mobile robot and storage medium

Abstract

The present invention discloses a ground obstacle map marking method, a mobile robot and a storage medium, and relates to the field of mobile robot technology. The technical key points of the method include reading a color image from a depth camera; detecting and identifying obstacles in the color image using a recognition model trained by deep learning, and generating an obstacle boundary box based on the identified obstacles; mapping the obstacle boundary box of the color image in the depth camera to the depth image, and obtaining the detection boundary box in the depth image; calculating the depth distance between the obstacle and the robot based on the boundary box in the depth image; obtaining the position coordinates of the robot in a three-dimensional coordinate system, and calculating the position coordinates of the obstacle in the map coordinate system; the robot communicates with a client, and sends the position coordinates of the obstacle in the map coordinate system to the client. The present invention has the advantage of reducing the workload of cleaners in subsequent obstacle searches.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile robots, and more specifically, to a ground obstacle map marking method, a mobile robot and a storage medium. Background Art

[0002] At present, the application scope of mobile robots is getting wider and wider. In addition to being used in home environments, mobile robots are also used to clean complex environments with a wide variety of garbage, such as shopping malls and basements. In complex environments, there will be obstacles that mobile robots cannot clean, such as mineral water bottles, cans or other smaller obstacles.

[0003] In the face of such obstacles, existing mobile robots cannot clean them and generally choose to bypass them. Therefore, in order to keep the basement or shopping mall clean, cleaners need to be hired to perform secondary cleaning and pick up obstacles. In the existing method, if the working range of the mobile robot is large, cleaners need to search one by one, which is a lot of work. Summary of the invention

[0004] In view of the existing technical problems, the purpose of the present invention is to provide a ground obstacle map marking method, which has the advantage of reducing the workload of cleaners in subsequent obstacle searches.

[0005] To achieve the above object, the present invention provides the following technical solution: a ground obstacle map marking method, comprising:

[0006] Read color images from the depth camera;

[0007] Use the recognition model trained by deep learning to detect and identify obstacles in the color image, and generate obstacle bounding boxes based on the identified obstacles;

[0008] Map the obstacle bounding box of the color image in the depth camera to the depth image to obtain the detection bounding box in the depth image;

[0009] Calculate the depth distance between the obstacle and the robot based on the bounding box in the depth image;

[0010] Obtain the position coordinates of the robot in the three-dimensional coordinate system, and obtain the position coordinates of the obstacle in the three-dimensional coordinate system according to the depth distance calculation, transform the three-dimensional coordinate system of the robot into the map coordinate system, and calculate the position coordinates of the obstacle in the map coordinate system;

[0011] The robot communication client sends the position coordinates of the obstacle in the map coordinate system to the client.

[0012] By adopting the above technical solution, according to the prior art, it is known that the depth camera can take two types of images, the first is a color image and the second is a depth image. Through the deep learning training model, the bounding box of the obstacle in the color image can be identified and detected in real time. According to the obtained obstacle bounding box and the obstacle depth distance information obtained by the depth camera and the robot's own position, the position coordinates of the obstacle in the map can be obtained in real time through coordinate transformation. The depth camera automatically detects and identifies ground obstacles, and marks the obstacle posture information on the map and sends it to the client, helping cleaners to quickly locate obstacles and improve search efficiency.

[0013] The present invention is further configured as follows: the depth camera is arranged below the front side of the mobile robot.

[0014] By adopting the above technical solution, since the mobile robot moves forward, the depth camera at the lower front side can continuously acquire obstacles in the moving direction of the mobile robot during the movement of the mobile robot, which can avoid obstacles and achieve a good ranging effect.

[0015] The present invention is further configured as follows: the recognition model includes a data set, and the data set includes a 70% training set, a 15% validation set and a 15% test set.

[0016] By adopting the above technical solution, the deep learning of the recognition model is based on the allocation of training sets, validation sets and test sets in the data set to ensure that the data information obtained by deep learning is stable and accurate.

[0017] The present invention is further configured to: compare the color image using the recognition model, and when the confidence of the color image exceeds a preset confidence value, the color image is recognized as having an obstacle.

[0018] By adopting the above technical solution, although there are commonalities between obstacles, there are also certain differences. Therefore, in the actual application of the recognition model, there is no need to determine whether the object in the color image is completely consistent with the recognition model. Therefore, setting the confidence level can improve the accuracy of obstacle recognition.

[0019] The present invention is further configured such that: the preset reliability value is at least 0.5.

[0020] By adopting the above technical solution, a confidence level of 0.5 can achieve a good recognition effect. Only when the similarity between the color image and the recognition model exceeds 0.5 can a good obstacle recognition accuracy be achieved.

[0021] The present invention is further configured as follows: calculating the depth distance between the obstacle and the robot comprises:

[0022] Obtain the depth information of each point within the bounding box based on the depth image calculation;

[0023] Combine the depth information of each point to get the point cloud;

[0024] Sort the point cloud by the size of the depth information, and select the point cloud with the depth information distance in the middle as the detection point cloud of the detection distance;

[0025] Obtain the depth distance of obstacles based on detection point cloud computing.

[0026] By adopting the above technical solution, the point clouds are sorted by distance size, and some point clouds in the middle are selected as the point clouds for detecting objects. The background distance information in the boundary box can be removed, so that more accurate distance information of the obstacle in the front and bottom depth camera coordinate system can be obtained without selecting the boundary of the obstacle.

[0027] The present invention is further configured as follows: the detection point cloud selects the point cloud whose depth information distance is in the middle 40-60%.

[0028] By adopting the above technical solution, 40-60% of the depth information can be used to more comprehensively remove the background distance information in the bounding box, and a certain amount of data can be guaranteed to ensure accurate calculation of the depth information.

[0029] The present invention is further configured as follows: after the robot communication client sends the position coordinates of the obstacle in the map coordinate system to the client, it also includes: the client sends the position coordinates of the obstacle in the map coordinate system to the mobile terminal via SMS.

[0030] By adopting the above technical solution, the mobile terminal can be a device with communication function such as a mobile phone. Therefore, during the cleaning process, the cleaner does not need to independently query the location of the obstacle, and can obtain the location coordinate information of the obstacle through the text message information on the mobile terminal.

[0031] The second object of the present invention is to provide a mobile robot having the characteristic of high effective utilization rate of ultrasonic probe frequency.

[0032] The second object of the present invention is achieved through the following technical solutions:

[0033] A mobile robot comprises a memory and a processor, wherein the memory stores a computer program which can be loaded by the processor and execute the above-mentioned ground obstacle map marking method.

[0034] The third object of the present invention is to provide a computer storage medium capable of storing corresponding programs and having the characteristic of facilitating high effective utilization of the ultrasonic probe frequency.

[0035] The third object of the present invention is achieved through the following technical solutions:

[0036] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any of the above-mentioned ground obstacle map marking methods.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) Obstacle location information is sent to cleaners via text messages. Cleaners do not need to patrol around looking for obstacles. They can view the location coordinates of all obstacles through the client, which improves cleaning efficiency.

[0039] (2) The mobile robot automatically marks the coordinates of obstacles on the map during operation, without the need for manual drawing;

[0040] (3) The method of first obtaining the value of the bounding box in the point cloud computing is adopted to ensure the efficiency and accuracy of calculating the depth distance. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the process of ground obstacle map marking method. DETAILED DESCRIPTION

[0042] The present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0043] An embodiment of the present invention provides a ground obstacle map marking method, which is mainly applied to a mobile robot equipped with a depth camera.

[0044] The biggest difference between a depth camera (also called a 3D camera) and an ordinary color camera is that the depth camera can detect the distance information of the shooting space. The pictures taken by ordinary color cameras can see and record all objects within the camera's field of view, but the recorded data does not include the distance of these objects from the camera. It can only judge which objects are farther and which are closer through semantic analysis of the image, and there is no exact data.

[0045] The depth camera can accurately know the distance of each point in the image from the camera. Therefore, in addition to obtaining color images, the images taken by the depth camera can also obtain depth images of the distance of each point from the camera.

[0046] A ground obstacle map marking method comprises the following steps:

[0047] S1. Read the color image from the depth camera;

[0048] The depth camera can be started by shooting at intervals of a specified duration, or by shooting when the mobile robot detects an obstacle in the moving direction. Among them, the method of shooting when the mobile robot detects an obstacle in the moving direction can be more energy-efficient and is preferred. The method of detecting obstacles in the moving direction by the mobile robot can be similar to that of existing mobile robots, using ultrasonic sensors and infrared sensors, so it will not be described in detail.

[0049] In order to ensure that the depth camera can stably capture image information of obstacles, the depth camera is installed at the bottom of the front side of the mobile robot to ensure that during the movement of the mobile robot, the depth camera at the bottom of the front side can capture images of obstacles in the direction of movement of the mobile robot to determine whether the obstacles are obstacles. After the depth camera shoots, two images, a color image and a depth image, are generated. The color image is first read from the depth camera to determine whether there are obstacles in the image.

[0050] S2. Detect and identify obstacles in the color image using a recognition model trained by deep learning, and generate an obstacle bounding box based on the identified obstacles;

[0051] The deep learning training of the recognition model uses the neural network training included in the data set. The data set includes 70% training set, 15% validation set and 15% test set. The training set: As the name implies, it refers to the sample set used for training, which is mainly used to train the parameters in the neural network. Validation set: From the literal meaning, it is a sample set used to verify the performance of the model. After different neural networks are trained on the training set, the performance of each model is compared and judged through the validation set. The different models here mainly refer to neural networks corresponding to different hyperparameters, and can also refer to neural networks with completely different structures. Test set: For the trained neural network, the test set is used to objectively evaluate the performance of the neural network.

[0052] The above recognition model is designed with a lightweight neural network that is suitable for multi-scale feature fusion of small obstacles, and the appropriate hyperparameters are adjusted for training to improve accuracy and model robustness. The model is used to detect and identify various obstacles, and the trained recognition model is quantized and pruned to reduce the number of model parameters and complexity.

[0053] In step S2, firstly, the color image captured by the depth camera is compared with the recognition model to determine whether there is an obstacle in the color image. When the confidence of the color image exceeds the preset confidence value, the color image is identified as having an obstacle. The preset confidence value is at least 0.5. That is, when there is an object in the color image and the confidence in the recognition model exceeds 0.5, it is determined that there is an obstacle in the color image, otherwise it does not exist.

[0054] When an obstacle is identified in the color image, a rectangular bounding box is generated at the boundary of the obstacle to reduce the amount of calculation for selecting the boundary of the obstacle.

[0055] S3, mapping the obstacle bounding box of the color image in the depth camera to the depth image, and obtaining the detection bounding box in the depth image;

[0056] The position of each point in the color image corresponds to that of each point in the depth image, so the detection bounding box of obstacles in the depth image can be quickly obtained through the mapping of the bounding box.

[0057] S4, calculating the depth distance between the obstacle and the robot based on the bounding box in the depth image;

[0058] Calculating the depth distance between the obstacle and the robot in step S4 includes the following steps:

[0059] S41, obtaining depth information of each point in the bounding box based on the depth image calculation;

[0060] S42, combining the depth information of each point to obtain a point cloud;

[0061] S43, sorting the point clouds according to the depth information size, and selecting the point cloud with the depth information distance in the middle as the detection point cloud of the detection distance;

[0062] S44. Obtain the depth distance of the obstacle based on the detection point cloud computing.

[0063] Among them, in step S43, the detection point cloud selects the point cloud with depth information distance in the middle 40-60%, which can more comprehensively remove the background distance information in the bounding box and ensure a certain amount of data to ensure accurate calculation of depth information.

[0064] In step S44, the depth distance of the obstacle is obtained based on the mean value of the depth information of the detection point cloud. Since the detection point cloud is most likely the depth information of the obstacle, the mean value calculation can ensure that the depth distance finally calculated falls within the outline of the obstacle, making the calculation result accurate.

[0065] S5. Obtain the position coordinates of the robot in the three-dimensional coordinate system, and obtain the position coordinates of the obstacle in the three-dimensional coordinate system according to the depth distance calculation, transform the three-dimensional coordinate system of the robot into the map coordinate system, and calculate the position coordinates of the obstacle in the map coordinate system;

[0066] S6, the robot communication client sends the position coordinates of the obstacle in the map coordinate system to the client;

[0067] Specifically, the robot sends the position coordinates of the obstacle in the map coordinate system to the client in the form of broadcast communication.

[0068] S7. The client sends the position coordinates of the obstacle in the map coordinate system to the mobile terminal via SMS.

[0069] The mobile terminal can be a device with communication function such as a mobile phone. Therefore, during the cleaning process, the cleaner does not need to independently query the location of the obstacle, but can obtain the location coordinate information of the obstacle through the text message information on the mobile terminal.

[0070] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or a mobile robot equipped with a computer. The computer instructions may be stored in a computer-readable removable storage device.

[0071] A person skilled in the art will appreciate that all or part of the steps for implementing the above embodiments may be accomplished by hardware or by instructing related hardware through a program. The program may be stored in a computer-readable mobile storage device. The mobile storage device mentioned above may be a USB flash drive, a TF card, an SD card, or the like.

[0072] In the present invention, in addition to avoiding obstacles, the mobile robot can also obtain image information taken by the depth camera while avoiding obstacles. The color image in the image information is compared with the recognition model to determine whether the obstacle to be avoided is an obstacle in the recognition model trained by deep learning. Obstacles are generally mineral water bottles, cans, etc. After determining that there is an obstacle in the color image, a rectangular bounding box is used in the color image.

[0073] Then, the corresponding relationship between the color image and the depth image in the depth camera is used to map the bounding box of the obstacle on the depth camera; a point cloud with depth information is formed in the obstacle bounding box, and the point cloud with depth information in the middle is selected as the detection point cloud to exclude the background. The average value of the depth information in the detection point cloud is calculated as the distance between the obstacle and the mobile robot.

[0074] Then the robot's position coordinates in the three-dimensional coordinate system are obtained, and the position coordinates of the obstacle in the map coordinate system are obtained based on the depth distance calculation. Finally, the position coordinates of the obstacle in the map coordinate system are communicated and sent to the client, and the client sends them to the mobile terminal held by the cleaner.

[0075] This enables the mobile robot to automatically mark the coordinates of the obstacle location in the client's map coordinate system during work, and send it to the cleaner's handheld mobile terminal through the client, thereby helping the cleaner to quickly locate the obstacle and improve search efficiency.

[0076] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A ground obstacle map marking method, characterized in that: include: Read color images from the depth camera; The depth camera is installed at the lower front side of the mobile robot and starts shooting at specified intervals or when an obstacle is detected in the moving direction; Use the recognition model trained by deep learning to detect and identify obstacles in the color image, and generate obstacle bounding boxes based on the identified obstacles; Map the obstacle bounding box of the color image in the depth camera to the depth image to obtain the detection bounding box in the depth image; Calculate the depth distance between the obstacle and the robot based on the detection bounding box in the depth image; The recognition model is designed to be a lightweight neural network that is suitable for multi-scale feature fusion of obstacles and small targets; The color image is compared with the recognition model, and when the confidence of the color image exceeds a preset confidence value, the color image is recognized as having an obstacle; The preset reliability value is at least 0.5; Calculating the depth distance between the obstacle and the robot includes: Obtain the depth information of each point within the bounding box based on the depth image calculation; Combine the depth information of each point to get the point cloud; Sort the point cloud by the size of the depth information, and select the point cloud with the depth information distance in the middle as the detection point cloud of the detection distance; Obtain the depth distance of obstacles based on the detection point cloud computing; Obtain the position coordinates of the robot in the three-dimensional coordinate system, and obtain the position coordinates of the obstacle in the three-dimensional coordinate system according to the depth distance calculation, transform the three-dimensional coordinate system of the robot into the map coordinate system, and calculate the position coordinates of the obstacle in the map coordinate system; The robot communication client sends the position coordinates of the obstacle in the map coordinate system to the client.

2. The ground obstacle map marking method according to claim 1, characterized in that: The recognition model includes a data set, which includes a 70% training set, a 15% validation set and a 15% test set.

3. The ground obstacle map marking method according to claim 1, characterized in that: The detection point cloud selects the point cloud with depth information distance in the middle 40-60%.

4. The ground obstacle map marking method according to claim 1, characterized in that: The robot communication client, after sending the position coordinates of the obstacle in the map coordinate system to the client, also includes: The client sends the location coordinates of the obstacle in the map coordinate system to the mobile terminal via SMS.

5. A mobile robot, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executes any one of the methods of claims 1 to 4.

6. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 4.

Images