Obstacle detection method and device, intelligent robot and storage medium

Abstract

Embodiments of the present application provide an obstacle detection method and device, an intelligent robot and a storage medium, and belong to the technical field of image processing. The method is applied to an intelligent robot and comprises: obtaining a target disparity map, the target disparity map being obtained based on a road environment image collected by the intelligent robot; determining each obstacle information in the target disparity map according to road information in the target disparity map and a pixel value of each pixel point; and obtaining classification information of each obstacle according to the road environment image, the obstacle information and an anchor box classification model. The present application can reduce the amount of data required in the process of detecting obstacles, reduce the complexity of the calculation process, and improve the efficiency of obstacle detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an obstacle detection method and device, an intelligent robot and a storage medium. BACKGROUND

[0002] With the rapid development of science and technology, various intelligent robots have been applied in people's daily life, and people can use intelligent robots to realize meal delivery, game, transportation and other functions.

[0003] In actual application, the intelligent robot needs to identify the surrounding environment during movement to ensure that it does not collide with other objects during movement. At present, the intelligent robot mainly detects obstacles based on the fusion of vision and laser radar, for example, the intelligent robot can match the obstacle target to the camera pixel coordinate system based on the conversion relationship between the radar coordinate system and the camera pixel coordinate system. On this basis, the clustering algorithm is used to cluster the data and compare it with the geometric features of the obstacle to detect and classify the obstacle.

[0004] In the above process, the intelligent robot needs to calculate the convolution features of each obstacle multiple times, and the amount of data to be processed is large, which has the problem of low efficiency of obstacle detection. SUMMARY

[0005] The embodiments of the present application provide an obstacle detection method, which reduces the data amount and complexity of the detection process of obstacles and improves the efficiency of obstacle detection.

[0006] In one aspect, the embodiments of the present application provide an obstacle detection method applied to an intelligent robot, which comprises:

[0007] Obtaining a target disparity map, wherein the target disparity map is obtained based on a road environment image collected by the intelligent robot;

[0008] Determining obstacle information in the target disparity map according to road information in the target disparity map and pixel values of each pixel point;

[0009] Obtaining classification information of each obstacle according to the road environment image, the obstacle information and an anchor box classification model.

[0010] As an optional implementation manner of the embodiments of the present application, the road information is a straight line function of the position of the road in the target disparity map, and the determination of the obstacle information in the target disparity map according to the road information in the target disparity map and the pixel values of each pixel point comprises:

[0011] obtaining a linear function of a position where the road is located according to the target disparity map;

[0012] obtaining a function output value corresponding to each pixel point by bringing a pixel value of each pixel point into the linear function;

[0013] determining each obstacle information in the target disparity map according to the function output value corresponding to each pixel point and a preset road value.

[0014] As an optional implementation of the embodiment of the present application, the method further comprises:

[0015] determining each pixel point with a function output value greater than the preset road value as an obstacle pixel point;

[0016] determining each obstacle information in the target disparity map according to the each obstacle pixel point.

[0017] As an optional implementation of the embodiment of the present application, the method further comprises:

[0018] determining a target anchor box parameter of the anchor box classification model according to the road environment image and the each obstacle information;

[0019] adjusting an anchor box parameter of the anchor box classification model to the target anchor box parameter;

[0020] inputting the road environment image and the each obstacle information into the anchor box classification model to obtain the classification information of the each obstacle.

[0021] As an optional implementation of the embodiment of the present application, the method further comprises:

[0022] extracting each target object information contained in the road environment image according to the road environment image;

[0023] obtaining each target object information belonging to an obstacle according to the each target object information and the each obstacle information;

[0024] determining the target anchor box parameter of the anchor box classification model according to the each target object information belonging to the obstacle.

[0025] As an optional implementation of the embodiment of the present application, the method further comprises:

[0026] obtain depth information corresponding to each target object information belonging to the obstacle;

[0027] determine a moving speed and an evading manner of the intelligent robot according to the depth information;

[0028] control the intelligent robot to move according to the moving speed and the evading manner.

[0029] As an optional implementation of the embodiment of the present application, the method further comprises:

[0030] display a first operation interface, wherein the first operation interface comprises an image selection control and a function test control;

[0031] in response to a triggering operation on the image selection control, determine that the selected target image is the road environment image;

[0032] in response to a triggering operation on the function test control, execute a test function corresponding to the function test control.

[0033] In another aspect, the embodiment of the present application provides an obstacle detection device applied to an intelligent robot, and the device comprises:

[0034] a first acquisition module, configured to acquire a target disparity map, wherein the target disparity map is obtained based on a road environment image collected by the intelligent robot;

[0035] a first determination module, configured to determine each obstacle information in the target disparity map according to road information in the target disparity map and a pixel value of each pixel point;

[0036] a second acquisition module, configured to acquire classification information of each obstacle according to the road environment image, the each obstacle information and an anchor box classification model.

[0037] In another aspect, the present application provides an intelligent robot, which comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to realize the obstacle detection method as described in one aspect.

[0038] In another aspect, the present application provides a computer readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to realize the obstacle detection method as described in one aspect.

[0039] In another aspect, an embodiment of the present application provides a computer program product, which, when running on a computer, causes the computer to perform the obstacle detection method according to any one of the above aspects.

[0040] In another aspect, an embodiment of the present application provides an application publishing platform for publishing a computer program product, which, when running on a computer, causes the computer to perform the obstacle detection method according to any one of the above aspects.

[0041] The technical scheme provided by the embodiments of the present application can have at least the following beneficial effects:

[0042] The present application obtains a target disparity map, which is obtained by an intelligent robot through collecting a road environment image; determines each obstacle information in the target disparity map according to road information in the target disparity map and a pixel value of each pixel point; and obtains classification information of each obstacle according to the road environment image, the each obstacle information and an anchor box classification model. The present application obtains a target disparity map, determines each obstacle information in the target disparity map based on road information in the target disparity map and a pixel value of each pixel point, and does not need to use radar data to perform multi-region repeated calculation, combines the road environment image, the each obstacle information and the anchor box classification model, and obtains the classification information of each obstacle by setting an anchor box. This process reduces the amount of data required in the process of detecting obstacles, reduces the complexity of the calculation process, and improves the efficiency of obstacle detection. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical scheme in the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0044] Figure 1 is a scene architecture diagram using an intelligent robot according to an example embodiment of the present application;

[0045] Figure 2 is a method flowchart of an obstacle detection method according to an example embodiment of the present application;

[0046] Figure 3 is a method flowchart of an obstacle detection method according to an example embodiment of the present application;

[0047] Figure 4 is a structure block diagram of an obstacle detection device according to an example embodiment of the present application;

[0048] Figure 5 is a structural schematic diagram of an intelligent robot provided by an example embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the schemes of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0050] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein; obviously, the embodiments in the description are only some of the embodiments of the present application, not all the embodiments.

[0051] The terms "first" and "second" and the like in the specification and claims of the present application are used to distinguish different objects, not to describe a specific order of the objects. For example, the first camera and the second camera are used to distinguish different cameras, not to describe a specific order of the cameras.

[0052] In the embodiments of the present application, the words "exemplary" or "for example" mean serving as an example, instance, or illustration, any embodiment or aspect described in the embodiments of the present application as "exemplary" or "for example" should not be construed as being more preferred or advantageous than other embodiments or aspects. Rather, the use of the words "exemplary" or "for example" is intended to present concepts in a particular manner. In addition, in the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0053] The scheme provided by the present application can be applied to the application scenario of detecting obstacles on the driving path in the intelligent robot used in people's daily life. In order to facilitate understanding, the application scenario related by the embodiments of the present application will be briefly introduced first.

[0054] With the continuous progress of science and technology, in recent years, the field of robots has developed rapidly, and various technologies have emerged. From the movement of four-legged robot dogs, to Alpha Dog defeating Ke Jie, to unmanned road testing, various new inventions and achievements show us that intelligent robots will bring earth-shattering changes to society. In the future, intelligent robots will lead the intelligent upgrading of people's work, helping people complete work more efficiently, accurately, and cost-effectively. And with the development of artificial intelligence and cloud computing, the cost of robots is constantly decreasing. In the foreseeable future, a large number of mobile robots will appear in the fields of front desk greeting, cleaning and rehabilitation. China's service robots started late, but developed rapidly and had huge demand, which is worth paying attention to. Mobile robots are usually applied to non-deterministic environments such as hotels, homes, and hospitals, and need to perceive the surrounding environment through sensors such as laser radars and depth cameras. They also need to interact with people through voice to complete tasks.

[0055] For the above various intelligent robots that are increasingly used in daily life, not only can technology meet people's daily needs, but also needs to ensure that intelligent robots are safe during movement. This process requires intelligent robots to detect obstacles on their travel paths to avoid collisions with other objects.

[0056] Please refer to Figure 1 , which shows a scene architecture diagram using an intelligent robot related to an example embodiment of the present application. As Figure 1 shown, it includes an intelligent robot 101, a terminal device 102, and a server 103.

[0057] Optionally, the intelligent robot 101 can include a camera assembly, which can include a camera that can take pictures, a high-speed camera, a laser radar, etc. For example, when the intelligent robot uses a binocular camera for shooting, the camera assembly can be composed of a binocular camera.

[0058] Optionally, the terminal device 102 can be a terminal device with communication function, for example, the terminal device can include but is not limited to wearable devices (such as bracelets, smart watches, smart glasses, etc.), mobile phones, tablet computers, notebook computers, smart glasses, smart watches, MP3 players (Moving Picture Experts Group Audio Layer III, dynamic image expert compression standard audio layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, dynamic image expert compression standard audio layer 4) player, desktop computer, laptop computer, smart home device, etc. Terminal devices with database storage function.

[0059] Optionally, the server 103 can be at least one of a server, a plurality of servers, a cloud computing platform, and a virtualization center. The server 103 is configured to provide background services for applications supporting a virtual environment. Optionally, the server 103 can undertake a major computing work, and the data acquisition device 101 and the terminal device 102 undertake a minor computing work; or the server 103 undertakes a minor computing work, and the data acquisition device 101 and the terminal device 102 undertake a major computing work; or the server 103 and the data acquisition device 101 and the terminal device 102 adopt a distributed computing architecture to perform collaborative computing.

[0060] Optionally, the intelligent robot 101 and the terminal device 102 can establish wireless communication connections with each other or with the server through a network providing device, for example, the network providing device can be a WiFi device in a home environment, an Access Point (AP) device, or the like, or the network providing device can also be a base station.

[0061] Optionally, the intelligent robot 101 and the server 103 can transmit the three-dimensional point cloud data collected by the intelligent robot 101 to each other after establishing a wireless communication connection with the network providing device. The terminal device 102 and the server 103 can also transmit image, video and other data through the communication connection. The wireless communication connection can also be referred to as a communication network or a network connection, and the communication connection uses standard communication technology and / or protocol. The network is usually the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network. In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML) and the like are used to represent the data exchanged through the network. In addition, all or some links can be encrypted using conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) and the like. In other embodiments, custom and / or dedicated data communication technologies can be used instead of or in addition to the above data communication technologies.

[0062] Optionally, at present, if the intelligent robot 101 is applied on the road, the detection of lane lines and obstacles is one of the most important functions that the intelligent robot 101 needs to have. In the process of obstacle detection of the intelligent robot 101, obstacle recognition needs to be performed through existing visual detection and recognition technology. Among them, the commonly used visual detection and recognition technology mainly includes cluster-based category recognition. For example, the intelligent robot can match the obstacle target to the camera pixel coordinate system based on the conversion relationship between the radar coordinate system and the camera pixel coordinate system when the radar detects the obstacle result, and on this basis, the data is clustered and compared with the geometric features of the obstacle by using a clustering algorithm, the obstacle is detected and classified, and the obstacle detection process is completed. In the above process, the intelligent robot needs to calculate the convolution features of each obstacle multiple times, and the amount of data to be processed is large, and there is a problem of low efficiency of obstacle detection.

[0063] In order to reduce the data amount and complexity of the obstacle detection process and improve the efficiency of obstacle detection, the application provides an obstacle detection method, which can determine the obstacle information in the target disparity map through the road information of the target disparity map and the pixel value of each pixel point, thereby reducing the data amount required in the obstacle detection process.

[0064] Please refer to Figure 2 , which shows a method flowchart of an obstacle detection method provided by an example embodiment of the application. The obstacle detection method can be executed by an intelligent robot, which can be the intelligent robot in the scenario shown in Figure 1 . As shown in Figure 2 , the obstacle detection method can include the following steps:

[0065] Step 201: Obtain a target disparity map, which is obtained based on the road environment image collected by the intelligent robot.

[0066] Optionally, in the present application, the intelligent robot can autonomously collect the environment on the driving road, obtain the road environment image, and obtain the target disparity map based on the collected road environment image. For example, the intelligent robot collects through a binocular camera. For the collected road environment image pair, the target disparity map of the road environment image pair can be based on any one of the images. For example, the target disparity map can be the V-disparity map obtained by the intelligent robot, taking the road environment image on the left in the road environment image pair as the reference.

[0067] Step 202: Determine the obstacle information in the target disparity map according to the road information in the target disparity map and the pixel value of each pixel point.

[0068] Optionally, the road information is the linear information of the linear relationship of the road in the target disparity map. For example, the road in the target disparity map is a straight line with a linear relationship. The application can obtain the linear function of the position of the road, and take the linear function as the road information of the road.

[0069] Optionally, the intelligent robot determines whether each pixel point is an obstacle according to the positional relationship between the pixel value of each pixel point and the road information, thereby determining the obstacle information in the target disparity map. Optionally, each obstacle information can be the coordinate information of each obstacle. For example, the intelligent robot defines all the parts above the road information as obstacles.

[0070] Step 203: Obtain the classification information of each obstacle according to the road environment image, the obstacle information, and the anchor box classification model.

[0071] The anchor box classification model is used for classifying each obstacle in the input road environment image. Optionally, the anchor box classification model can be pre-set in the intelligent robot by a developer through machine learning model training.

[0072] Optionally, the intelligent robot can determine parameters required to be input into the anchor box classification model according to each obstacle information and the road environment image, input the parameters required to be input into the anchor box classification model, and enable the anchor box classification model to classify each obstacle in the road environment image.

[0073] In summary, the application obtains a target disparity map, which is obtained by the intelligent robot through collection of a road environment image; determines each obstacle information in the target disparity map according to road information in the target disparity map and a pixel value of each pixel point; and obtains classification information of each obstacle according to the road environment image, each obstacle information and an anchor box classification model, wherein the anchor box classification model is used for classifying each obstacle in the input road environment image. The application obtains the target disparity map, determines each obstacle information in the target disparity map according to road information in the target disparity map and a pixel value of each pixel point, does not need to use radar data to perform multi-region repeated calculation, combines the road environment image, each obstacle information and the anchor box classification model, obtains the classification information of each obstacle by setting the anchor box, and this process reduces the amount of data required in the detection of obstacles, reduces the complexity of the calculation process, and improves the efficiency of obstacle detection.

[0074] In a possible implementation, the road information is a straight line function of a position of a road in the target disparity map, the application obtains the straight line function of the position of the road, determines each obstacle information through the straight line function and a pixel value of each pixel point, and the amount of data used in this process is less, achieving the effect of quickly detecting obstacles.

[0075] Please refer to Figure 3 which shows a method flowchart of an obstacle detection method provided by an example embodiment of the application. The obstacle detection method can be executed by an intelligent robot. The intelligent robot can be the intelligent robot in the scenario shown in Figure 1 . As shown in Figure 3 , the obstacle detection method can include the following steps.

[0076] Step 301: Obtain a target disparity map, which is obtained by an intelligent robot through collection of a road environment image.

[0077] Optionally, taking the example of a smart robot comprising a binocular camera, the smart robot can collect road environment images through the binocular camera during driving, to obtain a pair of road environment images, including a road environment image collected by a left-eye camera and a road environment image collected by a right-eye camera. The smart robot obtains a target disparity map based on the obtained pair of road environment images. For example, taking the road environment image collected by the right-eye camera in the pair of road environment images as a reference, the target disparity map can be a V-disparity map obtained by the smart robot.

[0078] Optionally, in general, the smart robot collects video data based on the binocular camera, and the smart robot can decompose the collected video data according to each video frame, and perform preprocessing operations on each road environment image. Optionally, the image preprocessing operations mainly include image graying, inverse perspective transformation, and filter smoothing, which can effectively reduce the amount of processing data. The above target disparity map can be obtained based on the preprocessed road environment images.

[0079] Optionally, the smart robot can also identify each adjacent video frame after decomposition. If the image information of a certain frame in the video data is not significantly different from that of the previous and subsequent frames, the smart robot can use the environmental information in the previous frame after preprocessing the first frame, without reprocessing and obtaining the target disparity map, and directly obtains the target disparity map corresponding to the first frame.

[0080] Step 302, determining each obstacle information in the target disparity map according to the road information in the target disparity map and the pixel value of each pixel point.

[0081] In one possible implementation, the road information is a linear function of the position of the road in the target disparity map, and the smart robot can determine the obstacle information as follows: obtaining a linear function of the position of the road in the target disparity map; inputting the pixel value of each pixel point into the linear function to obtain a function output value corresponding to each pixel point; and determining each obstacle information in the target disparity map according to the function output value corresponding to each pixel point and a preset road value.

[0082] For example, for the V-disparity map obtained above, the road (also referred to as the road surface) is a straight line with a linear relationship in the V-disparity map, and each pixel point included in the road in the V-disparity map has a coordinate and a coordinate system. The function of the straight line is obtained, and the straight line is defined as y road = k road *x + b road . Wherein, y road represents the value on the road, k roada slope of a straight line at a position of the road, b road is a constant term of the straight line at the position of the road.

[0083] Optionally, the intelligent robot can define all the parts above the road information as obstacles. For example, the intelligent robot can determine the manner of each obstacle information in the target disparity map according to the function output value corresponding to each pixel point and the preset road value as follows: each pixel point corresponding to the function output value greater than the preset road value is determined as an obstacle pixel point; and each obstacle information in the target disparity map is determined according to each obstacle pixel point. That is, the intelligent robot traverses all the pixel points in the disparity map in turn according to the screening process to screen all the obstacle information.

[0084] For example, the preset road value is y road For example, f represents that the intelligent robot brings the pixel value of each pixel point into the straight line function to obtain the function output value corresponding to each pixel point, and each f is compared with y road If the function output value f corresponding to a certain pixel point is greater than y road , the position of the pixel point is determined as belonging to the position of the obstacle. Optionally, in order to further enhance the reliability of the judgment, considering the unevenness of the road surface, the intelligent robot can record the value of such area as M, and if f>y road +M, it is proved that the pixel value at this position is higher than the pixel value of the road surface, and the pixel point at this position is determined as the position of the obstacle. For example, the intelligent robot can set the above preset road value as y road +M.

[0085] Optionally, in the present scheme, the intelligent robot further comprises a gravity acceleration device and a gyroscope device, and the intelligent robot can obtain the movement data of the gravity acceleration device and the gyroscope device, determine the flatness of the road where the intelligent robot is located through the movement data, and obtain the preset road value according to the flatness. For example, the flatness of the road and the preset road value have a corresponding relationship, which can be stored in the intelligent robot in advance. The intelligent robot obtains the flatness of the road and queries the corresponding relationship to obtain the preset road value corresponding to the flatness. The flatness of the road can be divided into 1-10 levels, each level corresponds to a different preset road value, the higher the level, the flatter the road, and the smaller the preset road value.

[0086] In step 303, the target anchor frame parameter of the anchor frame classification model is determined according to the road environment image and each obstacle information.

[0087] In a possible implementation, the intelligent robot can determine the target anchor box parameter of the anchor box classification model according to the road environment image and the obstacle information as follows: extracting target object information contained in the road environment image according to the road environment image; obtaining target object information belonging to the obstacle according to the target object information and the obstacle information; and determining the target anchor box parameter of the anchor box classification model according to the target object information belonging to the obstacle.

[0088] Optionally, the intelligent robot can extract the target object information contained in the road environment image according to the road environment image as follows: the intelligent robot extracts the target object information contained in the road environment image by performing feature information segmentation and extraction on the road environment image. Optionally, the feature information can be object contour, size, and shape, and the extracted feature information is obtained as the target object information. For example, if two vehicles are contained in a road environment image, the contours, sizes, and shapes of the two vehicles in the road environment image can be obtained by segmentation and extraction, so that the target object information of the two vehicles contained in the road environment image is obtained.

[0089] Optionally, the intelligent robot can obtain the target object information belonging to the obstacle according to the target object information and the obstacle information as follows: the intelligent robot determines the contour and coordinates of each obstacle according to the obstacle information, determines the contour and coordinates of each target object according to the target object information, calculates the coincidence degree between each two, fuses each obstacle according to the coincidence degree, and determines the target object information belonging to the obstacle. For example, if the coincidence degree of two objects is higher than a preset coincidence degree threshold, the union of the respective regions of the target object information and the obstacle information is obtained, and the merged region is taken as the obstacle region corresponding to the target object information, that is, the target object is the obstacle in the merged region.

[0090] Optionally, the target anchor box parameter includes the number of anchor boxes, the size of the anchor box, and the shape of the anchor box. The intelligent robot determines the target anchor box parameter of the anchor box classification model according to the determined target object information belonging to the obstacle. If the number of obstacles is 5 and the target object information belonging to the obstacle is 4, which is determined by combining the target object information contained in the road environment image, the number of anchor boxes used by the anchor box classification model is 4, the shape of the anchor box used by the anchor box classification model is the same as the contour, and the area of the anchor box used by the anchor box classification model is the same as the area enclosed by the contour.

[0091] Optionally, the anchor box classification model is pre-trained by a developer. For example, the scheme can use YOLOV5 classification tool to train an existing data set, summarize and induce its own model to realize target category recognition, and finally set it in the intelligent robot as an anchor box classification model.

[0092] In step 304, the anchor box parameters of the anchor box classification model are adjusted to the target anchor box parameters.

[0093] Optionally, the intelligent robot sets the anchor box parameters of the anchor box classification model as the target anchor box parameters obtained above, and classifies the road environment image to obtain the classification information of each obstacle.

[0094] In step 305, the road environment image and the information of each obstacle are input into the anchor box classification model to obtain the classification information of each obstacle.

[0095] Optionally, in this application, the road environment image and the information of each obstacle are input into the anchor box classification model, and the output of the anchor box classification model is the classification information of each obstacle. For example, a road environment image contains 3 pedestrians and 4 vehicles, and the information of each obstacle obtained by the V-disparity map is 7 obstacle information. The intelligent robot inputs the road environment image and the 7 obstacle information into the anchor box classification model, and the anchor box classification model outputs the classification information of the 7 obstacles, which are categories such as cars and pedestrians.

[0096] In one possible implementation, the intelligent robot can also obtain the depth information corresponding to each target object information belonging to the obstacle; determine the moving speed and avoidance mode of the intelligent robot according to the depth information; and control the intelligent robot to move according to the moving speed and avoidance mode. For example, the intelligent robot obtains the depth information corresponding to each target object information belonging to the obstacle; matches the depth information and the disparity map to determine the depth distance between each obstacle and itself, thereby calculating the moving speed and avoidance mode required by the intelligent robot at present, and controlling the intelligent robot to move according to the calculated moving speed and avoidance mode.

[0097] For example, a global path planner is established for the intelligent robot by a preset SLAM algorithm. After the global path planner plans a global path, in order to make the intelligent robot walk along the planned global path as much as possible while avoiding colliding with suddenly appearing obstacles in the route, a local path planner plans a local path according to the scanning information of the laser radar, and then continuously sends a speed to the chassis controller according to the local path to complete navigation.

[0098] In a possible implementation, the intelligent robot can further display a first operation interface, the first operation interface including an image selection control and a function test control. In response to a triggering operation on the image selection control, the intelligent robot determines that a selected target image is a road environment image. In response to a triggering operation on the function test control, the intelligent robot executes a test function corresponding to the function test control. That is, the intelligent robot can also provide the first operation interface to enable a user to select a certain image as a road environment image and perform obstacle detection on the image. Optionally, the function test control can correspond to different test functions, such as path planning, target detection, autonomous movement, environment perception, target tracking, and the like, which are not described herein again.

[0099] In summary, the present application obtains a target disparity map, which is obtained by an intelligent robot through collection of a road environment image. According to road information in the target disparity map and a pixel value of each pixel point, the present application determines each obstacle information in the target disparity map. According to the road environment image, each obstacle information, and an anchor box classification model, the present application obtains classification information of each obstacle, where the anchor box classification model is used to classify each obstacle in an input road environment image. The present application obtains the target disparity map, determines each obstacle information in the target disparity map based on the road information in the target disparity map and the pixel value of each pixel point, and does not need to perform multi-region repeated calculation by using radar data. In combination with the road environment image, each obstacle information, and the anchor box classification model, the present application obtains the classification information of each obstacle by setting an anchor box. This process reduces the amount of data required in the process of detecting obstacles, reduces the complexity of the calculation process, and improves the efficiency of obstacle detection.

[0100] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0101] Please refer to Figure 4 , which shows a structural block diagram of an obstacle detection apparatus provided by an example embodiment of the present application. The obstacle detection apparatus 400 can be used in an intelligent robot, which can be the intelligent robot in the above Figure 1 . The obstacle detection apparatus 400 can be used to execute all or part of the steps in the method provided by the embodiments shown in Figure 2 or Figure 3 . The obstacle detection apparatus 400 includes:

[0102] A first obtaining module 401 is configured to obtain a target disparity map, which is obtained by an intelligent robot through collection of a road environment image.

[0103] The first determining module 402 is configured to determine each obstacle information in the target disparity map according to road information in the target disparity map and pixel values of each pixel point.

[0104] The second obtaining module 403 is configured to obtain classification information of each obstacle according to the road environment image, the each obstacle information and an anchor box classification model, where the anchor box classification model is used to classify each obstacle in the input road environment image.

[0105] In summary, the target disparity map is obtained based on the intelligent robot by collecting the road environment image, each obstacle information in the target disparity map is determined according to road information in the target disparity map and pixel values of each pixel point, and the classification information of each obstacle is obtained according to the road environment image, the each obstacle information and the anchor box classification model, where the anchor box classification model is used to classify each obstacle in the input road environment image. The target disparity map is obtained, each obstacle information in the target disparity map is determined based on road information in the target disparity map and pixel values of each pixel point, and the classification information of each obstacle is obtained by setting the anchor box in combination with the road environment image, the each obstacle information and the anchor box classification model. This process reduces the amount of data required in the detection process of the obstacle, reduces the complexity of the calculation process, and improves the efficiency of the obstacle detection.

[0106] Optionally, the road information is a linear function of a position of the road in the target disparity map, and the first determining module includes a first obtaining unit, a second obtaining unit and a first determining unit.

[0107] The first obtaining unit is configured to obtain the linear function of the position of the road in the target disparity map.

[0108] The second obtaining unit is configured to input the pixel value of each pixel point into the linear function to obtain a function output value corresponding to each pixel point.

[0109] The first determining unit is configured to determine each obstacle information in the target disparity map according to the function output value corresponding to each pixel point and a preset road value.

[0110] Optionally, the first determining unit includes a first determining subunit and a second determining subunit.

[0111] The first determining subunit is configured to determine each pixel point greater than the preset road value in the function output value corresponding to each pixel point as an obstacle pixel point.

[0112] The second determining sub-unit is configured to determine each obstacle information in the target disparity map according to the respective obstacle pixel points.

[0113] Optionally, the second acquisition module 403 comprises a second determining unit, a first adjusting unit and a third acquisition unit.

[0114] The second determining unit is configured to determine target anchor frame parameters of the anchor frame classification model according to the road environment image and the respective obstacle information.

[0115] The first adjusting unit is configured to adjust the anchor frame parameters of the anchor frame classification model to the target anchor frame parameters.

[0116] The third acquisition unit is configured to input the road environment image and the respective obstacle information into the anchor frame classification model to acquire classification information of the respective obstacles.

[0117] Optionally, the second determining unit comprises a first extracting sub-unit, a first acquisition sub-unit and a third determining sub-unit.

[0118] The first extracting sub-unit is configured to extract respective target object information contained in the road environment image according to the road environment image.

[0119] The first acquisition sub-unit is configured to acquire respective target object information belonging to obstacles according to the respective target object information and the respective obstacle information.

[0120] The third determining sub-unit is configured to determine target anchor frame parameters of the anchor frame classification model according to the respective target object information belonging to obstacles.

[0121] Optionally, the device further comprises:

[0122] A third acquisition module is configured to acquire depth information corresponding to respective target object information belonging to obstacles.

[0123] A second determining module is configured to determine a moving speed and an evading manner of the intelligent robot according to the depth information.

[0124] A first control module is configured to control the intelligent robot to move according to the moving speed and the evading manner.

[0125] Optionally, the device further comprises:

[0126] A first display module is configured to display a first operation interface, wherein the first operation interface contains an image selection control and a function test control.

[0127] The third determining module is configured to determine the selected target image as the road environment image in response to a triggering operation on the image selection control.

[0128] The first executing module is configured to execute a test function corresponding to the function test control in response to a triggering operation on the function test control.

[0129] Please refer to Figure 5 which shows a structural schematic diagram of an intelligent robot according to an example embodiment of the present application. As shown in Figure 5 , the intelligent robot includes a processor 510, a transceiver 520, and a display unit 570. The display unit 570 can include a display screen.

[0130] Optionally, the intelligent robot can further include a memory 530. The processor 510, the transceiver 520, and the memory 530 can communicate with each other through internal connection paths to transfer ranging data. The memory 530 is configured to store a computer program, and the processor 510 is configured to call and run the computer program from the memory 530.

[0131] The processor 510 can be combined with the memory 530 into one processing device, or more commonly, be independent components. The processor 510 is configured to execute program codes stored in the memory 530 to implement the above functions. In a specific implementation, the memory 530 can be integrated in the processor 510, or be independent of the processor 510.

[0132] It can be understood that Figure 5 The intelligent robot shown in the figure can include one or more processing units, for example: the processor 510 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or can be integrated in one or more processors.

[0133] The processor 510 can also include a memory for storing instructions and data. In some embodiments, the memory in the processor 510 is a cache memory. The memory can hold instructions or data that the processor 510 has just used or is using repeatedly. If the processor 510 needs to use the instructions or data again, it can call them directly from the memory. This avoids repeated access and reduces the waiting time of the processor 510, thus improving the efficiency of the system.

[0134] In some embodiments, the processor 510 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I-C) interface, an inter-integrated circuit sound (I-S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0135] The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is usually used to connect the processor 510 and the transceiver 520. For example, the processor 510 communicates with the Bluetooth module in the transceiver 520 through the UART interface to realize the Bluetooth function.

[0136] The MIPI interface can be used to connect the processor 510 and peripheral devices such as the display unit 570. The MIPI interface includes a camera serial interface (CSI), a display screen serial interface (DSI), etc. In some embodiments, the processor 510 and the display unit 570 communicate through the DSI interface to realize the display function of the intelligent robot.

[0137] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 510 to the display unit 570, the transceiver 520, etc. The GPIO interface can also be configured as an I-C interface, an I-S interface, a UART interface, an MIPI interface, etc.

[0138] The transceiver 520 can provide a wireless communication solution applied to the intelligent robot, including a wireless local area network (WLAN) (such as a wireless fidelity (Wi-Fi) network), Bluetooth (BT), a global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The transceiver 520 can be one or more devices integrated with at least one communication processing module, for example, can include a Bluetooth module.

[0139] The memory 530 can be used to store computer executable program codes including instructions. The memory 530 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required by a function (such as a sound playing function, an image playing function, etc.), etc. The data storage area can store data created during use of the intelligent robot (such as positioning data), etc. In addition, the memory 530 can include a high-speed random access memory, and can also include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 510 executes various function applications and data processing of the intelligent robot by running instructions stored in the memory 530 and / or instructions stored in a memory disposed in the processor.

[0140] In addition, in order to make the functions of the intelligent robot more perfect, the intelligent robot can further include one or more of a power supply 550, an input unit 550, an audio circuit 580, and a sensor 502, etc.

[0141] The power supply 550 is used to provide power supply for various devices or circuits in the intelligent robot. Preferably, the power supply 550 can be logically connected to the processor 510 through a power management device, so as to realize functions such as management of charging, discharging, and power consumption management through the power management device.

[0142] The input unit 550 can be used to receive input digital or character information, and to generate key signal input with respect to user settings of the intelligent robot and function control. Specifically, the input unit 550 can include a touch panel and other input devices. The touch panel, also called a touch screen, can collect a user's touch operation on or near the touch panel, such as a user's operation on or near the touch panel using a finger, a stylus, or any suitable object or accessory, and drive a corresponding connection device according to a pre-set program. Optionally, the touch panel can include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the touch position of the user and detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch coordinates, and then sends it to the processor 510, and can also receive the command from the processor 510 and execute it. In addition, the touch panel can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel, the input unit 550 can also include other input devices. Specifically, the other input devices can include one or more of a function key, a trackball, an operation lever, etc.

[0143] The display unit 570 can be used to display information input by the user or information provided to the user, as well as various menus of the intelligent robot. The display unit 570 can include a display panel, which can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel can cover the display panel, and when the touch panel detects a touch operation on or near it, it transmits to the processor 510 to determine the type of touch event, and then the processor 510 provides corresponding visual output on the display panel according to the type of touch event.

[0144] The intelligent robot can further include at least one sensor 502, such as a gyroscope sensor, a motion sensor, and other sensors. Specifically, the gyroscope sensor can be used to determine the motion posture of the intelligent robot. In some embodiments, the angular velocity of the intelligent robot around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor. The gyroscope sensor can also be used for navigation, motion sensing game scenarios. As one of the motion sensors, the acceleration sensor can detect the magnitude of acceleration in each direction (i.e., x, y, and z axes), the magnitude and direction of gravity when at rest, and can be used for applications such as identifying the posture of the intelligent robot (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometers, tapping), and the like. As for other sensors that can also be configured for the intelligent robot, such as pressure gauges, barometers, hygrometers, thermometers, infrared sensors, and the like, they will not be described here.

[0145] The audio circuit 580 can include a speaker and a microphone, providing an audio interface between the user and the intelligent robot. The audio circuit 580 can convert received audio data into an electrical signal, transmit the electrical signal to the speaker, and convert the electrical signal into a sound signal output by the speaker. On the other hand, the microphone collects a sound signal and converts it into an electrical signal, which is received by the audio circuit 580 and converted into audio data. The audio data is output to the processor 510 for processing, and then transmitted to another intelligent robot via the RF circuit, or output to the memory 530 for further processing.

[0146] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the terminal device. In other embodiments of the present application, the terminal device can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0147] It should be understood that in the embodiments of the present application, the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0148] The embodiments of the present application also provide a computer readable medium storing at least one instruction, which is loaded and executed by the processor to implement all or part of the steps performed by the intelligent robot in the obstacle detection method according to the various embodiments described above.

[0149] The embodiments of the present application further provide a computer program product, which stores at least one instruction, the at least one instruction is loaded and executed by the processor to implement the obstacle detection method according to the various embodiments above, all or part of the steps performed by the intelligent robot.

[0150] It should be noted that the above-described embodiments provide the device when performing the control of the electronic equipment, only the above-mentioned each functional module is divided and exemplified, and in actual application, the above-mentioned functions can be distributed by different functional modules to be completed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided by the above-described embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be described here.

[0151] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0152] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program to instruct related hardware to complete, the program can be stored in a computer readable storage medium, the above-mentioned storage medium can be read only memory, magnetic disk or optical disk, etc.

[0153] The above describes the obstacle detection method, device, intelligent robot and storage medium disclosed by the embodiments of the present application, and the principle and implementation mode of the present application are described by examples. The above embodiment description is only configured to help understand the method and core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as the limitation of the present application.

Claims

1. An obstacle detection method, characterized in that, Applied to intelligent robots, the method includes: A target disparity map is obtained, which is based on images of the road environment acquired by the intelligent robot. Based on the road information and the pixel value of each pixel in the target disparity map, determine the information of each obstacle in the target disparity map; Based on the road environment image, the information of each obstacle, and the anchor frame classification model, the classification information of each obstacle is obtained; The road information is a linear function of the road's location in the target disparity map. The step of determining the obstacle information in the target disparity map based on the road information and the pixel value of each pixel includes: Based on the target disparity map, obtain the straight line function of the road location; Substitute the pixel value of each pixel into the line function to obtain the function output value corresponding to each pixel; Based on the function output value corresponding to each pixel and the preset road value, the obstacle information in the target disparity map is determined. The preset road value is obtained based on the flatness of the road where the intelligent robot is located. The flatness is determined by acquiring the movement data of the intelligent robot's gravity acceleration device and gyroscope device.

2. The method according to claim 1, characterized in that, The step of determining the obstacle information in the target disparity map based on the function output value corresponding to each pixel and the preset road value includes: Each pixel whose function output value is greater than the preset road value is identified as an obstacle pixel. Based on the individual obstacle pixels, the information of each obstacle in the target disparity map is determined.

3. The method according to claim 1, characterized in that, The step of obtaining classification information for each obstacle based on the road environment image, the information of each obstacle, and the anchor box classification model includes: Based on the road environment image and the information of each obstacle, the target anchor frame parameters of the anchor frame classification model are determined; Adjust the anchor frame parameters of the anchor frame classification model to the target anchor frame parameters; The road environment image and the information of each obstacle are input into the anchor frame classification model to obtain the classification information of each obstacle.

4. The method according to claim 3, characterized in that, The step of determining the target anchor box parameters of the anchor box classification model based on the road environment image and the information of each obstacle includes: Based on the road environment image, extract information about each target object contained in the road environment image; Based on the information of each target object and the information of each obstacle, obtain the information of each target object belonging to the obstacle; Based on the information of each target object belonging to the obstacle, the target anchor frame parameters of the anchor frame classification model are determined.

5. The method according to claim 4, characterized in that, The method further includes: Obtain depth information corresponding to each target object belonging to the obstacle; Based on the depth information, the moving speed and avoidance method of the intelligent robot are determined; The intelligent robot is controlled to move according to the stated moving speed and the stated evasion method.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The first operation interface is displayed, which includes an image selection control and a function test control. In response to a trigger operation on the image selection control, the selected target image is determined to be the road environment image; In response to a trigger operation on the functional test control, the test function corresponding to the functional test control is executed.

7. An obstacle detection device, characterized in that, The device, used in intelligent robots, includes: The first acquisition module is used to acquire a target disparity map, which is obtained by the intelligent robot by collecting road environment images. The first determining module is used to determine the obstacle information in the target disparity map based on the road information and the pixel value of each pixel in the target disparity map; The second acquisition module is used to acquire the classification information of each obstacle based on the road environment image, the information of each obstacle, and the anchor frame classification model; The road information is a linear function of the road location in the target disparity map, and the first determining module is specifically used for: Based on the target disparity map, obtain the straight line function of the road location; Substitute the pixel value of each pixel into the line function to obtain the function output value corresponding to each pixel; Based on the function output value corresponding to each pixel and the preset road value, the obstacle information in the target disparity map is determined. The preset road value is obtained based on the flatness of the road where the intelligent robot is located. The flatness is determined by acquiring the movement data of the intelligent robot's gravity acceleration device and gyroscope device.

8. An intelligent robot, characterized in that, The intelligent robot includes a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set, the at least one instruction, the at least one program, the code set, or instruction set being loaded and executed by the processor to implement the obstacle detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or instruction set is loaded and executed by a processor to implement the obstacle detection method as described in any one of claims 1 to 6.

Images