Detection method, device, multi-legged robot and storage medium

Abstract

The present application discloses a method, device, multi-legged robot and storage medium for detecting the drivable space of a multi-legged robot. The detection method comprises: obtaining image data collected by a camera at the current moment, preprocessing the image data to generate a preprocessed image that meets the input format requirements of a preset drivable space detection model; processing the preprocessed image according to the drivable space detection model to generate the boundary information of the drivable space of the multi-legged robot, the boundary information of the drivable space of the multi-legged robot includes boundary line position information and boundary attribute information, and mapping the boundary line position information through the calibration parameters of the camera to generate the target drivable space. The detection method of the embodiment of the present application generates the boundary information of the drivable space of the multi-legged robot from the preprocessed image data through the drivable space detection model, and maps the drivable space boundary information through the calibration parameters of the camera to obtain the target drivable space, thereby ensuring the drivable safety of the multi-legged robot.

Description

Technical Field

[0001] The present application relates to the field of image recognition, and in particular to a method and a detection device for detecting a drivable space of a multi-legged robot, a multi-legged robot, and a computer-readable storage medium. Background Art

[0002] With the rapid development of multi-legged robot technology, the application of multi-legged robots is becoming more and more extensive. Multi-legged robots are gradually becoming an indispensable part of life. The most important thing for multi-legged robots is to have autonomous navigation capabilities, and they can walk to the target location completely autonomously given a target location. This involves the perception of the surrounding environment, path planning, and control. Environmental perception is crucial as an upstream module, affecting whether the downstream can plan a reasonable route, especially the avoidance of obstacles. The drivable space detection provides the multi-legged robot with information about the space where it can move freely and the information about obstacles, providing guarantee for the autonomous driving safety of the multi-legged robot.

[0003] In related technologies, the detection of drivable space is mainly based on sensors with ranging capabilities such as depth cameras, laser radars, and ultrasonic radars, which detect the spatial information of obstacles and thus obtain the drivable space of the multi-legged robot. However, such sensors are usually expensive and have limited field of view. In addition, they are not accurate in detecting small objects, suspended objects, and hollow objects. Summary of the invention

[0004] In view of this, the present application provides a method and a detection device for detecting a drivable space of a multi-legged robot, a multi-legged robot, and a non-volatile computer-readable storage medium.

[0005] The detection method of the embodiment of the present application comprises:

[0006] Get the image data collected by the camera at the current moment;

[0007] Preprocessing the image data to generate a preprocessed image that meets the input format requirements of a preset driving space detection model;

[0008] The preprocessed image is processed according to a preset driving space detection model to generate driving space boundary information of the multi-legged robot, wherein the driving space detection model is obtained by training a deep learning model through annotated images, and the driving space boundary information of the multi-legged robot includes boundary line position information and boundary attribute information, wherein the boundary line position information is the coordinate information of the boundary point on the boundary line; and the boundary attribute information is the category attribute information of the boundary point on the boundary line;

[0009] The boundary line position information is mapped and processed by using the calibration parameters of the camera to generate a target driving space.

[0010] In some embodiments, the pre-processed image is processed according to a preset driving space detection model to generate the driving space boundary information of the multi-legged robot, and further includes:

[0011] The pre-processed image is detected according to the scene detection model to determine the current scene of the multi-legged robot, and the drivable space detection model trained for the current scene is used as the preset drivable space detection model.

[0012] In some embodiments, the driving space detection model includes a feature extraction network and a task branching network, and the pre-processed image is processed according to the preset driving space detection model to generate the driving space boundary information of the multi-legged robot, including:

[0013] Processing the preprocessed image through the feature extraction network to generate a basic feature map;

[0014] The basic feature map is processed by the task branch network to obtain the boundary information of the multi-legged robot's driving space.

[0015] In some embodiments, the driving space detection model is obtained by training a deep learning model through labeled images, including:

[0016] The annotated image is composed of multiple columns of pixels, and target pixels are selected from each column of the annotated image according to a preset boundary point selection principle and annotated as boundary points. The boundary point selection principle is to search upward from the bottom edge of the annotated image to the obstacle that is touched first, and the grounding point of the obstacle or the projection point of the obstacle onto the ground is used as the boundary point;

[0017] All of the boundary points constitute a boundary line.

[0018] In some embodiments, the boundary attribute information includes stationary obstacles and / or moving obstacles, and the stationary obstacles include suspended obstacles, transparent obstacles, mirror obstacles, stairs, and doors.

[0019] In some embodiments, the mapping of the boundary position information to generate a target driving space by using the calibration parameters of the camera includes:

[0020] Determine image coordinate information of boundary information of the multi-legged robot's travel space;

[0021] The image coordinate information is mapped to a physical space according to the calibration parameters to obtain the target driving space in a physical coordinate system.

[0022] In some embodiments, preprocessing the image data to generate a preprocessed image that meets the preset driving space detection model input format requirement includes:

[0023] Calculating the brightness value of the image data;

[0024] It is determined whether the brightness value of the image data is greater than a preset threshold value. If so, the image data is preprocessed according to the input format requirement of the preset driving space detection model.

[0025] In certain embodiments, the detection method further comprises:

[0026] sending the boundary attribute information to a controller, and the controller outputting different obstacle avoidance measures according to the boundary attribute information; and / or

[0027] The boundary attribute information is sent to a mapping module, a positioning module, and a path planning module to provide semantic topology information.

[0028] The device for detecting the drivable space of a multi-legged robot according to the embodiment of the present application comprises:

[0029] An acquisition module is used to obtain image data collected by the camera at the current moment;

[0030] A generating module, used for preprocessing the image data to generate a preprocessed image that meets the input format requirements of a preset driving space detection model;

[0031] a processing module, for processing the preprocessed image according to a preset driving space detection model to generate driving space boundary information of the multi-legged robot, wherein the driving space detection model is obtained by training a deep learning model through annotated images, and the driving space boundary information of the multi-legged robot includes boundary line position information and boundary attribute information, wherein the boundary line position information is the coordinate information of the boundary points on the boundary line; and the boundary attribute information is the category attribute information of the boundary points on the boundary line; and

[0032] A mapping module is used to map the boundary line position information through the calibration parameters of the camera to generate a target driving space.

[0033] The multi-legged robot of the embodiment of the present application includes a processor and a storage unit; a computer program is stored in the storage unit, and when the computer program is executed by the processor, the processor executes the above-mentioned detection method. The detection method includes: obtaining image data collected by the camera at the current moment; preprocessing the image data to generate a preprocessed image that meets the input format requirements of the preset driving space detection model, processing the preprocessed image according to the preset driving space detection model to generate the multi-legged robot driving space boundary information, the driving space detection model is obtained by training the deep learning model through annotated images, and the multi-legged robot driving space boundary information includes boundary line position information and boundary attribute information, the boundary line position information is the coordinate information of the boundary point on the boundary line; the boundary attribute information is the category attribute information of the boundary point on the boundary line; the boundary line position information is mapped and processed by the calibration parameters of the camera to generate the target driving space.

[0034] The non-volatile computer-readable storage medium of the embodiment of the present application includes a computer program. When the computer program is executed by a processor, the processor executes the above-mentioned detection method. The detection method includes: obtaining image data collected by the camera at the current moment; preprocessing the image data to generate a preprocessed image that meets the input format requirements of the preset driving space detection model, processing the preprocessed image according to the preset driving space detection model to generate the driving space boundary information of the multi-legged robot, the driving space detection model is obtained by training the deep learning model through annotated images, and the driving space boundary information of the multi-legged robot includes boundary line position information and boundary attribute information, the boundary line position information is the coordinate information of the boundary point on the boundary line; the boundary attribute information is the category attribute information of the boundary point on the boundary line; the boundary line position information is mapped and processed by the calibration parameters of the camera to generate the target driving space.

[0035] In the detection method, device, multi-legged robot and readable storage medium of the multi-legged robot drivable space of the implementation mode of the present application, by obtaining the image data collected by the camera at the current moment, and pre-processing the image data, a pre-processed image that can be input in the format required by the preset drivable space detection model is generated, so that the drivable space detection module can process the pre-processed image, generate the multi-legged robot drivable space boundary information including the boundary line position information and the boundary attribute information, and finally, the boundary line position information is mapped and processed by the calibration parameters of the camera, and the target drivable space can be accurately obtained. In this way, the multi-legged robot can travel according to the target drivable space, which ensures the driving safety of the multi-legged robot. Compared with the sensors with ranging capabilities such as depth cameras, laser radars, ultrasonic radars, etc., which detect the spatial information of obstacles to obtain the drivable space of the multi-legged robot, the present application only needs to set up a camera to achieve it, saving costs.

[0036] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0038] Figure 1 is a schematic diagram of the hardware structure of a multi-legged robot in certain embodiments of the present application;

[0039] Figure 2 is a schematic diagram of the structure of a multi-legged robot according to certain embodiments of the present application;

[0040] Figure 3 It is a flowchart of a method for detecting a drivable space of a multi-legged robot in certain embodiments of the present application;

[0041] Figure 4 It is a schematic diagram of a module of a detection device for a drivable space of a multi-legged robot in certain embodiments of the present application;

[0042] Figure 5 It is a scene schematic diagram of a method for detecting a drivable space of a multi-legged robot in certain embodiments of the present application;

[0043] Figure 6 It is a flowchart of a method for detecting a drivable space of a multi-legged robot in certain embodiments of the present application;

[0044] Figure 7 It is a schematic diagram of a module of a detection device for a drivable space of a multi-legged robot in certain embodiments of the present application;

[0045] Figure 8-12 It is a flowchart of a method for detecting a drivable space of a multi-legged robot in certain embodiments of the present application. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0047] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent components are only used to facilitate the description of the present invention, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0048] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of the hardware structure of a multi-legged robot 100 according to one embodiment of the present invention. Figure 1 In the illustrated embodiment, the multi-legged robot 100 includes a mechanical unit 101, a communication unit 102, a sensor unit 103, an interface unit 104, a storage unit 105, a control module 110, and a power supply 111. The various components of the multi-legged robot 100 can be connected in any manner, including wired or wireless connections. Those skilled in the art will appreciate that Figure 1 The specific structure of the multi-legged robot 100 shown in the figure does not constitute a limitation of the multi-legged robot 100. The multi-legged robot 100 may include more or fewer components than shown in the figure. Some components are not essential components of the multi-legged robot 100 and can be omitted or combined as needed without changing the essence of the invention.

[0049] Combine the following Figure 1 The various components of the multi-legged robot 100 are introduced in detail:

[0050] The mechanical unit 101 is the hardware of the multi-legged robot 100. Figure 1 As shown, the mechanical unit 101 may include a driving plate 1011, a motor 1012, and a mechanical structure 1013. Figure 2 As shown, the mechanical structure 1013 may include a body 1014, extendable legs 1015, and a foot 1016. In other embodiments, the mechanical structure 1013 may also include an extendable mechanical arm (not shown), a rotatable head structure 1017, a swingable tail structure 1018, a load-bearing structure 1019, a saddle structure 1020, a camera structure 1021, etc. It should be noted that each component module of the mechanical unit 101 may be one or more, and may be arranged according to specific circumstances. For example, there may be four legs 1015, and each leg 1015 may be configured with three motors 1012, and the corresponding motors 1012 are 12.

[0051] The communication unit 102 can be used to receive and send signals, and can also communicate with the network and other devices, for example, after receiving the instruction information sent by the remote control or other multi-legged robot 100 to move in a specific direction at a specific speed according to a specific gait, the communication unit 102 is transmitted to the control module 110 for processing. The communication unit 102 includes, for example, a WiFi module, a 4G module, a 5G module, a Bluetooth module, an infrared module, etc.

[0052] The sensing unit 103 is used to obtain information data about the environment around the multi-legged robot 100 and to monitor parameter data of each component inside the multi-legged robot 100, and send it to the control module 110. The sensing unit 103 includes a variety of sensors, such as sensors for obtaining information about the surrounding environment: laser radar (for long-range object detection, distance determination and / or speed value determination), millimeter wave radar (for short-range object detection, distance determination and / or speed value determination), camera, infrared camera, global navigation satellite system (GNSS, Global Navigation Satellite System), etc. For example, sensors for monitoring each component inside the multi-legged robot 100: inertial measurement unit (IMU, Inertial Measurement Unit) (for measuring the value of velocity value, acceleration value and angular velocity value), plantar sensor (for monitoring the position of the plantar force point, plantar posture, ground contact force size and direction), temperature sensor (for detecting component temperature). As for other sensors such as load sensors, touch sensors, motor angle sensors, torque sensors, etc. that can also be configured for the multi-legged robot 100, they will not be repeated here.

[0053] The interface unit 104 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more components within the multi-legged robot 100, or may be used to output (e.g., data information, power, etc.) to an external device. The interface unit 104 may include a power port, a data port (such as a USB port), a memory card port, a port for connecting a device having an identification module, an audio input / output (I / O) port, a video I / O port, etc.

[0054] The storage unit 105 is used to store software programs and various data. The storage unit 105 may mainly include a program storage area and a data storage area, wherein the program storage area may store operating system programs, motion control programs, application programs (such as text editors), etc.; the data storage area may store data generated by the multi-legged robot 100 during use (such as various sensor data obtained by the sensor unit 103, log file data), etc. In addition, the storage unit 105 may include a high-speed random access memory, and may also include a non-volatile memory, such as a disk storage, a flash memory, or other volatile solid-state memory.

[0055] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0056] The input unit 107 can be used to receive input digital or character information. Specifically, the input unit 107 may include a touch panel 1071 and other input devices 1072. The touch panel 1071, also known as a touch screen, can collect the user's touch operation (such as the user's operation on the touch panel 1071 or near the touch panel 1071 using the palm, finger or suitable accessory), and drive the corresponding connection device according to a pre-set program. The touch panel 1071 may include two parts: a touch detection device 1073 and a touch controller 1074. Among them, the touch detection device 1073 detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller 1074; the touch controller 1074 receives the touch information from the touch detection device 1073, converts it into contact coordinates, and then sends it to the control module 110, and can receive and execute commands sent by the control module 110. In addition to the touch panel 1071, the input unit 107 may also include other input devices 1072. Specifically, other input devices 1072 may include but are not limited to one or more of a remote control operating handle, etc., which are not specifically limited here.

[0057] Furthermore, the touch panel 1071 may cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the control module 110 to determine the type of the touch event. Then, the control module 110 provides corresponding visual output on the display panel 1061 according to the type of the touch event. Figure 1 In the embodiment, the touch panel 1071 and the display panel 1061 are used as two independent components to respectively realize the input and output functions. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to realize the input and output functions, which is not limited here.

[0058] The control module 110 is the control center of the multi-legged robot 100. It uses various interfaces and lines to connect the various components of the entire multi-legged robot 100, and controls the multi-legged robot 100 as a whole by running or executing software programs stored in the storage unit 105 and calling data stored in the storage unit 105.

[0059] The power supply 111 is used to supply power to each component. The power supply 111 may include a battery and a power control board. The power control board is used to control functions such as battery charging, discharging, and power consumption management. Figure 1 In the illustrated embodiment, the power supply 111 is electrically connected to the control module 110. In other embodiments, the power supply 111 may also be electrically connected to the sensor unit 103 (such as a camera, a radar, a speaker, etc.) and the motor 1012. It should be noted that each component may be connected to a different power supply 111 or powered by the same power supply 111.

[0060] On the basis of the above embodiments, specifically, in some embodiments, a terminal device can be used to communicate with the multi-legged robot 100. When the terminal device communicates with the multi-legged robot 100, the terminal device can send instruction information to the multi-legged robot 100. The multi-legged robot 100 can receive the instruction information through the communication unit 102, and can transmit the instruction information to the control module 110 after receiving the instruction information, so that the control module 110 can process the target speed value according to the instruction information. The terminal device includes but is not limited to: a mobile phone with an image shooting function, a tablet computer, a server, a personal computer, a wearable smart device, and other electrical devices.

[0061] The instruction information can be determined according to preset conditions. In one embodiment, the multi-legged robot 100 may include a sensor unit 103, and the sensor unit 103 may generate instruction information according to the current environment where the multi-legged robot 100 is located. The control module 110 may determine whether the current speed value of the multi-legged robot 100 meets the corresponding preset conditions according to the instruction information. If satisfied, the current speed value and current gait of the multi-legged robot 100 will be maintained; if not satisfied, the target speed value and the corresponding target gait will be determined according to the corresponding preset conditions, so that the multi-legged robot 100 can be controlled to move with the target speed value and the corresponding target gait. The environmental sensor may include a temperature sensor, an air pressure sensor, a visual sensor, and a sound sensor. The instruction information may include temperature information, air pressure information, image information, and sound information. The communication method between the environmental sensor and the control module 110 may be wired communication or wireless communication. The wireless communication method includes but is not limited to: wireless network, mobile communication network (3G, 4G, 5G, etc.), Bluetooth, infrared.

[0062] See also Figure 3 The present application provides a method for detecting a drivable space of a multi-legged robot, comprising the steps of:

[0063] 01, obtain the image data collected by the camera at the current moment;

[0064] 02, preprocessing the image data to generate a preprocessed image that meets the input format requirements of the preset driving space detection model;

[0065] 03. Process the preprocessed image according to the driving space detection model to generate the driving space boundary information of the multi-legged robot. The driving space detection model is obtained by training the deep learning model through the labeled image. The driving space boundary information of the multi-legged robot includes the boundary line position information and boundary attribute information. The boundary line position information is the coordinate information of the boundary points on the boundary line; the boundary attribute information is the category attribute information of the boundary points on the boundary line;

[0066] 04. The boundary line position information is mapped through the camera calibration parameters to generate the target driving space.

[0067] See also Figure 4 The embodiment of the present application provides a detection device 10 for a drivable space of a multi-legged robot. The detection device 10 includes an acquisition module 11, a generation module 12, a processing module 13 and a mapping module 14.

[0068] Among them, 01 can be implemented by the acquisition module 11, 02 can be implemented by the generation module 12, 03 can be implemented by the processing module 13, and 04 can be implemented by the mapping module 14. In other words, the acquisition module 11 can be used to obtain the image data collected by the camera at the current moment; the generation module 12 can be used to preprocess the image data to generate a preprocessed image that meets the input format requirements of the preset driving space detection model. The processing module 13 can be used to process the preprocessed image according to the driving space detection model to generate the driving space boundary information of the multi-legged robot. The driving space detection model is obtained by training the deep learning model through annotated images. The driving space boundary information of the multi-legged robot includes boundary line position information and boundary attribute information. The boundary line position information is the coordinate information of the boundary points on the boundary line; the boundary attribute information is the category attribute information of the boundary points on the boundary line. The mapping module 14 can be used to map the boundary line position information through the calibration parameters of the camera to generate the target driving space.

[0069] Please combine Figure 1The multi-legged robot 100 provided in the present application also includes a processor (not shown in the figure), and the storage unit 105 stores a computer program. When the computer program is executed by the processor, the processor implements the above-mentioned detection method, that is, the processor is used to obtain image data collected by the camera at the current moment, pre-process the image data to generate a pre-processed image that meets the input format requirements of the preset driving space detection model, and process the pre-processed image according to the preset driving space detection model to generate the driving space boundary information of the multi-legged robot. The driving space detection model is obtained by training the deep learning model through annotated images. The driving space boundary information of the multi-legged robot includes boundary line position information and boundary attribute information. The boundary line position information is the coordinate information of the boundary points on the boundary line; the boundary attribute information is the category attribute information of the boundary points on the boundary line. The boundary line position information is mapped and processed by the calibration parameters of the camera to generate the target driving space.

[0070] In the detection method, detection device and multi-legged robot 100 of the present application, image data of the current driving space captured by the camera is acquired and pre-processed to generate a pre-processed image that can be processed by a driving space detection model trained by a deep learning model, so that the driving space detection module can process the pre-processed image to generate accurate driving space boundary information of the multi-legged robot, wherein the driving space boundary information of the multi-legged robot includes boundary line position information and boundary attribute information. Finally, the boundary line position information of the driving space boundary information of the multi-legged robot is mapped through the calibration parameters of the camera to obtain the accurate target driving space. In this way, the multi-legged robot 100 can ensure the driving safety of the multi-legged robot 100 by driving along the target driving space, thereby improving the application scenarios of the multi-legged robot 100. Furthermore, since the present application obtains the image data of the current driving space of the multi-legged robot through a camera to obtain the target driving space, compared with the related technology based on depth cameras, lidars, ultrasonic radars and other sensors with ranging capabilities, which detect the spatial information of obstacles to obtain the drivable space of the multi-legged robot, the accuracy of the target driving space is improved, the driving safety of the multi-legged robot 100 is improved, and the cost is saved.

[0071] This embodiment can be explained by taking the multi-legged robot 100 as a quadruped robot as an example, that is, the detection method and the detection device 10 are applied to but not limited to quadruped robots. The detection device can be hardware or software pre-installed on the quadruped robot, and the detection method can be executed when the quadruped robot is started and operated. For example, the detection device 10 can be a bottom-level software code segment of the quadruped robot or a part of the operating system. In this way, the quadruped robot can generate a target driving space based on the image data collected by the camera and travel along the target driving space.

[0072] In some embodiments, the detection device 10 may be a part of the multi-legged robot 100 . In other words, the multi-legged robot 100 includes the detection device 10 .

[0073] In some embodiments, the detection device 10 may be a discrete component assembled in a certain manner to have the aforementioned functions, or a chip having the aforementioned functions in the form of an integrated circuit, or a computer software code segment that enables the computer to have the aforementioned functions when running on the computer.

[0074] In some embodiments, as hardware, the detection device 10 can be independent or added as an additional peripheral component to the multi-legged robot 100. The detection device 10 can also be integrated into the multi-legged robot 100. For example, when the detection device 10 is a part of the multi-legged robot 100, the detection device 10 can be integrated into the processor.

[0075] Specifically, the multi-legged robot 100 can collect image data of the current driving space through a camera. In this embodiment, the camera can be a monocular camera. Those skilled in the relevant field can understand that the monocular camera has rich color and texture features, and has the characteristics of wide angle, high resolution, low price, etc. Therefore, the cost of the multi-legged robot 100 is reduced, and the multi-legged robot 100 collects a wide range of the current driving space through the monocular camera, thereby providing a guarantee for the multi-legged robot 100 to travel safely along the target driving space, so that the multi-legged robot 100 can adapt to a wider range of application scenarios.

[0076] It should be noted that the labeled image training refers to the process of marking the areas in the image where the multi-legged robot 100 can pass safely without collision. Among them, the images used for the labeled image training can be historical image data collected by the camera. The images may include multiple images. It can be understood that since the driving space detection model is obtained by the deep learning model through labeled image training, the more images and the more complex the scene, the higher the accuracy of the driving space detection model obtained through the labeled image training, and the more accurate the generated target driving space.

[0077] Please combine Figure 5 The boundary information of the multi-legged robot driving space refers to the boundary information of the area where the multi-legged robot 100 can drive normally and safely. The boundary attribute information includes stationary obstacles and / or moving obstacles. It should be noted that obstacles refer to objects that will hinder or collide with the normal passage of the multi-legged robot 100. Stationary obstacles refer to objects that will not move autonomously, and moving obstacles refer to objects that will move autonomously.

[0078] Stationary obstacles include but are not limited to suspended obstacles, transparent obstacles, mirror obstacles, stairs, doors and other boundaries of drivable areas. Among them, suspended obstacles refer to the boundary line here that is not the grounding line between the obstacle and the ground, and is not an objectively existing and actually visible line, but the boundary of the suspended object projected onto the ground. For example, if a table collides with the head of the multi-legged robot 100, the boundary of the table projected onto the ground is the boundary of the drivable space, and the attribute of this boundary is a suspended obstacle. Transparent obstacles refer to transparent objects, such as floor-to-ceiling windows, glass doors, etc.; mirror obstacles refer to objects with mirror reflection, such as mirrors.

[0079] Moving obstacles include but are not limited to pedestrians, animals, vehicles, other multi-legged robots 100 and other objects that can move autonomously.

[0080] Calibration parameters refer to parameters preset on the processor for calibrating the camera. The calibration parameters may include an intrinsic parameter matrix and a distortion parameter matrix. That is, the boundary line position information can be mapped and processed through the intrinsic parameter matrix and distortion parameter matrix of the camera to generate a target driving space that is accessible without obstacles.

[0081] See also Figure 6 In some embodiments, before step 03, the detection method further comprises the steps of:

[0082] 05. Detect the preprocessed image according to the scene detection model to determine the current scene of the multi-legged robot, and use the drivable space detection model trained for the current scene as the preset drivable space detection model.

[0083] Please combine Figure 7 In some embodiments, the detection device 10 further includes a selection module 15 , and step 05 can be implemented by the selection module 15 .

[0084] In other words, the selection module 15 can be used to detect the preprocessed image according to the scene detection model, determine the current scene of the multi-legged robot, and use the drivable space detection model trained for the current scene as the preset drivable space detection model.

[0085] In some embodiments, the processor can be used to detect the preprocessed image according to the scene detection model, determine the current scene of the multi-legged robot, and use the drivable space detection model trained for the current scene as the preset drivable space detection model.

[0086] Specifically, the scene detection model is a classification model based on deep learning. The scene detection model is used to detect the current driving space in the preprocessed image, so as to determine which scene the current driving space is in. The scene may include a factory, a hospital, a street community, a shopping mall, a home or other open roads. The storage unit 105 may also pre-store a plurality of driving space detection models trained by different scenes. For example, the storage unit 105 is preset with a plurality of driving space detection models trained in scenes such as a factory, a hospital, a street community, a shopping mall, a home or other open roads.

[0087] Understandably, obstacles may be different in different scenes. For example, in a shopping mall, there are more pedestrians and more mobile obstacles than static obstacles, while in a home, there are more static obstacles than mobile obstacles. Therefore, the corresponding drivable space detection model is trained according to the scene, and the drivable space detection model corresponding to the scene processes the preprocessed image in the scene to generate accurate drivable space boundary information for the multi-legged robot.

[0088] For instance, in some examples, the scene detection model detects the preprocessed image and determines that the scene of the preprocessed image is a factory scene, and the drivable space detection model trained with the image as a factory scene is used as the preset driving space detection model. For another example, in some examples, the scene detection model detects the preprocessed image and determines that the scene of the preprocessed image is a shopping mall scene, and the drivable space detection model trained with the image as a shopping mall scene is used as the preset driving space detection model.

[0089] See also Figure 8 In some embodiments, the driving space detection model includes a feature extraction network and a task branching network, and step 03 includes sub-steps:

[0090] 031, the preprocessed image is processed through a feature extraction network to generate a basic feature map;

[0091] 032, the basic feature map is processed through the task branch network to obtain the boundary information of the multi-legged robot's driving space.

[0092] In some embodiments, sub-steps 031 and 032 may be completed by the processing module 13. In other words, the processing module 13 may be used to process the pre-processed image through a feature extraction network to generate a basic feature map, and to process the basic feature map through a task branch network to obtain the multi-legged robot's driving space boundary information.

[0093] In some embodiments, the processor can be used to process the preprocessed image through a feature extraction network to generate a basic feature map, and process the basic feature map through a task branch network to obtain the boundary information of the multi-legged robot's driving space.

[0094] The feature extraction network may include but is not limited to convolutional neural networks such as Restnet, VGG, MobileNet, etc. For example, in this embodiment, the feature extraction network may be Restnet, that is, the pre-processed image is processed by Restnet to generate a basic feature map.

[0095] The methods of the task branch network include but are not limited to regression, classification, semantic segmentation, instance segmentation and other methods. The task branch network takes the basic feature map generated by the feature extraction network as input, and the task branch network includes two outputs, one output is the driving boundary line position information, and the other output is the boundary attribute information. In this embodiment, the task branch network can use the regression method to generate the driving boundary line position information according to the basic feature map, and use the classification method to generate the boundary attribute information according to the basic feature map.

[0096] See also Fig. 9 In some embodiments, step 03 further includes the sub-steps of:

[0097] 033, the image is composed of multiple columns of pixels. The target pixels in each column of the image are screened out and marked as boundary points according to the preset boundary point selection principle. The boundary point selection principle is to search from the bottom edge of the image upward to the obstacle that is touched first, and the grounding point of the obstacle or the projection point of the obstacle onto the ground is used as the boundary point;

[0098] 034, all boundary points form a boundary line.

[0099] Please further combine Figure 2 In some embodiments, sub-steps 033 and 034 can be completed by the processing module 13. In other words, the processing module 13 can be used for an image composed of multiple columns of pixels, and each column of the image pixels is screened out according to a preset boundary point selection principle to mark the target pixels as boundary points. The boundary point selection principle is to search from the bottom edge of the image upward to the obstacle that is touched first, and use the grounding point of the obstacle or the projection point of the obstacle projected onto the ground as the boundary point, and all boundary points constitute a boundary line.

[0100] In some embodiments, the processor can be used for an image composed of multiple columns of pixels, and the target pixels in each column of the image are screened out according to a preset boundary point selection principle and marked as boundary points. The boundary point selection principle is to search from the bottom edge of the image upward to the first obstacle touched, and use the grounding point of the obstacle or the projection point of the obstacle onto the ground as the boundary point, and all boundary points form a boundary line.

[0101] It should be noted that, in this embodiment, the image may be historical image data captured by a camera. It is understandable that the obstacle that is touched first is searched upward from the bottom edge of the image, and the grounding point of the obstacle or the projection point of the obstacle projected onto the ground is used as the boundary point (for example, the table surface may protrude more than the table legs, and the projection of the table surface onto the ground is used as the boundary position), which can ensure that no part of the multi-legged robot 100 touches the boundary point, thus avoiding interference of suspended objects on the multi-legged robot 100.

[0102] After obtaining the boundary points, mark them from left to right on the image to form multiple continuous line segments, where the intersection of the line segments is the turning point. The turning points need to follow a principle. That is, the next turning point must be on the right side, directly above or directly below the previous turning point, and cannot be on the left side. Each line segment needs to be marked with the corresponding attributes, so that the boundary line can be obtained.

[0103] When training the deep learning model, the information annotated on the image is taken as the true value. The boundary points and boundary attributes of the image width are predicted according to the deep learning model. The loss function is designed (the loss function can be but is not limited to the logarithmic loss function (logLoss, LR), square error loss function (quadratic loss, QL) or mse, etc.), the boundary points and boundary attributes are trained, and finally the driving space detection model is obtained.

[0104] See also Fig.10 In some embodiments, step 04 includes the sub-steps of:

[0105] 041, determine the image coordinate information of the boundary information of the multi-legged robot's driving space;

[0106] 042, mapping the image coordinate information to the physical space according to the calibration parameters to obtain the target driving space in the physical coordinate system.

[0107] Please further combine Figure 2 In some embodiments, sub-steps 041 and 042 can be implemented by the mapping module 14, or in other words, the mapping module 14 can be used to determine the image coordinate information of the boundary information of the multi-legged robot's driving space, and map the image coordinate information to the physical space according to the calibration parameters to obtain the target driving space in the physical coordinate system.

[0108] In some embodiments, the processor can be used to determine image coordinate information of the boundary information of the multi-legged robot's driving space, and map the image coordinate information to the physical space according to the calibration parameters to obtain the target driving space in the physical coordinate system.

[0109] Specifically, after the pre-processed image is processed according to the preset driving space detection model to generate the multi-legged robot driving space boundary information, the multi-legged robot driving space boundary information is post-processed to decode the image coordinate information of the multi-legged robot driving space boundary information on the image data (the image data of the current driving space collected by the camera). Among them, the post-processing includes but is not limited to translation, scaling, corrosion, expansion, classification, clustering, non-maximum suppression and other methods.

[0110] Furthermore, the image coordinate information can be mapped to the physical space through the camera's intrinsic parameter matrix and distortion parameter matrix, etc., to obtain the target driving space in the physical coordinate system. In this embodiment, the object space can be the ground of the current driving space. In this way, the multi-legged robot 100 can move along the target driving space.

[0111] See also Fig.11 In some embodiments, step 02 includes the sub-steps of:

[0112] 021, calculate the brightness value of the image data;

[0113] 022, determine whether the brightness value of the image data is greater than a preset threshold value. If so, pre-process the image data according to the input format requirements of the preset driving space detection model.

[0114] In some embodiments, sub-steps 021 and 022 may be implemented by the generation module 12. In other words, the generation module 12 may be used to calculate the brightness value of the image data, determine whether the brightness value of the image data is greater than a preset threshold, and if so, pre-process the image data according to the input format requirements of the preset driving space detection model.

[0115] In some embodiments, the processor can be used to calculate the brightness value of the image data and determine whether the brightness value of the image data is greater than a preset threshold. If so, the image data is preprocessed according to the input format requirements of the preset driving space detection model.

[0116] It can be understood that the brightness of the current driving space environment is crucial for the multi-legged robot 100 to identify obstacles. When the brightness of the current driving space environment is not good, the driving space boundary information of the multi-legged robot generated by the driving space detection model based on the preprocessed image obtained from the image data may be inaccurate. For example, when the current driving space of the multi-legged robot 100 is dim, transparent obstacles may not be recognized, resulting in the presence of transparent obstacles in the target driving space, which affects the driving safety of the multi-legged robot 100.

[0117] Therefore, after acquiring the image data of the current driving space collected by the camera, the image data is first judged to determine whether the current lighting environment is suitable for driving space detection. If the current lighting environment is suitable for driving space detection, the image data is preprocessed according to the input format requirements of the preset driving space detection model to generate the target driving space. Otherwise, the processing is stopped. Specifically, the brightness value of the image data is counted. If the brightness value is lower than the set threshold, the driving space detection task is not performed, and a voice prompt is broadcasted that the current light is too dark to perform driving space detection, and the multi-legged robot 100 stops moving. If the brightness value is greater than the preset threshold, the image data is preprocessed to generate a preprocessed image of the preset requirement.

[0118] In this way, the accuracy of the target driving space is avoided from being affected by the lighting environment of the current driving space, thereby improving the driving safety of the multi-legged robot 100.

[0119] In some embodiments, preprocessing includes but is not limited to image normalization processing, downsampling processing, etc. Sub-step 022 includes:

[0120] 0221, downsampling the image data to generate a preprocessed image of a preset size; or

[0121] 0222, perform image normalization processing on the image data to generate a pre-processed image of a predetermined standard.

[0122] In some embodiments, sub-steps 0221 and 0222 may be implemented by the generating module 12. In other words, the generating module 12 may be used to downsample the image data to generate a pre-processed image of a preset size, or to perform image normalization on the image data to generate a pre-processed image of a predetermined standard.

[0123] In some embodiments, the processor may be used to downsample the image data to generate a pre-processed image of a preset size, or to perform image normalization on the image data to generate a pre-processed image of a predetermined standard.

[0124] It should be noted that downsampling, also known as acquisition reduction, is a multi-rate digital signal processing technology or the process of reducing the signal sampling rate. By downsampling the image data, the size of the image data can be reduced. Image normalization refers to the process of performing a series of standard processing transformations on the image data to transform it into a fixed standard form.

[0125] In this way, a pre-processed image is generated by performing image normalization processing or down-sampling processing on the image data, so that the driving space detection model can generate the driving space boundary information of the multi-legged robot according to the pre-processed image.

[0126] Please combine Fig.12 In some embodiments, after step 03, the detection method further comprises:

[0127] 06. Send the boundary attribute information to the controller so that the controller outputs different obstacle avoidance measures according to the boundary attribute information.

[0128] Please further combine Figure 7 In some embodiments, the detection device 10 may further include a sending module 16, and step 06 may be implemented by the sending module 16. In other words, the sending module 16 may be used to send the boundary attribute information to the controller, so that the controller outputs different obstacle avoidance measures according to the boundary attribute information.

[0129] In some embodiments, the processor may be configured to send boundary attribute information to the controller so that the controller outputs different obstacle avoidance measures according to the boundary attribute information.

[0130] In this embodiment, the controller outputs different obstacle avoidance measures according to the boundary attribute information to control the movement of the multi-legged robot 100. The movement of the multi-legged robot 100 may include but is not limited to moving forward, backward, turning or deforming. For example, when there is a suspended obstacle in the current driving space of the multi-legged robot 100, the multi-legged robot 100 will collide with it during normal passage, but it can pass through by crawling. Therefore, the controller generates obstacle avoidance measures after receiving the boundary attribute information, thereby controlling the posture, gait and structure of the multi-legged robot 100 to pass through the current driving space.

[0131] Please further combine Fig.12 In certain embodiments, the detection method further comprises:

[0132] 07, send the boundary attribute information to the mapping module, positioning module and path planning module to provide semantic topology information.

[0133] Please combine Figure 7 In some implementations, step 07 may be implemented by the sending module 16. In other words, the sending module 16 is used to send the boundary attribute information to the mapping module, the positioning module, and the path planning module to provide semantic topology information.

[0134] In some embodiments, the processor can be used to send boundary attribute information to a mapping module, a positioning module, and a path planning module to provide semantic topology information.

[0135] Specifically, the multi-legged robot 100 may also include a mapping module, a positioning module, a path planning module, etc. The mapping module is used to construct a map of the surrounding environment of the multi-legged robot 100, the positioning module is used to determine the location of the multi-legged robot 100, and the path planning module is used to plan a travel route.

[0136] Semantic topological information may include, but is not limited to, expanding the obstacle avoidance range by moving obstacles, simultaneous localization and mapping (SLAM), etc.

[0137] The embodiment of the present application further provides a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes the above-mentioned detection method.

[0138] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0139] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0140] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0141] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0142] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for detecting a drivable space of a multi-legged robot, characterized in that: include: Get the image data collected by the camera at the current moment; The camera is a monocular camera; Preprocessing the image data to generate a preprocessed image that meets the input format requirements of a preset driving space detection model; The preprocessed image is processed according to the driving space detection model to generate the driving space boundary information of the multi-legged robot, wherein the driving space detection model is obtained by training the deep learning model through the labeled image, and the driving space boundary information of the multi-legged robot includes boundary line position information and boundary attribute information, wherein the boundary line position information is the coordinate information of the boundary point on the boundary line; and the boundary attribute information is the category attribute information of the boundary point on the boundary line; Mapping the boundary line position information through the calibration parameters of the camera to generate a target driving space; Before the pre-processed image is processed according to the driving space detection model to generate the driving space boundary information of the multi-legged robot, the method further includes: Detecting the preprocessed image according to the scene detection model, determining the current scene of the multi-legged robot, and using the drivable space detection model trained for the current scene as a preset drivable space detection model; Preprocessing the image data to generate a preprocessed image that meets the preset driving space detection model input format requirements includes: Calculating the brightness value of the image data; It is determined whether the brightness value of the image data is greater than a preset threshold value. If so, the image data is preprocessed according to the input format requirement of the preset driving space detection model.

2. The detection method according to claim 1, characterized in that The driving space detection model includes a feature extraction network and a task branch network. The preprocessed image is processed according to the preset driving space detection model to generate the driving space boundary information of the multi-legged robot, including: Processing the preprocessed image through the feature extraction network to generate a basic feature map; The basic feature graph is processed by the task branch network to obtain the boundary information of the multi-legged robot's driving space.

3. The detection method according to claim 1, characterized in that The driving space detection model is obtained by training a deep learning model through labeled images, and includes: The image is composed of multiple columns of pixels, and target pixels are screened out from each column of the image according to a preset boundary point selection principle and marked as boundary points. The boundary point selection principle is to search upward from the bottom edge of the image to the obstacle that is touched first, and the grounding point of the obstacle or the projection point of the obstacle onto the ground is used as the boundary point; All of the boundary points constitute a boundary line.

4. The detection method according to claim 1, characterized in that The boundary attribute information includes stationary obstacles and / or moving obstacles, and the stationary obstacles include suspended obstacles, transparent obstacles, mirror obstacles, stairs, and doors.

5. The detection method according to claim 1, characterized in that The step of mapping the boundary line position information through the calibration parameters of the camera to generate a target driving space includes: Determine image coordinate information of boundary information of the multi-legged robot's travel space; The image coordinate information is mapped to a physical space according to the calibration parameters to obtain the target driving space in a physical coordinate system.

6. The detection method according to claim 1, characterized in that The detection method further comprises: sending the boundary attribute information to a controller so that the controller outputs different obstacle avoidance measures according to the boundary attribute information; and / or The boundary attribute information is sent to a mapping module, a positioning module, and a path planning module to provide semantic topology information.

7. A device for detecting the drivable space of a multi-legged robot, characterized in that: include: An acquisition module is used to obtain image data collected by the camera at the current moment; The camera is a monocular camera; A generating module, used for preprocessing the image data to generate a preprocessed image that meets the input format requirements of a preset driving space detection model; A processing module is used to process the pre-processed image according to a preset driving space detection model to generate driving space boundary information of the multi-legged robot, wherein the driving space detection model is obtained by training a deep learning model through annotated images, and the driving space boundary information of the multi-legged robot includes boundary line position information and boundary attribute information, wherein the boundary line position information is the coordinate information of the boundary points on the boundary line; and the boundary attribute information is the category attribute information of the boundary points on the boundary line; and A mapping module, used for mapping the boundary line position information to generate a target driving space through calibration parameters of the camera; A selection module is used to detect the pre-processed image according to the scene detection model, determine the current scene of the multi-legged robot, and use the drivable space detection model trained for the current scene as the preset drivable space detection model; The generating module is also used to: calculate the brightness value of the image data; It is determined whether the brightness value of the image data is greater than a preset threshold value. If so, the image data is preprocessed according to the input format requirement of the preset driving space detection model.

8. A multi-legged robot, characterized in that: The method comprises a processor and a storage unit, wherein the storage unit stores at least one computer program, and when the computer program is executed by the processor, the processor executes the method for detecting the drivable space of a multi-legged robot according to any one of claims 1 to 6.

9. A non-volatile computer-readable storage medium containing a computer program, characterized in that: When the computer program is executed by a processor, the processor executes the method for detecting the drivable space of a multi-legged robot according to any one of claims 1 to 6.

Images