Fall-prevention method for mobile robot, and mobile robot and electronic device

Abstract

A fall-prevention method for a mobile robot, and a mobile robot, an electronic device and a machine-readable storage medium. In the method, by means of performing a differential operation on depth values of matched pixel point positions in a depth map corresponding to an image collected by a mobile robot during working and in a depth template map, a height difference between a working face and a reference plane is represented on the basis of the difference between the depth value of each pixel point in the image collected by the mobile robot during working and the depth value of a corresponding pixel point in the image collected by the mobile robot when working on the reference plane; whether there is a falling risk area on a working path of the mobile robot is predicted on the basis of the height difference; and when it is found that there is a falling risk area on the working path, the working path of the mobile robot is modified, so as to control the mobile robot to avoid the falling risk area to work.

Description

Mobile robot anti-fall method, mobile robot and electronic equipment Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method for preventing a mobile robot from falling, a mobile robot, and an electronic device. Background Art

[0002] With the development of society and advancements in technology, mobile robots are widely used in various industries. For example, they are used in homes for cleaning and in restaurants for food delivery. During their work, mobile robots may encounter areas with sudden changes in height, such as door thresholds. To prevent the robot from falling and causing safety hazards, mobile robots generally have anti-fall features.

[0003] Summary of the Invention

[0004] The present application provides a method for preventing a mobile robot from falling, a mobile robot, and an electronic device.

[0005] An embodiment of the present application provides a method for preventing a mobile robot from falling, the method comprising: obtaining a depth map corresponding to an image collected by the mobile robot during operation, wherein the image is about a working path of the mobile robot; performing a differential operation on depth values ​​at matching pixel positions in the depth map and a depth template map to obtain a differential map, wherein the depth template map is a depth map corresponding to an image collected when the mobile robot is working on a reference plane; predicting whether there is a fall risk area on the working path of the mobile robot based on the depth values ​​at each pixel position in the differential map; if there is a fall risk area on the working path, modifying the working path of the mobile robot to control the mobile robot to avoid the fall risk area to work.

[0006] An embodiment of the present application also provides a mobile robot, including a monocular image acquisition component and a processor; the monocular image acquisition component is used to acquire images about the working path of the mobile robot during the operation of the mobile robot; the processor is used to obtain a depth map corresponding to the image; a differential operation is performed on the depth values ​​at the matching pixel positions in the depth map and the depth template map to obtain a differential map, wherein the depth template map is a depth map corresponding to the image acquired when the mobile robot is working on a reference plane; based on the depth values ​​at the respective pixel positions in the differential map, it is predicted whether there is a fall risk area on the working path of the mobile robot; if there is a fall risk area on the working path, the working path of the mobile robot is modified to control the mobile robot to avoid the fall risk area to work.

[0007] An embodiment of the present application further provides an electronic device, comprising: a processor and a memory for storing computer program instructions, wherein the computer program instructions, when executed by the processor, enable the processor to execute the above method.

[0008] An embodiment of the present application also provides a machine-readable storage medium, which stores computer program instructions. When the computer program instructions are executed, the above method can be implemented.

[0009] From the above, it can be seen that in the embodiment of the present application, a differential operation is performed on the depth values ​​at the matching pixel positions in the depth map corresponding to the image collected by the mobile robot during the work process and the depth template map (the depth map corresponding to the image collected when the mobile robot works on the reference plane), so as to realize the difference between the depth values ​​of each pixel point in the image collected by the mobile robot during the work process and the corresponding pixel point in the image collected when the mobile robot works on the reference plane to reflect the height difference between the working surface and the reference plane, and based on the height difference, predict whether there is a fall risk area on the working path of the mobile robot, which can effectively predict the fall risk of the mobile robot and improve the anti-fall effect.

[0010] Furthermore, in an embodiment of the present application, when a fall risk area is found on the working path, the working path of the mobile robot is modified to control the mobile robot to avoid the fall risk area to work, thereby improving the anti-fall effect of the mobile robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG1 is a flow chart of a method for preventing a mobile robot from falling provided by an exemplary embodiment of the present application.

[0012] FIG2 is a schematic diagram of a flow chart of predicting whether there is a fall risk area on a working path of a mobile robot, provided by an exemplary embodiment of the present application.

[0013] FIG3 is a schematic diagram of a process for locating a fall risk area provided by an exemplary embodiment of the present application.

[0014] FIG4 is a schematic structural diagram of a mobile robot provided by an exemplary embodiment of the present application.

[0015] FIG5 is a schematic diagram of the hardware structure of an electronic device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0016] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0017] The terms used in this application are for the purpose of describing specified embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0018] A common anti-fall method in related technologies involves installing an infrared transmitter and receiver on the bottom of a mobile robot. This device detects changes in the infrared signal waveform to determine the robot's current state, thus preventing it from falling. However, this infrared transmitter and receiver can only detect the terrain conditions below the robot and cannot predict the risk of falling on the robot's subsequent route, resulting in less than ideal anti-fall effects.

[0019] In view of this, the present application provides a mobile robot anti-falling method, a mobile robot and an electronic device, which can effectively predict the falling risk of the mobile robot and improve the anti-falling effect.

[0020] In order to enable those skilled in the art to better understand the technical solutions provided by the exemplary embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0021] Refer to FIG1 , which is a flow chart of a method for preventing a mobile robot from falling provided by an exemplary embodiment of the present application.

[0022] As shown in FIG1 , the process includes the following steps S101 to S104 .

[0023] In step S101, a depth map corresponding to an image collected by a mobile robot during operation is obtained.

[0024] In this step, there are many specific ways for the mobile robot to collect images during operation. For example, as an embodiment, the mobile robot collects images during operation using a monocular image acquisition component disposed at a designated position thereof.

[0025] Exemplarily, the monocular image acquisition component can be a monocular camera, which can be set at a specified position of the shell of the mobile robot. The monocular camera collects images about the working path of the mobile robot during the operation of the mobile robot.

[0026] Depth images, also known as range images, refer to images that use the distance (depth) values ​​of each point in the scene captured by the image collector as pixel values. They directly reflect the geometric shape of the visible surface in the scene.

[0027] In this step, there are many specific ways to obtain the depth map. For example, as an embodiment, the image collected by the mobile robot during operation is input into a trained monocular depth prediction neural network. The monocular depth prediction neural network performs prediction and outputs the result, and the output result of the monocular depth prediction neural network is determined as the depth map corresponding to the image.

[0028] Specifically, the monocular depth prediction neural network is trained using video streams captured by a specified model of monocular image acquisition component as training samples. This specified model of monocular image acquisition component is the same model as the monocular image acquisition component used by the mobile robot to collect images during operation. The following example illustrates how the monocular depth prediction neural network is trained.

[0029] First, the monodepth2 network is selected as the neural network model to be trained. The video stream captured by the monocular image acquisition component of the above-specified model is input into the neural network model as a training sample. The neural network model selects the i-th frame image and the i+1-th frame image in the video stream, where i is a positive integer, and performs feature extraction and pose analysis on the i-th frame image and the i+1-th frame image respectively to obtain the feature map and pose map of the i-th frame image and the i+1-th frame image respectively. Then, the reprojection error between the pose map of the i-th frame image and the pose map of the i+1-th frame image is calculated. The reprojection error is used to adjust the parameters of the neural network model until convergence, resulting in a trained monocular depth prediction neural network. The trained monocular depth prediction neural network has the function of predicting the depth map corresponding to the input image.

[0030] After obtaining a trained monocular depth prediction neural network, the trained monocular depth prediction neural network can be pre-installed on the mobile robot in the form of a program, optionally on the mobile robot's processor. Images captured by the monocular image acquisition component during the operation of the mobile robot are input into the trained monocular depth prediction neural network, and the depth map corresponding to the image is determined based on the output of the monocular depth prediction neural network.

[0031] In the above-mentioned specific method for obtaining a depth map, images captured by the monocular image acquisition component during the operation of the mobile robot are introduced into the monocular depth prediction neural network for prediction, thereby obtaining a depth map corresponding to the captured images. This method eliminates the need for expensive depth cameras and can reduce the manufacturing cost of the mobile robot compared to directly capturing depth maps with a depth camera installed on the mobile robot.

[0032] In step S102, a differential operation is performed on the depth values ​​at the matching pixel positions in the depth map and the depth template map to obtain a differential map; wherein the depth template map is a depth map corresponding to the image collected when the mobile robot works on the reference plane.

[0033] There are many ways to implement this step S102. For example, as an embodiment, a region of interest (ROI) is extracted from the depth map, a template ROI is extracted from the depth template map, the size of the template ROI is matched with the size of the ROI, and the depth values ​​of the pixels at the same coordinates in the ROI and the template ROI are differentially calculated to obtain a differential map.

[0034] Here, the ROI is the area in the depth map that contains the most work surface information, and the template ROI is the area in the depth template map that matches the ROI. For example, the lower third of the depth map is selected from the bottom to the top as the ROI, and the lower third of the depth template map is also selected from the bottom to the top as the template ROI corresponding to the ROI. The work surface can be the working ground on which the mobile robot is located during operation, but the present application is not limited thereto, and the work surface can also be other surfaces.

[0035] This step S102 can also be implemented in other ways. For example, as an embodiment, the ROI is the entire depth map, and correspondingly, the template ROI is the entire depth template map. The depth values ​​of the pixels at the same coordinates in the entire depth map and the entire depth template map are differentially calculated to obtain a differential map.

[0036] In step S103, based on the depth value of each pixel position in the difference image, it is predicted whether there is a fall risk area on the working path of the mobile robot; the fall risk area refers to an area where the mobile robot has a fall risk.

[0037] Since the depth template image is the depth map corresponding to the image collected by the mobile robot on the reference plane, and the depth map is the depth map corresponding to the image collected when the mobile robot is working on the working surface, a differential operation is performed on the depth values ​​at the matching pixel positions in the depth map and the depth template image. This can reflect the height difference between the working surface and the reference plane based on the difference in the depth value of each pixel in the image collected by the mobile robot during work and the corresponding pixel in the image collected by the mobile robot on the reference plane, thereby predicting whether there is a fall risk area on the working path of the mobile robot based on the height difference.

[0038] There are many specific implementation methods for predicting whether there is a fall risk area on the working path of the mobile robot. Examples will be given later and will not be described in detail here.

[0039] In step S104, if there is a falling risk area on the working path, the working path of the mobile robot is modified to control the mobile robot to avoid the falling risk area to work.

[0040] If there is a fall risk area on the work path, it is necessary to accurately locate the fall risk area. The specific method of locating the fall risk area will be described in detail later, so it is not detailed here.

[0041] At this point, the process shown in Figure 1 is completed.

[0042] Through the process shown in Figure 1, by performing differential operations on the depth values ​​at the matching pixel positions in the depth map corresponding to the image collected by the mobile robot during the work process and the depth template map (the depth map corresponding to the image collected when the mobile robot works on the reference plane), the height difference between the working surface and the reference plane can be reflected based on the difference in the depth value of each pixel point in the image collected by the mobile robot during the work process and the corresponding pixel point in the image collected when the mobile robot works on the reference plane, and based on the height difference, it is predicted whether there is a fall risk area on the working path of the mobile robot. This can effectively predict the fall risk of the mobile robot and improve the anti-fall effect.

[0043] Furthermore, in an embodiment of the present application, when a fall risk area is found on the working path, the working path of the mobile robot is modified to control the mobile robot to avoid the fall risk area to work, thereby improving the anti-fall effect of the mobile robot.

[0044] The following is a detailed description of the above-mentioned prediction of whether there is a fall risk area on the working path of the mobile robot.

[0045] Refer to FIG2 , which is a flowchart illustrating a method of predicting whether there is a fall risk area on a working path of a mobile robot according to an exemplary embodiment of the present application.

[0046] As shown in FIG. 2 , the process may include the following steps S201 to S203 .

[0047] In step S201 , the depth value at each pixel position in the difference image is binarized to obtain a binarized image.

[0048] In this step, the binarization process can be implemented in many ways. For example, as one embodiment, a preset depth threshold is used as the boundary value for the binary segmentation. The preset depth threshold is set based on the critical depth value that causes the mobile robot to fall. If the depth value at a pixel position is greater than or equal to the preset depth threshold, the pixel position is marked as a first mark, such as 1. If the depth value at a pixel position is less than the preset depth threshold, the pixel position is marked as a second mark, such as 0.

[0049] The critical depth value for a mobile robot's fall is related to the size of its wheels: larger wheels increase the critical depth value. Therefore, the preset depth threshold is positively correlated with the size of the mobile robot's wheels: larger wheels increase the preset depth threshold, and smaller wheels decrease the preset depth threshold.

[0050] In step S202, at least one connected domain is determined in the binary image, and it is determined whether there is a target connected domain in the at least one connected domain; the depth value of each pixel position in the target connected domain is greater than or equal to a preset depth threshold, and the area of ​​the target connected domain is greater than a preset area threshold.

[0051] In this step, the preset area threshold is set based on the critical area that causes the mobile robot to fall. The critical area is the maximum area of ​​the area with a sharp height change that the mobile robot can cross on the working surface. Furthermore, given that the mobile robot's crossing ability is related to the size of its wheels, the preset area threshold is also positively correlated with the size of the wheels. Larger wheels indicate a greater range for the mobile robot to cross, while a larger preset area threshold indicates a smaller range for the mobile robot to cross, and a smaller preset area threshold is also set.

[0052] If the depth value at each pixel in a connected domain is greater than the preset depth threshold, but the area of ​​the connected domain is very small, this indicates that there is a certain height difference between the working surface corresponding to the pixel and the reference plane, but the mobile robot can cross this height difference. Therefore, this connected domain is not the target connected domain. Only connected domains with depth values ​​greater than or equal to the preset depth threshold at each pixel and an area greater than the preset area threshold are considered target connected domains. In other words, each pixel in the target connected domain has the first marker mentioned above, and the area of ​​the target connected domain is greater than the preset area threshold.

[0053] In step S203 , if the target connected domain exists in the at least one connected domain, it is determined that a fall risk area exists on the working path of the mobile robot.

[0054] Specifically, if a target connected domain exists in the binary image, a fall risk region is determined to exist in the mobile robot's working path. This fall risk region is the physical area on the working surface corresponding to the target connected domain. Similarly, if multiple target connected domains exist in the binary image, multiple fall risk regions are determined to exist in the mobile robot's working path. These multiple fall risk regions correspond one-to-one to these multiple target connected domains.

[0055] At this point, the process shown in Figure 2 is completed.

[0056] Through the process shown in Figure 2, it is possible to predict whether there is a fall risk area on the working path of the mobile robot. Therefore, when there is a fall risk area on the working path of the mobile robot, the working path of the mobile robot can be modified to control the mobile robot to avoid the fall risk area to work, thereby improving the anti-fall effect of the mobile robot.

[0057] How to locate the fall risk area is described in detail below with reference to embodiments.

[0058] FIG3 is a schematic diagram of a process for locating a fall risk area according to an exemplary embodiment of the present application. As shown in FIG3 , the process may include the following steps S301 and S302 .

[0059] In step S301, the positioning angle value and the positioning distance value corresponding to each pixel point position in the target connected domain are obtained.

[0060] In this step, the positioning angle value corresponding to any pixel point position refers to the relative angle between the physical position on the working path corresponding to the pixel point position and the geometric center position of the mobile robot, and the positioning distance value corresponding to any pixel point position refers to the relative distance between the physical position on the working path corresponding to the pixel point position and the geometric center position of the mobile robot.

[0061] Since the mobile robot already has preset positioning angles and distances for each pixel in the captured image when it captures it, and the binary image is obtained by processing this image, each pixel in the binary image also has a corresponding positioning angle and distance. Therefore, the positioning angle and distance values ​​corresponding to each pixel in the target connected domain can be obtained from the positioning angle and distance values ​​corresponding to each pixel in the binary image.

[0062] In an optional embodiment, the specific implementation form of obtaining the positioning angle value and positioning distance value corresponding to each pixel point position in the target connected domain from the positioning angle value and positioning distance value corresponding to each pixel point position in the binary image can be that the positioning angle value and positioning distance value corresponding to each pixel point position in the binary image are listed in a table, which lists the coordinates of each pixel point and the positioning angle value and positioning distance value corresponding to the coordinates of the pixel point. Based on the coordinates of each pixel point in the binary image, the positioning angle value and positioning distance value corresponding to the coordinates of each pixel point in the target connected domain are queried from the table.

[0063] In step S302, the orientation of the fall risk area is determined according to the positioning angle value and the positioning distance value corresponding to each pixel point position in the target connected domain.

[0064] Specifically, since the positioning angle value corresponding to any pixel position can represent the relative angle between the physical position on the working path corresponding to the pixel position and the geometric center position of the mobile robot, the positioning distance value corresponding to any pixel position can represent the relative distance between the physical position on the working path corresponding to the pixel position and the geometric center position of the mobile robot, then, after obtaining the positioning angle value and positioning distance value corresponding to each pixel position in the target connected domain, the orientation of the fall risk area can be determined.

[0065] At this point, the process shown in Figure 3 is completed.

[0066] Through the process shown in FIG3 , the fall risk area can be accurately located, and based on the determined location of the fall risk area, the mobile robot can be controlled to avoid the fall risk area, thereby reducing the probability of the mobile robot falling.

[0067] Thus, the anti-falling method of the mobile robot provided by an exemplary embodiment of the present application is completed. The mobile robot provided by an exemplary embodiment of the present application is described below.

[0068] Referring to Figure 4, which is a schematic diagram of the structure of a mobile robot provided by an exemplary embodiment of the present application, as shown in Figure 4, the mobile robot includes a monocular image acquisition component and a processor.

[0069] The monocular image acquisition component is used to acquire images during the operation of the mobile robot, wherein the images are about the working path of the mobile robot.

[0070] A processor is configured to obtain a depth map corresponding to the image; perform a differential operation on depth values ​​at matching pixel positions in the depth map and a depth template map to obtain a differential map, wherein the depth template map is a depth map corresponding to an image acquired when the mobile robot is operating on a reference plane; predict whether there is a fall risk area on a working path of the mobile robot based on the depth values ​​at each pixel position in the differential map; if there is a fall risk area on the working path, modify the working path of the mobile robot to control the mobile robot to avoid the fall risk area to operate.

[0071] As an embodiment, when obtaining the depth map corresponding to the image, the processor is further used to: input the image into a trained monocular depth prediction neural network, the monocular depth prediction neural network is trained using a video stream captured by a specified model of a monocular image acquisition component as a training sample, and the specified model of the monocular image acquisition component is the same model as the monocular image acquisition component used by the mobile robot to capture images during operation; based on the output result of the monocular depth prediction neural network, determine the depth map corresponding to the image.

[0072] As an embodiment, when performing a differential operation on the depth values ​​at the matching pixel positions in the depth map and the depth template map to obtain a differential map, the processor is further used to: extract the region of interest ROI from the depth map; extract the template ROI from the depth template map, and the template ROI matches the size of the ROI; perform a differential operation on the depth values ​​at the matching pixel positions in the ROI and the template ROI to obtain a differential map.

[0073] As an embodiment, when predicting whether there is a fall risk area on the working path of the mobile robot based on the depth values ​​at each pixel point position in the differential image, the processor is further used to: binarize the depth values ​​at each pixel point position in the differential image to obtain a binary image; determine at least one connected domain in the binary image, and judge whether there is a target connected domain in the at least one connected domain; the depth value at each pixel point position in the target connected domain is greater than or equal to a preset depth threshold, and the area of ​​the target connected domain is greater than a preset area threshold; if the target connected domain exists in the at least one connected domain, it is determined that there is a fall risk area on the working path of the mobile robot.

[0074] As an embodiment, the fall risk area is located through the following steps: obtaining the positioning angle value and positioning distance value corresponding to each pixel position in the target connected domain; wherein, for each pixel point, the positioning angle value corresponding to the pixel point position refers to the relative angle between the physical position on the working path corresponding to the pixel point position and the geometric center position of the mobile robot, and the positioning distance value corresponding to the pixel point position refers to the relative distance between the physical position on the working path corresponding to the pixel point position and the geometric center position of the mobile robot; based on the positioning angle value and positioning distance value corresponding to each pixel point position in the target connected domain, the orientation of the fall risk area is determined.

[0075] As an embodiment, the preset depth threshold and the preset area threshold are respectively set based on the critical depth value and the critical area that cause the mobile robot to fall, and the preset depth threshold and the preset area threshold are both positively correlated with the size of the wheels of the mobile robot.

[0076] For the mobile robot embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment of the present application. Those of ordinary skill in the art can understand and implement it without paying any creative work.

[0077] Please refer to Figure 5, which is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 501, a communication interface 502, a memory 503 and a communication bus 504. The processor 501, the communication interface 502 and the memory 503 communicate with each other through the communication bus 504. Among them, a computer program is stored on the memory 503; the processor 501 can execute the anti-fall method of the mobile robot described in the above embodiment by executing the program stored on the memory 503. According to the actual function of the electronic device, the electronic device may also include other hardware, which will not be described in detail.

[0078] Embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0079] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform the corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0080] Computers suitable for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to such mass storage devices to receive data from them or to transmit data to them, or both. However, a computer does not necessarily have such devices. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0081] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0082] Although this specification contains many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of specific embodiments of the specified invention. Certain features described in multiple embodiments within this specification can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features can work in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a sub-combination or a variation of the sub-combination.

[0083] Similarly, although operations are depicted in a specified order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the specified order shown or performed sequentially, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0084] Thus, specified embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.

[0085] The above descriptions are merely some embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for preventing a mobile robot from falling, comprising: Obtaining a depth map corresponding to an image collected by the mobile robot during operation, wherein the image is about a working path of the mobile robot; Performing a differential operation on depth values ​​at matching pixel positions in the depth map and the depth template map to obtain a differential map, wherein the depth template map is a depth map corresponding to an image collected when the mobile robot works on a reference plane; Predicting whether there is a fall risk area on the working path of the mobile robot according to the depth value at each pixel position in the difference image; If the falling risk area exists on the working path, the working path of the mobile robot is modified to control the mobile robot to avoid the falling risk area to work.

2. The method according to claim 1, wherein: A monocular image acquisition component is provided at a designated position of the mobile robot; Among them, the images collected by the mobile robot during the working process are collected by the monocular image acquisition component.

3. The method according to claim 1 or 2, wherein: The obtaining of a depth map corresponding to an image collected by the mobile robot during operation includes: Inputting images collected by the mobile robot during operation into a trained monocular depth prediction neural network, wherein the monocular depth prediction neural network is trained using a video stream collected by a monocular image acquisition component of a specified model as a training sample, and the monocular image acquisition component of the specified model and the monocular image acquisition component used by the mobile robot to collect images during operation are of the same model; Determine the depth map corresponding to the image based on the output result of the monocular depth prediction neural network.

4. The method according to any one of claims 1 to 3, wherein: The step of performing a differential operation on the depth values ​​at the positions of the matching pixels in the depth map and the depth template map to obtain a differential map includes: Extracting a template region of interest ROI from the depth template map; Extracting a ROI from the depth map, where the extracted ROI matches the size of the template ROI; A difference operation is performed on the depth values ​​at the positions of the matching pixels in the ROI and the template ROI to obtain the difference map.

5. The method according to any one of claims 1 to 4, wherein: The predicting, based on the depth value at each pixel position in the differential image, whether there is a falling risk area on the working path of the mobile robot comprises: Binarizing the depth value of each pixel in the differential image to obtain a binary image; Determine at least one connected domain in the binary image, and judge whether there is a target connected domain in the at least one connected domain, the depth value of each pixel point in the target connected domain is greater than or equal to a preset depth threshold, and the area of ​​the target connected domain is greater than a preset area threshold; If the target connected domain exists in the at least one connected domain, it is determined that there is a falling risk area on the working path of the mobile robot.

6. The method according to claim 5, wherein: The fall risk area is located by the following steps: Obtaining a positioning angle value and a positioning distance value corresponding to each pixel position in the target connected domain; wherein, for each pixel, the positioning angle value corresponding to the pixel position refers to the relative angle between the physical position on the working path corresponding to the pixel position and the geometric center position of the mobile robot, and the positioning distance value corresponding to the pixel position refers to the relative distance between the physical position on the working path corresponding to the pixel position and the geometric center position of the mobile robot; The orientation of the fall risk area is determined according to the positioning angle value and the positioning distance value corresponding to the position of each pixel point in the target connected domain.

7. The method according to claim 5 or 6, wherein: The preset depth threshold and the preset area threshold are respectively set based on the critical depth value and the critical area that cause the mobile robot to fall, and are both positively correlated with the size of the wheels of the mobile robot.

8. A mobile robot comprising a monocular image acquisition component and a processor; The monocular image acquisition component is used to acquire images about the working path of the mobile robot during the working process of the mobile robot; The processor is used to obtain a depth map corresponding to the image; perform a differential operation on the depth values ​​at the matching pixel positions in the depth map and the depth template map to obtain a differential map, wherein: The depth template map is a depth map corresponding to an image collected when the mobile robot works on a reference plane; Predicting whether there is a fall risk area on the working path of the mobile robot according to the depth value at each pixel position in the difference image; If the falling risk area exists on the working path, the working path of the mobile robot is modified to control the mobile robot to avoid the falling risk area to work.

9. The mobile robot according to claim 8, wherein: When obtaining the depth map corresponding to the image, the processor is further used to: Inputting the image into a trained monocular depth prediction neural network, wherein the monocular depth prediction neural network is trained using a video stream acquired by a monocular image acquisition component of a specified model as a training sample, and the monocular image acquisition component of the specified model is the same model as the monocular image acquisition component used by the mobile robot to acquire images during operation; Determine the depth map corresponding to the image based on the output result of the monocular depth prediction neural network.

10. The mobile robot according to claim 8 or 9, wherein: When performing a differential operation on the depth values ​​at the matching pixel positions in the depth map and the depth template map to obtain a differential map, the processor is further configured to: Extracting a template region of interest ROI from the depth template map; Extracting a ROI from the depth map, where the extracted ROI matches the size of the template ROI; A difference operation is performed on the depth values ​​at the positions of the matching pixels in the ROI and the template ROI to obtain the difference map.

11. The mobile robot according to any one of claims 8 to 10, wherein: When predicting whether there is a fall risk area on the working path of the mobile robot according to the depth value at each pixel point position in the difference image, the processor is further used to: Binarizing the depth value of each pixel in the differential image to obtain a binary image; Determine at least one connected domain in the binary image, and judge whether there is a target connected domain in the at least one connected domain, the depth value of each pixel point in the target connected domain is greater than or equal to a preset depth threshold, and the area of ​​the target connected domain is greater than a preset area threshold; If the target connected domain exists in the at least one connected domain, it is determined that there is a falling risk area on the working path of the mobile robot.

12. The mobile robot according to claim 11, wherein: The fall risk area is located by the following steps: Obtaining a positioning angle value and a positioning distance value corresponding to each pixel position in the target connected domain; wherein, for each pixel, the positioning angle value corresponding to the pixel position refers to the relative angle between the physical position on the working path corresponding to the pixel position and the geometric center position of the mobile robot, and the positioning distance value corresponding to the pixel position refers to the relative distance between the physical position on the working path corresponding to the pixel position and the geometric center position of the mobile robot; The orientation of the fall risk area is determined according to the positioning angle value and the positioning distance value corresponding to the position of each pixel point in the target connected domain.

13. The mobile robot according to claim 11 or 12, wherein: The preset depth threshold and the preset area threshold are respectively set based on the critical depth value and the critical area that cause the mobile robot to fall, and are both positively correlated with the size of the wheels of the mobile robot.

14. An electronic device comprising: processor; as well as A memory, in which computer program instructions are stored, and when the computer program instructions are executed by the processor, the processor is caused to perform the method according to any one of claims 1 to 7.

15. A machine-readable storage medium storing computer program instructions, which can implement the method according to any one of claims 1 to 7 when the computer program instructions are executed.

Images