Fall prevention method for mobile robot, mobile robot, and electronic device

Abstract

This application provides a fall prevention method for a mobile robot, the mobile robot itself, and an electronic device. The method performs a difference operation on the depth values ​​at matching pixel locations in the depth map corresponding to images captured by the mobile robot during operation and a depth map template. This difference reflects the height difference between the working surface and the reference surface, allowing for the prediction of whether a fall-risk area exists on the mobile robot's working path. This effectively predicts the fall risk of the mobile robot, improving fall prevention performance. Furthermore, when a fall-risk area is detected on the working path, the method modifies the mobile robot's path to avoid this area, further enhancing fall prevention.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a method for preventing a mobile robot from falling, the mobile robot, and an electronic device. Background Technology

[0002] With social development and technological advancements, mobile robots are widely used across various industries. For example, they are used for cleaning in homes and for food delivery in restaurants. Mobile robots often encounter areas with sudden changes in height, such as thresholds. To prevent them from falling and causing safety hazards, mobile robots are generally equipped with fall protection features.

[0003] A common fall prevention method in existing technologies involves installing an infrared transmitter and receiver on the bottom of a mobile robot. By analyzing changes in the infrared signal waveform, the robot's current state is determined, thus preventing falls. However, this infrared transmitter and receiver can only receive information about the terrain beneath the robot and cannot predict the risk of falls along the robot's subsequent path, resulting in a less than ideal fall prevention effect. Summary of the Invention

[0004] In view of this, this application provides a method for preventing falls of a mobile robot, a mobile robot, and an electronic device, which can effectively predict the fall risk of the mobile robot and improve the fall prevention effect.

[0005] This application provides a method for preventing a mobile robot from falling, the method comprising:

[0006] Obtain the depth map corresponding to the images collected by the mobile robot during its operation;

[0007] The depth values ​​at matching pixel locations in the depth map and the obtained depth template map are differentially analyzed to obtain a difference map; where the depth template map is the depth map corresponding to the image captured by the mobile robot when it is working on the reference plane.

[0008] Based on the depth values ​​at each pixel location in the difference map, predict whether there are fall-risk areas on the mobile robot's working path; fall-risk areas refer to areas where the mobile robot is at risk of falling.

[0009] If there are areas with a risk of falling on the work path, modify the work path of the mobile robot to control the mobile robot to avoid the areas with a risk of falling on the work path.

[0010] This application also provides a mobile robot, including: a monocular image acquisition component and a processor;

[0011] A monocular image acquisition component is used to acquire images during the operation of a mobile robot.

[0012] The processor is used to obtain depth maps corresponding to images captured by the mobile robot during its operation.

[0013] The depth values ​​at matching pixel locations in the depth map and the obtained depth template map are differentially analyzed to obtain a difference map; where the depth template map is the depth map corresponding to the image captured by the mobile robot when it is working on the reference plane.

[0014] Based on the depth values ​​at each pixel location in the difference map, predict whether there are fall-risk areas on the mobile robot's working path; fall-risk areas refer to areas where the mobile robot is at risk of falling.

[0015] If there are areas with a risk of falling on the work path, modify the work path of the mobile robot to control the mobile robot to avoid the areas with a risk of falling on the work path.

[0016] This application also provides an electronic device, including: a processor and a memory for storing computer program instructions, which, when executed by the processor, cause the processor to perform the steps of the method described above.

[0017] This application also provides a machine-readable storage medium storing computer program instructions that, when executed, enable the implementation of the steps described above.

[0018] As can be seen from the above technical solutions, in this embodiment, the depth values ​​at the matching pixel positions in the depth map corresponding to the image collected by the mobile robot during its working process (hereinafter referred to as the working ground) and the depth map template (the depth map corresponding to the image collected by the mobile robot when it works on the reference plane) are differentially calculated. This allows the difference between the depth values ​​of each pixel in the image collected by the mobile robot during its working process and each pixel on the reference plane to reflect the height difference between the working ground and the reference ground. Based on this height difference, the system can predict whether there is a fall risk area on the working path of the mobile robot. This enables the effective prediction of the fall risk of the mobile robot, thereby improving the fall prevention effect.

[0019] Furthermore, in this embodiment, when a fall-risk area is detected on the work path, the work path of the mobile robot is modified to control the mobile robot to avoid the fall-risk area and improve the fall prevention effect of the mobile robot. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a method for preventing a mobile robot from falling, provided as an exemplary embodiment of this application.

[0021] Figure 2This is a schematic diagram of a process for predicting whether there is a fall risk area on the working path of a mobile robot, provided as an exemplary embodiment of this application.

[0022] Figure 3 This is a schematic diagram of a process for locating a fall risk area, provided as an exemplary embodiment of this application.

[0023] Figure 4 This is a schematic diagram of the structure of a mobile robot provided as an exemplary embodiment of this application.

[0024] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided as an exemplary embodiment of this application. Detailed Implementation

[0025] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0026] The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0027] To enable those skilled in the art to better understand the technical solutions provided in an exemplary embodiment of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0028] See Figure 1 , Figure 1 This is a schematic flowchart illustrating a fall prevention method for a mobile robot provided as an exemplary embodiment of this application. As an example, this method can be applied to a mobile robot.

[0029] like Figure 1 As shown, the process may include the following steps:

[0030] S101, Obtain the depth map corresponding to the image collected by the mobile robot during its operation.

[0031] In this step, there are many specific ways for the mobile robot to acquire images during its work. For example, as an embodiment, the mobile robot uses a monocular image acquisition component set at its designated location to acquire images during its work.

[0032] For example, the monocular image acquisition component can be a monocular camera, which can be set at a designated position on the shell of the mobile robot, and the monocular camera acquires images during the operation of the mobile robot.

[0033] Depth images, also known as range images, are images that use the distance (depth) values ​​of points in a scene captured by an image acquisition device as pixel values. They directly reflect the geometry of visible surfaces in the scene.

[0034] In this step, there are many specific ways to obtain the depth map. For example, as an embodiment, the image collected by the mobile robot during its operation is input into a trained monocular depth prediction neural network. The monocular depth prediction neural network predicts and outputs the result, and the output result of the monocular depth prediction neural network is determined as the depth map corresponding to the image.

[0035] Specifically, the monocular depth prediction neural network is trained using video streams acquired by a specified model of monocular image acquisition component as training samples. This specified model of monocular image acquisition component is the same model as the monocular image acquisition component used by the mobile robot to acquire images during its operation. The following example illustrates how the monocular depth prediction neural network is trained.

[0036] First, the monodepth2 network was selected as the neural network model to be trained. The video stream acquired by the specified monocular image acquisition component was used as training samples and input into the neural network model. The model selected the i-th frame and the (i+1)-th frame (where i is a positive integer) from the video stream and performed feature extraction and pose analysis on the i-th and (i+1)-th frames respectively, obtaining their respective feature maps and pose maps. Then, using the pose maps of the i-th and (i+1)-th frames, the reprojection error between them was calculated. This reprojection error was used to adjust the parameters of the neural network model until convergence, resulting in the trained monocular depth prediction neural network. The trained monocular depth prediction neural network has the function of predicting the depth map corresponding to the input image.

[0037] The above explains how a monocular depth prediction neural network is trained.

[0038] After obtaining the trained monocular depth prediction neural network, it can be pre-programmed onto the mobile robot, or optionally onto the processor. Images acquired by the monocular image acquisition component during the operation of the mobile robot are input into the trained monocular depth prediction neural network, which predicts and outputs the results. The output of the monocular depth prediction neural network is determined as the depth map corresponding to the image.

[0039] In the specific method of obtaining depth maps described above, images are acquired by a monocular image acquisition component during the operation of the mobile robot, and a monocular depth prediction neural network is introduced for prediction to obtain the depth map of the acquired images. Therefore, the purpose of obtaining depth maps can be achieved without using expensive depth cameras. Compared to directly capturing depth maps using a depth camera mounted on the mobile robot, the above method of obtaining depth maps can reduce the manufacturing cost of the mobile robot.

[0040] S102, perform a difference operation on the depth values ​​at the matching pixel positions in the depth map and the obtained depth template map to obtain a difference map; wherein, the depth template map is the depth map corresponding to the image collected by the mobile robot when it works on the reference plane.

[0041] This step S102 can be implemented in many ways. For example, as an embodiment, the Region of Interest (ROI) is extracted from the depth map, and the template ROI is extracted from the depth template map. The size of the extracted ROI is matched with that of the template ROI. The depth values ​​of the pixels in the ROI and the template ROI at the same coordinates are differentially calculated to obtain the difference map.

[0042] Here, the ROI region is the area in the depth map that contains the most information about the working ground, and the template ROI region is a region in the depth template map that matches the ROI region. For example, if the lower third of the area in the depth map is selected from the bottom up as the ROI region, the lower third of the area in the depth template map is also selected from the bottom up as the corresponding template ROI region.

[0043] As another embodiment, step S102 can also be implemented in other ways. For example, as an embodiment, the ROI region is the entire depth map, and correspondingly, the template ROI region is the entire depth template map. A difference operation is performed on the depth values ​​of pixels at the same coordinates in the entire depth map and the entire depth template map to obtain a difference map.

[0044] S103, based on the depth values ​​at each pixel location in the difference map, predict whether there is a fall risk area on the working path of the mobile robot; the fall risk area refers to the area where the mobile robot is at risk of falling.

[0045] Since the depth template image is the depth map corresponding to the image captured by the mobile robot on the reference plane, and the depth map of the image is the depth map corresponding to the image captured by the mobile robot on the working ground, by performing a difference operation on the depth values ​​at the matching pixel positions in the depth map and the depth map template, it is possible to reflect the height difference between the working ground and the reference ground based on the difference in depth values ​​between each pixel in the image captured by the mobile robot during its operation and each pixel on the reference plane. This allows for the prediction of whether there are fall risk areas on the working path of the mobile robot based on this height difference.

[0046] There are many ways to predict whether there are fall-risk areas on the working path of a mobile robot. Examples of specific implementation methods will be given later, and they will not be elaborated here.

[0047] S104. If there is a fall risk area on the working path, modify the working path of the mobile robot to control the mobile robot to avoid the fall risk area when working.

[0048] If there are drop-risk areas along the work path, these areas need to be accurately located. The specific steps for locating drop-risk areas will be described in detail later; they will not be elaborated upon here.

[0049] This concludes the process. Figure 1 The process is shown below.

[0050] pass Figure 1 The process shown involves performing a differential calculation on the depth values ​​at matching pixel locations in the depth map corresponding to the images captured by the mobile robot during its work process (referred to as the work surface) and the depth map template (the depth map corresponding to the images captured when the mobile robot is working on the reference plane). This allows for the determination of the height difference between the work surface and the reference surface based on the difference in depth values ​​between each pixel in the images captured by the mobile robot during its work process and each pixel on the reference plane. Based on this height difference, the system can predict whether there are fall risk areas on the mobile robot's work path. This enables effective prediction of the mobile robot's fall risk, thereby improving the fall prevention effect.

[0051] Furthermore, in this embodiment, when a fall-risk area is detected on the work path, the work path of the mobile robot is modified to control the mobile robot to avoid the fall-risk area and improve the fall prevention effect of the mobile robot.

[0052] The following is a detailed description of whether there are fall-risk areas along the predicted working path of the mobile robot:

[0053] See Figure 2 , Figure 2 This is a schematic diagram of a process for predicting whether there is a fall risk area on the working path of a mobile robot, provided as an exemplary embodiment of this application.

[0054] like Figure 2 As shown, the process may include the following steps:

[0055] S201, binarize the depth values ​​at each pixel location in the difference image to obtain a binarized image.

[0056] In this step, binarization can be implemented in many ways. For example, in one embodiment, since the preset depth threshold is based on the critical depth value that would cause the mobile robot to fall, it is used as the boundary value for binarization segmentation. If the depth value at a pixel location is greater than or equal to the preset depth threshold, that pixel location is marked as a first label, such as 1. If the depth value at a pixel location is less than the preset depth threshold, that pixel location is marked as a second label, such as 0.

[0057] The critical depth at which a mobile robot can fall is related to the size of its wheels; the larger the wheels, the greater the critical depth. Therefore, the preset depth threshold is positively correlated with the size of the mobile robot's wheels; the larger the wheels, the greater the preset depth threshold, and vice versa.

[0058] S202, determine at least one connected component in the binarized image, and determine whether there is a target connected component in the binarized image that satisfies the set conditions; the depth value at each pixel position in the target connected component is greater than or equal to a preset depth threshold, and the area of ​​the target connected component is greater than a preset area threshold.

[0059] In this step, the preset area threshold is set based on the area of ​​the critical zone that would cause the mobile robot to fall. The critical zone area is the maximum area of ​​a region with a sharp height change that the mobile robot can traverse on the working surface, taking into account that the mobile robot's traversing ability is related to the size of its wheels. Therefore, the preset area threshold is also positively correlated with the size of the mobile robot's wheels; the larger the mobile robot's wheels, the larger the range the mobile robot can traverse, and the larger the preset depth threshold, and vice versa.

[0060] If the depth value at every pixel location in a connected component is greater than a preset depth threshold, but the area of ​​the connected component is very small, it indicates that there is a certain height difference between the working ground corresponding to that pixel location and the reference plane, but the mobile robot can cross this height difference. Therefore, this connected component is not a target connected component. Only a connected component where the depth value at every pixel location is greater than or equal to a preset depth threshold, and its area is greater than a preset area threshold, is a target connected component. In other words, in a target connected component, each pixel location has the first label mentioned above, and the area of ​​the connected component is greater than a preset area threshold.

[0061] S203, if a target connected region exists in the binarized image, then it is determined that there is a fall risk area on the working path of the mobile robot.

[0062] Specifically, if a target connected component exists in the binarized image, then a fall risk region is determined to exist in the mobile robot's working path. This fall risk region is the physical area on the working ground corresponding to the target connected component. Correspondingly, if a target connected component exists in the binarized image, then multiple fall risk regions are determined to exist in the mobile robot's working path, and each of these multiple fall risk regions corresponds one-to-one with the target connected component.

[0063] The above provides a detailed description of whether there are areas with a risk of falling along the working path of the aforementioned mobile robot.

[0064] This concludes the process. Figure 2 The process is shown below.

[0065] pass Figure 2 The process shown can predict whether there are fall-risk areas on the working path of a mobile robot. When fall-risk areas exist, the working path of the mobile robot can be modified to control the mobile robot to avoid fall-risk areas and improve the fall prevention effect of the mobile robot.

[0066] The following describes in detail, with reference to specific examples, how to locate the fall risk area.

[0067] Figure 3 This is a schematic diagram of a process for locating a fall risk area, provided as an exemplary embodiment of this application.

[0068] like Figure 3 As shown, the process may include the following steps:

[0069] S301, obtain the positioning angle value and positioning distance value corresponding to the position of each pixel in the target connected domain.

[0070] In this step, the positioning angle value corresponding to any pixel position refers to the relative angle between the physical position on the working path corresponding to that pixel position and the geometric center position of the mobile robot, and the positioning distance value corresponding to any pixel position refers to the relative distance between the physical position on the working path corresponding to that pixel position and the geometric center position of the mobile robot.

[0071] Since the mobile robot pre-sets the positioning angle and distance values ​​corresponding to each pixel in the acquired image, and the binarized image is obtained by processing this image, each pixel in the binarized image also has a corresponding positioning angle and distance value. Therefore, it is possible to obtain the positioning angle and distance values ​​corresponding to each pixel in the target connected component from the positioning angle and distance values ​​corresponding to each pixel in the binarized image.

[0072] In an optional embodiment, the specific implementation of obtaining the positioning angle and positioning distance values ​​corresponding to the position of each pixel in the target connected component from the positioning angle and positioning distance values ​​corresponding to the position of each pixel in the binarized image can be as follows: the positioning angle and positioning distance values ​​corresponding to the position of each pixel in the binarized image are listed in a table, which lists the coordinates of each pixel and the positioning angle and positioning distance values ​​corresponding to the coordinates of that pixel. Based on the coordinates of each pixel in the binarized image, the positioning angle and positioning distance values ​​corresponding to the coordinates of each pixel in the target connected component are queried from the table.

[0073] S302, determine the location of the fall risk area based on the positioning angle value and positioning distance value corresponding to the position of each pixel in the target connected domain.

[0074] Specifically, since the positioning distance value corresponding to any pixel position can represent the relative distance between the physical position on the working path corresponding to that pixel position and the geometric center position of the mobile robot, the location of the fall risk area can be determined after obtaining the positioning angle value and positioning distance value of each pixel position in the target connected region.

[0075] This concludes the process. Figure 3 The process is shown below.

[0076] pass Figure 3 The process shown can accurately locate the fall risk area, and based on the determined location of the fall risk area, it can control the mobile robot to avoid the fall risk area, thereby reducing the probability of the mobile robot falling.

[0077] This concludes the description of the method provided in this exemplary embodiment. The following describes a mobile robot provided in an exemplary embodiment of this application:

[0078] See Figure 4 , Figure 4 This is a schematic diagram of the structure of a mobile robot provided as an exemplary embodiment of this application. Figure 4 As shown, the mobile robot includes a monocular image acquisition component and a processor.

[0079] A monocular image acquisition component is used to acquire images during the operation of a mobile robot.

[0080] The processor is used to obtain depth maps corresponding to images captured by the mobile robot during its operation.

[0081] The depth values ​​at matching pixel locations in the depth map and the obtained depth template map are differentially analyzed to obtain a difference map; where the depth template map is the depth map corresponding to the image captured by the mobile robot when it is working on the reference plane.

[0082] Based on the depth values ​​at each pixel location in the difference map, predict whether there are fall-risk areas on the mobile robot's working path; fall-risk areas refer to areas where the mobile robot is at risk of falling.

[0083] If there are areas with a risk of falling on the work path, modify the work path of the mobile robot to control the mobile robot to avoid the areas with a risk of falling on the work path.

[0084] As an example, obtaining the depth map corresponding to the images collected by the mobile robot during its operation includes:

[0085] Images captured by the mobile robot during its operation are input into a trained monocular depth prediction neural network. The monocular depth prediction neural network is used to output a depth map corresponding to the input image. The monocular depth prediction neural network is trained using video streams captured by a specified model of monocular image acquisition component as training samples. The specified model of monocular image acquisition component is the same model as the monocular image acquisition component used by the mobile robot to capture images during its operation.

[0086] Based on the output of the monocular depth prediction neural network, the depth map corresponding to the image collected by the mobile robot during its operation is determined.

[0087] As one example, a difference operation is performed on the depth values ​​at matching pixel locations in the depth map and the obtained depth template map to obtain a difference map, including:

[0088] Extracting the Region of Interest (ROI) from the depth map;

[0089] Extract the template ROI region from the depth template map. The extracted ROI region is matched with the size of the template ROI region.

[0090] The depth values ​​at matching pixel locations in the ROI region and the template ROI region are differentially analyzed to obtain a difference map.

[0091] As an example, based on the depth values ​​at each pixel location in the difference map, predicting whether there are fall-risk areas on the mobile robot's working path includes:

[0092] The depth values ​​at each pixel location in the difference image are binarized to obtain a binarized image.

[0093] In a binarized image, at least one connected component is identified, and it is determined whether a target connected component exists in the binarized image that satisfies the set conditions. The depth value at each pixel position in the target connected component is greater than or equal to a preset depth threshold, and the area of ​​the target connected component is greater than a preset area threshold. The preset depth threshold and the preset area threshold are set based on the critical depth value and the critical region area that cause the mobile robot to fall, respectively.

[0094] If a target connected region exists in the binarized image, then a fall risk area is determined to exist on the working path of the mobile robot.

[0095] As an example, the fall risk area is located through the following steps:

[0096] Obtain the positioning angle value and positioning distance value corresponding to each pixel position in the target connected component; wherein, the positioning angle value corresponding to any pixel position refers to the relative angle between the physical position on the working path corresponding to the pixel position and the geometric center position of the mobile robot, and the positioning distance value corresponding to any pixel position refers to the relative distance between the physical position on the working path corresponding to the pixel position and the geometric center position of the mobile robot.

[0097] Based on the positioning angle and positioning distance values ​​corresponding to the position of each pixel in the target connected domain, the location of the fall risk area is determined.

[0098] As an example, the preset depth threshold and the preset area threshold are positively correlated with the size of the mobile robot's wheels.

[0099] For the mobile robot embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0100] Please see Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 501, a communication interface 502, a memory 503, and a communication bus 504. The processor 501, communication interface 502, and memory 503 communicate with each other via the communication bus 504. The memory 503 stores a computer program; the processor 501 can execute the steps of the anti-fall method for the mobile robot described in the above embodiment by executing the program stored in the memory 503. Depending on the actual function of the electronic device, other hardware may also be included, which will not be elaborated further.

[0101] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0102] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0103] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0104] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0105] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe the features of specific embodiments of the specified invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0106] Similarly, although the operations are depicted in a specified order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specified order or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0107] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily in the specified order or sequence shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0108] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for preventing falls from mobile robots, characterized in that, include: Obtain the depth map corresponding to the images collected by the mobile robot during its operation; A difference map is obtained by performing a difference operation on the depth values ​​at the matching pixel positions in the depth map and the obtained depth template map; wherein, the depth template map is the depth map corresponding to the image collected by the mobile robot when it is working on the reference plane; Based on the depth values ​​at each pixel location in the difference map, as well as the critical depth value and critical area area that would cause the mobile robot to fall, it is predicted whether there is a fall risk area on the working path of the mobile robot; wherein, the critical area area is the maximum value of the area of ​​a region with a sharp height change that the mobile robot can cross on the working ground; the fall risk area refers to the area where the mobile robot is at risk of falling. If the fall risk area exists on the work path, the work path of the mobile robot is modified to control the mobile robot to avoid the fall risk area when working.

2. The method according to claim 1, characterized in that, A monocular image acquisition component is installed at a designated location on the mobile robot; The images acquired by the mobile robot during its operation are acquired through the monocular image acquisition component.

3. The method according to claim 1, characterized in that, The depth map obtained from the images captured by the mobile robot during its operation includes: The images captured by the mobile robot during its operation are input into a trained monocular depth prediction neural network; the monocular depth prediction neural network is used to output a depth map corresponding to the input image; the monocular depth prediction neural network is trained using video streams captured by a specified model of monocular image acquisition component as training samples, and the specified model of monocular image acquisition component is the same model as the monocular image acquisition component used by the mobile robot to capture images during its operation. Based on the output of the monocular depth prediction neural network, the depth map corresponding to the image collected by the mobile robot during its operation is determined.

4. The method according to claim 1, characterized in that, The step of performing a difference operation on the depth values ​​at the matching pixel positions in the depth map and the obtained depth template map to obtain a difference map includes: Extract the template ROI region from the depth template map; The region of interest (ROI) is extracted from the depth map, and the extracted ROI is matched with the size of the template ROI. The difference map is obtained by performing a difference operation on the depth values ​​at the matching pixel positions in the ROI region and the template ROI region.

5. The method according to claim 1, characterized in that, The step of predicting whether there is a fall risk area on the working path of the mobile robot based on the depth value at each pixel position in the difference map, as well as the critical depth value and critical area area that would cause the mobile robot to fall, includes: The depth values ​​at each pixel location in the difference image are binarized to obtain a binarized image; In the binarized image, at least one connected component is determined, and it is determined whether there is a target connected component in the binarized image that satisfies a set condition; the depth value at each pixel position in the target connected component is greater than or equal to a preset depth threshold, and the area of ​​the target connected component is greater than a preset area threshold; the preset depth threshold and the preset area threshold are set based on the critical depth value and critical region area that cause the mobile robot to fall, respectively. If the target connected component exists in the binarized image, then it is determined that there is a fall risk area on the working path of the mobile robot.

6. The method according to claim 5, characterized in that, The fall risk area is located through the following steps: Obtain the positioning angle value and positioning distance value corresponding to each pixel position in the target connected domain; wherein, the positioning angle value corresponding to any pixel position refers to the relative angle between the physical position on the working path corresponding to the pixel position and the geometric center position of the mobile robot, and the positioning distance value corresponding to any pixel position refers to the relative distance between the physical position on the working path corresponding to the pixel position and the geometric center position of the mobile robot. The location of the fall risk area is determined based on the positioning angle and positioning distance values ​​corresponding to the position of each pixel in the target connected region.

7. The method according to claim 5, characterized in that, The preset depth threshold and the preset area threshold are positively correlated with the size of the mobile robot's wheels.

8. A mobile robot, characterized in that, include: Monocular image acquisition components and processor; The monocular image acquisition component is used to acquire images during the operation of the mobile robot; The processor is used to obtain the depth map corresponding to the image collected by the mobile robot during its operation; A difference map is obtained by performing a difference operation on the depth values ​​at the matching pixel positions in the depth map and the obtained depth template map; wherein, the depth template map is the depth map corresponding to the image collected by the mobile robot when it is working on the reference plane; Based on the depth values ​​at each pixel location in the difference map, as well as the critical depth value and critical area area that would cause the mobile robot to fall, it is predicted whether there is a fall risk area on the working path of the mobile robot; wherein, the critical area area is the maximum value of the area of ​​a region with a sharp height change that the mobile robot can cross on the working ground; the fall risk area refers to the area where the mobile robot is at risk of falling. If the fall risk area exists on the work path, the work path of the mobile robot is modified to control the mobile robot to avoid the fall risk area when working.

9. The mobile robot according to claim 8, characterized in that, The depth map obtained from the images captured by the mobile robot during its operation includes: Images captured by the mobile robot during its operation are input into a trained monocular depth prediction neural network; the monocular depth prediction neural network outputs a depth map corresponding to the input image, and is trained using video streams captured by a specified model of monocular image acquisition component as training samples. The specified model of monocular image acquisition component is the same model as the monocular image acquisition component used by the mobile robot to capture images during its operation; and / or, The step of performing a difference operation on the depth values ​​at the matching pixel positions in the depth map and the obtained depth template map to obtain a difference map includes: Extract the template ROI region from the depth template map; The region of interest (ROI) is extracted from the depth map, and the extracted ROI is matched with the size of the template ROI. Perform a difference operation on the depth values ​​at matching pixel locations in the ROI region and the template ROI region to obtain the difference map; and / or, The step of predicting whether there is a fall risk area on the working path of the mobile robot based on the depth value at each pixel position in the difference map, as well as the critical depth value and critical area area that would cause the mobile robot to fall, includes: The depth values ​​at each pixel location in the difference image are binarized to obtain a binarized image; In the binarized image, at least one connected component is determined, and it is determined whether there is a target connected component in the binarized image that satisfies a set condition; the depth value at each pixel position in the target connected component is greater than or equal to a preset depth threshold, and the area of ​​the target connected component is greater than a preset area threshold; the preset depth threshold and the preset area threshold are set based on the critical depth value and critical region area that cause the mobile robot to fall, respectively. If the target connected component exists in the binarized image, then a fall risk area is determined to exist on the working path of the mobile robot; and / or, The fall risk area is located through the following steps: Obtain the positioning angle value and positioning distance value corresponding to each pixel position in the target connected domain; wherein, the positioning angle value corresponding to any pixel position refers to the relative angle between the physical position on the working path corresponding to the pixel position and the geometric center position of the mobile robot, and the positioning distance value corresponding to any pixel position refers to the relative distance between the physical position on the working path corresponding to the pixel position and the geometric center position of the mobile robot. Based on the positioning angle and positioning distance values ​​corresponding to the positions of each pixel in the target connected component, the orientation of the fall risk area is determined; and / or, The preset depth threshold and the preset area threshold are positively correlated with the size of the mobile robot's wheels.

10. An electronic device, characterized in that, include: processor; as well as A memory storing computer program instructions that, when executed by the processor, cause the processor to perform the steps of the method as described in any one of claims 1 to 7.

Images