Obstacle avoidance method for self-moving cleaning robot and self-moving cleaning robot

Abstract

The application discloses a self-moving cleaning robot obstacle avoidance method and a self-moving cleaning robot in the field of computer technology. The application detects image information of the robot advancing direction through two different image recognition methods, so that the information of the same obstacle in the image information is mutually supplemented and fused, then the image fusion information of the same obstacle obtained by supplementing and fusing is comprehensively combined with the fourth obstacle information detected by the line laser sensor again, so that an obstacle map of the area where the self-moving cleaning robot is located is generated, and finally the self-moving cleaning robot is controlled to avoid obstacles according to the obstacle map, so that the robot can realize edge obstacle avoidance for various obstacles. The scheme does not depend on the recognition of the geometric characteristics and categories of the objects, can have good detection effect on various types of objects, has high detection coverage, and can make the robot accurately and effectively avoid obstacles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and particularly relates to a self-moving cleaning robot obstacle avoidance method and a self-moving cleaning robot. BACKGROUND

[0002] Generally, the self-moving cleaning robot can adopt a double-line laser radar, a single-line laser radar, an infrared sensor, an ultrasonic sensor and the like to realize obstacle avoidance; such a scheme judges whether there is an obstacle in front of the self-moving cleaning robot through ranging signals, laser scanning or reflected wave intensity, so as to control the robot to detour. Of course, the self-moving cleaning robot can also adopt a depth camera or a stereo vision camera to shoot the field of view in front of the robot, and realize obstacle avoidance and detour of the self-moving cleaning robot by detecting the target category such as shoes, chair legs and pets from the shot image; but such an image recognition scheme depends on the geometric features of the objects, and has poor recognition ability for transparent objects, black floor mats and low-height sundries (such as electric wires and socks), and is difficult to adapt to complex home environments and multi-target scenes, and is also prone to misjudgment in the case of occlusion, overlap or complex lighting, so that the robot is difficult to accurately avoid obstacles.

[0003] Therefore, how to make the self-moving cleaning robot accurately and effectively avoid obstacles is a problem to be solved by those skilled in the art. SUMMARY

[0004] Therefore, how to make the self-moving cleaning robot accurately and effectively avoid obstacles is a problem to be solved by those skilled in the art.

[0005] In a first aspect, the present application provides an obstacle avoidance method for a self-moving cleaning robot. The self-moving cleaning robot comprises a main body. The front side of the main body along the direction of travel is provided with a line laser sensor and an image acquisition assembly. The line laser sensor is configured to emit a first line laser beam in a horizontal direction and a second line laser beam inclined downward relative to the horizontal plane. The obstacle avoidance method comprises: acquiring image information of the direction of travel of the self-moving cleaning robot by the image acquisition assembly; detecting a plurality of first obstacle information from the image information by an obstacle category detection method; the first obstacle information comprises an initial obstacle category and an obstacle detection frame; detecting a plurality of second obstacle information from the image information by an obstacle contour detection method; the second obstacle information comprises an obstacle contour; fusing the first obstacle information and the second obstacle information located at the same position in the image information to determine third obstacle information at the position; the third obstacle information comprises an obstacle category and an obstacle boundary; acquiring fourth obstacle information of the direction of travel of the self-moving cleaning robot by the line laser sensor; generating an obstacle map of the area where the self-moving cleaning robot is located based on the third obstacle information and the fourth obstacle information; and controlling the self-moving cleaning robot to avoid obstacles according to the obstacle map.

[0006] Optionally, fusing the first obstacle information and the second obstacle information located at the same position in the image information to determine the third obstacle information at the position comprises: fusing the first obstacle information and the second obstacle information located at the same position in the image information, and performing the following steps on the fused information: if the fused information comprises an obstacle contour, determining an obstacle boundary based on the obstacle contour; if the fused information only comprises an obstacle detection frame, determining an obstacle boundary based on the obstacle detection frame; and if the fused information comprises an obstacle contour and an obstacle detection frame, determining an obstacle category based on the intersection-over-union of the obstacle contour and the obstacle detection frame. In this way, the information of the same obstacle in the same image information obtained by the two image recognition methods (i.e., the first obstacle information detected by the obstacle category detection method and the second obstacle information detected by the obstacle contour detection method) can be supplemented and fused with each other, improving the detection coverage and facilitating accurate and effective obstacle avoidance by the robot.

[0007] Optionally, determining the obstacle category based on the intersection-over-union of the obstacle contour and the obstacle bounding box comprises: if the intersection-over-union is greater than a preset intersection-over-union threshold, taking the initial obstacle category as the obstacle category; otherwise, determining the obstacle category as a general obstacle. This way uses the intersection-over-union to measure the coincidence degree of the obstacle contour and the obstacle bounding box. When the coincidence degree is high (i.e. the intersection-over-union is greater than the preset intersection-over-union threshold), it is considered that the initial obstacle category output by the obstacle category detection method is relatively accurate, so the initial obstacle category is taken as the obstacle category. When the coincidence degree is low (i.e. the intersection-over-union is not greater than the preset intersection-over-union threshold), it is considered that the initial obstacle category output by the obstacle category detection method is not highly confident, so the obstacle category is determined as a general obstacle. Here, it is set that the confidence of the first obstacle information detected by the obstacle category detection method is lower than the confidence of the second obstacle information detected by the obstacle contour detection method.

[0008] Optionally, generating the obstacle map of the area where the self-moving cleaning robot is located based on the third obstacle information and the fourth obstacle information comprises: for the target obstacle on the carpet, inserting the target obstacle into the instant positioning map of the self-moving cleaning robot according to the obstacle boundary in the third obstacle information to obtain the obstacle map; for the object obstacle on the non-carpet or the low obstacle determined based on the second line laser beam, merging the obstacle frame in the fourth obstacle information with the obstacle boundary in the third obstacle information to obtain a merged area; and inserting the object obstacle or the low obstacle into the instant positioning map of the self-moving cleaning robot according to the merged area to obtain the obstacle map. This way distinguishes the target obstacle on the carpet and the object obstacle on the non-carpet. Due to the interference of the carpet, the accuracy of the fourth obstacle information determined by the second line laser beam may not be high. Fusing such low-accuracy information to the third obstacle information will not only affect the accuracy of the third obstacle information, but also may further cause inaccurate obstacle recognition and inaccurate obstacle-avoiding trajectory. Therefore, for the target obstacle on the carpet, only the target obstacle is inserted into the instant positioning map of the self-moving cleaning robot according to the obstacle boundary in the third obstacle information to ensure the accuracy of obstacle recognition and obstacle-avoiding trajectory. On the other hand, for the object obstacle on the non-carpet or the low obstacle determined based on the second line laser beam, fusing the fourth obstacle information and the third obstacle information can ensure the comprehensiveness and accuracy of the obstacle information, thereby improving the accuracy of obstacle recognition and obstacle-avoiding trajectory. Therefore, for the low obstacle or the object obstacle on the non-carpet, the obstacle frame in the fourth obstacle information is merged with the obstacle boundary in the third obstacle information to obtain a merged area (i.e. the area enclosed by the obstacle boundary and the maximum coverage range after merging the obstacle frame); and the object obstacle or the low obstacle is inserted into the instant positioning map of the self-moving cleaning robot according to the merged area.

[0009] Optionally, the side surface of the body main body is further provided with at least one edge sensor; and correspondingly, the method of controlling the self-moving cleaning robot to avoid obstacles according to the obstacle map comprises: for a non-low obstacle determined based on the first line laser beam, determining an equidistant curve outside the obstacle in the obstacle map based on a safety distance corresponding to the obstacle category; planning an edge travel trajectory for avoiding the obstacle based on the equidistant curve; and controlling the self-moving cleaning robot to move according to the edge travel trajectory, and controlling the travel direction of the self-moving cleaning robot to avoid the obstacle according to the distance data collected by the edge sensor and the first line laser beam during the movement. Since the non-low obstacle can not be captured by the image acquisition assembly, in order to realize accurate edge obstacle avoidance, the at least one edge sensor on the side surface of the body main body can be used to optimize and adjust the obstacle avoidance trajectory in real time during the obstacle avoidance process.

[0010] Optionally, the method of controlling the self-moving cleaning robot to avoid obstacles according to the obstacle map comprises: for a low obstacle determined based on the second line laser beam, determining an equidistant curve outside the obstacle in the obstacle map based on a safety distance corresponding to the obstacle category; planning an edge travel trajectory for avoiding the obstacle based on the equidistant curve; and controlling the self-moving cleaning robot to move according to the edge travel trajectory, and controlling the travel direction of the self-moving cleaning robot to avoid the obstacle according to the first line laser beam during the movement.

[0011] Optionally, the method further comprises: during the process of avoiding obstacles by the self-moving cleaning robot, judging whether the self-moving cleaning robot has moved to the end of the obstacle boundary; if yes, controlling the self-moving cleaning robot to retreat in the case that the obstacle does not fall within the range of the image acquisition assembly; in the case that the self-moving cleaning robot retreats to the range of the image acquisition assembly, acquiring the latest image information of the travel direction of the self-moving cleaning robot through the image acquisition assembly; determining contour completion information based on the latest image information; filling the contour completion information to the obstacle map to obtain a latest obstacle map; and controlling the self-moving cleaning robot to avoid obstacles according to the latest obstacle map, and performing the step of judging whether the self-moving cleaning robot has moved to the end of the obstacle boundary and the subsequent other steps during the process of avoiding obstacles by the self-moving cleaning robot until the obstacle contour in the latest obstacle map forms a closed loop, thereby supplementing the field of view of the robot in real time and optimizing and adjusting the travel trajectory of the robot accordingly.

[0012] Optionally, the contour completion information is determined based on the latest image information, including: detecting a plurality of first latest obstacle information from the latest image information by an obstacle category detection method; detecting a plurality of second latest obstacle information from the latest image information by an obstacle contour detection method; fusing the first latest obstacle information and the second latest obstacle information located at the same position of the image information to determine third latest obstacle information at the position; obtaining fourth latest obstacle information from a direction of travel of the mobile cleaning robot by a line laser sensor; and determining the contour completion information based on the third latest obstacle information and the fourth latest obstacle information.

[0013] Optionally, the edge-following trajectory for avoiding the obstacle is planned based on the equidistant curve, including: obtaining a historical obstacle map; and planning the edge-following trajectory based on the equidistant curve and a historical travel trajectory in the historical obstacle map, thereby optimizing the current obstacle-avoiding trajectory in combination with historical obstacle-avoiding data.

[0014] In a second aspect, the present application provides a self-moving cleaning robot, including: a memory for storing a computer program; and a processor for executing the computer program to implement the self-moving cleaning robot obstacle-avoiding method disclosed above.

[0015] In a third aspect, the present application provides a non-volatile storage medium for storing a computer program, wherein the computer program is executed by a processor to implement the self-moving cleaning robot obstacle-avoiding method disclosed above.

[0016] In a fourth aspect, the present application provides a computer program product, including a computer program / instruction, which is executed by a processor to implement the steps of the self-moving cleaning robot obstacle-avoiding method disclosed above.

[0017] From the above solutions, the present application has the following technical effects:

[0018] (1) For the same image information of the robot travel direction collected by the image acquisition assembly, two different image recognition methods are used for detection: A, a plurality of first obstacle information is detected from the image information by an obstacle category detection method to identify the obstacle category in the image information, which is conducive to accurate obstacle-avoiding and detouring for specific objects such as shoes, chair legs, and pets; B, a plurality of second obstacle information is detected from the image information by an obstacle contour detection method to identify the general obstacle contour without category distinction in the image information, which reduces the dependence on the recognition of the geometric features and categories of objects, and can have certain recognition ability for transparent objects, black floor mats, and low-lying objects (such as wires, socks, etc.).

[0019] (2) The information detected by the above two different image recognition methods is matched and fused, so that the information of the same obstacle in the image information is mutually complementary, that is, the first obstacle information and the second obstacle information located at the same position in the image information are fused to determine the third obstacle information of the same obstacle based on the image information (i.e., the fusion information of the same obstacle determined by the two image recognition methods), which improves the comprehensiveness of the obstacle information extraction and the detection coverage, and is conducive to adapting to complex home environments, multi-target scenes, and scenes with occlusion, overlap, or complex lighting.

[0020] (3) Since the main body of the self-moving cleaning robot is provided with a line laser sensor on the front side thereof along the advancing direction, the line laser sensor can emit a first line laser beam in the horizontal direction and a second line laser beam downwardly inclined relative to the horizontal plane, the fourth obstacle information of the advancing direction of the self-moving cleaning robot (i.e., the obstacle information determined by the distance information measured by the line laser sensor) can also be obtained by the line laser sensor, and then the fourth obstacle information and the third obstacle information are fused again to generate an obstacle map of the area where the self-moving cleaning robot is located, so that the obstacles in the obstacle map are based on more comprehensive information such as the obstacle information determined based on the image and the obstacle information determined based on the line laser sensor, and finally the self-moving cleaning robot is controlled to avoid obstacles according to the obstacle map, which can realize accurate edge avoidance of various obstacles by the robot, and also can reduce the cleaning blind area and realize cleaning of the edges of the obstacles as much as possible.

[0021] In summary, the present application does not rely on the identification of the geometric features and categories of the objects, and can have good detection effect on various types of objects. The obstacle detection result information of the above two different image recognition methods and the obstacle detection result based on the line laser sensor can be mutually supplemented and fused, the detection coverage is improved through two information fusion steps, the robot can accurately and effectively avoid obstacles, the obstacle avoidance effect is improved, and the possibility of obstacle avoidance is increased.

[0022] Correspondingly, the self-moving cleaning robot obstacle avoidance device, equipment, medium and program product provided by the present application also have the above technical effects. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0024] Figure 1This is a schematic diagram of a self-moving cleaning robot disclosed in this application;

[0025] Figure 2 This is a schematic diagram of a line laser sensor disclosed in this application;

[0026] Figure 3 This is a flowchart of an obstacle avoidance method for a self-moving cleaning robot disclosed in this application;

[0027] Figure 4 This is a schematic diagram of a self-moving cleaning robot that avoids obstacles along the edge, as disclosed in this application.

[0028] Figure 5 In accordance with Figure 4 A schematic diagram of the obstacle map obtained after bypassing the obstacle;

[0029] Figure 6 This is a schematic diagram of image information and corresponding region segmentation results disclosed in this application;

[0030] Figure 7 This application discloses another obstacle avoidance method for a self-propelled sweeping cleaning robot.

[0031] Figure 8 A server architecture diagram provided for this application;

[0032] Figure 9 A terminal structure diagram provided for this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0034] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0035] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Generally, the self-moving cleaning robot comprises a body, a controller, one or more cleaning components, a self-moving component, and a related sensor system. The self-moving component of the self-moving cleaning robot can comprise universal wheels, travel wheels, auxiliary wheels, etc.

[0037] It should be noted that the shape of the body can be circular, square, or other shapes. For example, one part of the body can be circular, and the other part can be square. The controller can comprise a microcontroller unit (MCU). Of course, the controller can also comprise other devices that can have control functions.

[0038] In an example, the cleaning components can specifically comprise side brushes, main brushes (or roll brushes), mop plates, etc., which can have circular, square, or other shapes (such as semi-circular, arc-shaped, triangular, and other special shapes). The circular shape facilitates the rotation of the cleaning components. The special shape facilitates the cleaning of the corner areas. The side brushes can gather foreign matter and move it towards the center of the bottom of the self-moving cleaning robot. The roll brushes can sweep up the foreign matter at the bottom of the self-moving cleaning robot and make it enter the dust collection box through the suction port. The mop plate is used for mopping or wiping the floor, and can be a disc mop, a roller mop, and a flat plate mop, etc. A water tank can be provided on the self-moving cleaning robot. The water in the water tank flows through the holes to the mop to wet it. The wet mop is used for wiping the floor. The main brush is arranged in the main brush cavity at the bottom of the body of the self-moving cleaning robot. The main brush cavity is in communication with the suction channel of the self-moving cleaning robot. The small-sized garbage such as dust and hair swept up by the main brush and / or the side brush can be sucked into the self-moving cleaning robot through the main brush cavity.

[0039] In this application, the related sensor system can comprise a line laser sensor arranged on the front side of the body along the travel direction and an image acquisition assembly, which can be integrated with the line laser sensor. The image acquisition assembly can comprise a camera and a fill light. The line laser sensor is used to emit a first line laser beam emitted in the horizontal direction and a second line laser beam emitted downwardly inclined relative to the horizontal plane. Of course, the related sensor system can also comprise a plane laser sensor, an LDS sensor, a Dtof sensor, an Itof sensor, etc.

[0040] And in this application, the controller of the self-moving cleaning robot can execute the self-moving cleaning robot obstacle avoidance method provided in this application to make the self-moving cleaning robot effectively avoid obstacles. In the obstacle avoidance process, the self-moving cleaning robot performs at least one of the following obstacle avoidance actions: the self-moving cleaning robot turns around at the edge of the obstacle; the self-moving cleaning robot moves along the edge of the obstacle; the self-moving cleaning robot stops at the edge of the obstacle and then moves in a turning direction; the self-moving cleaning robot stops at the edge of the obstacle, then backs up, and then moves in a turning direction; the self-moving cleaning robot cleans along the edge of the obstacle. For example, if a bottle-shaped obstacle is detected during the self-moving cleaning robot's travel, the self-moving cleaning robot can turn around at the edge of the bottle-shaped obstacle to avoid the bottle-shaped obstacle boundary for cleaning. For another example, if a bottle-shaped obstacle is detected during the self-moving cleaning robot's travel, the self-moving cleaning robot can move along the edge of the bottle-shaped obstacle and / or clean along the edge, and then go to clean a place farther away from the bottle-shaped obstacle. For another example, if a bottle-shaped obstacle is detected during the self-moving cleaning robot's travel, the self-moving cleaning robot can first stop, then turn left or right, and then go to clean a place farther away from the bottle-shaped obstacle. For another example, if a bottle-shaped obstacle is detected during the self-moving cleaning robot's travel, the self-moving cleaning robot can stop at the edge of the bottle-shaped obstacle, then back up, then turn left or right, and then go to clean a place farther away from the bottle-shaped obstacle.

[0041] Referring to Figure 1 As shown in the drawings, the embodiment of the present application discloses a self-moving cleaning robot, which comprises a body main body; the front side of the body main body along the travel direction is provided with a line laser sensor and an image acquisition assembly, which can include a camera and a fill light. The line laser sensor is used to emit: a first line laser beam emitted along the horizontal direction and a second line laser beam emitted downwardly inclined relative to the horizontal plane. The line laser sensor and the line laser sensor can be integrated together.

[0042] In this embodiment, the line laser sensor comprises a base and a mirror assembly; the base is fixedly provided with a signal emitting module and a signal receiving module; the mirror assembly is rotatably arranged on the base and comprises a signal reflecting part, which comprises a first signal reflecting part and a second signal reflecting part.

[0043] Referring to Figure 2The signal emitting module is configured to emit: a first linear laser beam emitted in a horizontal direction and a second linear laser beam emitted downwardly inclined relative to the horizontal plane, the first linear laser beam corresponding to the first signal reflection part, and the second linear laser beam corresponding to the second signal reflection part. That is, the signal emitting module comprises a first emitting module and a second emitting module, the first emitting module emits the first linear laser beam, and the second emitting module emits the second linear laser beam. The signal emitting module and the mirror assembly are configured such that the first signal reflection part can reflect the first linear laser beam in the horizontal direction, and the second signal reflection part can reflect the second linear laser beam downwardly inclined relative to the horizontal plane.

[0044] The signal emitting module is configured to emit: a first linear laser beam emitted in a horizontal direction and a second linear laser beam emitted downwardly inclined relative to the horizontal plane, the first linear laser beam corresponding to the first signal reflection part, and the second linear laser beam corresponding to the second signal reflection part. That is, the signal emitting module comprises a first emitting module and a second emitting module, the first emitting module emits the first linear laser beam, and the second emitting module emits the second linear laser beam. The signal emitting module and the mirror assembly are configured such that the first signal reflection part can reflect the first linear laser beam in the horizontal direction, and the second signal reflection part can reflect the second linear laser beam downwardly inclined relative to the horizontal plane.

[0045] Referring to Figure 3 The application further provides an obstacle avoidance method for a self-moving cleaning robot, comprising:

[0046] S301, acquiring image information of a moving direction of the self-moving cleaning robot by using an image acquisition assembly.

[0047] In this embodiment, the image acquisition assembly can comprise a camera, which can be an RGB camera, a depth camera, a stereo vision camera, a camera with an adaptive exposure mode, etc. Further, at least one fill light can be arranged beside the camera to provide fill light for image acquisition when the environment is dark, so that the image acquisition assembly further comprises the fill light. In addition, the moving direction of the robot can be any direction such as front, back, left, right, etc.

[0048] In order to ensure the clarity of the image information, the brightness and the like of the image information can be detected and controlled. In an embodiment, an environment image of the robot running direction is captured by using the camera; the brightness information of the environment image is detected; if the brightness information is not lower than a preset brightness threshold, the environment image is determined as the image information, that is, it is considered that the current environment image can be used for normal execution of the subsequent detection steps; if the brightness information is lower than the preset brightness threshold, it is considered that the current environment image cannot be used for normal execution of the subsequent detection steps, the clarity of the image information is not enough, therefore the light supplement lamp of the robot running direction is turned on and / or the camera is controlled to be in the adaptive exposure mode, and the step of capturing the environment image of the robot running direction by using the camera and other subsequent steps are executed to re-shoot and capture new image information. In this way, the clarity and accuracy of the detected image information can be ensured, effective data support can be provided for the subsequent detection steps, and unnecessary detection process on the blurred invalid image can be avoided, the device computing power consumption is reduced, and the device resources are saved.

[0049] S302, detecting a plurality of first obstacle information from the image information by using the obstacle category detection method; the first obstacle information includes an obstacle initial category and an obstacle detection frame.

[0050] In this embodiment, the obstacle category detection method can be any type of target detection method, such as a template matching method and a target detection model trained based on deep learning, etc. After obtaining the relevant object image to be detected, the template matching method can compare it with the existing database, and find the target in the image and determine its coordinate position by calculating the mean, gradient, distance, variance and other characteristics of the image. The deep learning method can train a corresponding target detection model to identify the type and position of the object in the image.

[0051] In an embodiment, the target category (i.e. the obstacle initial category) and the detection frame (i.e. the obstacle detection frame) of a plurality of targets are detected from the image information by using the obstacle category detection method; a plurality of targets are determined as a plurality of first obstacles, and the target category and the detection frame of the plurality of targets are determined as a plurality of first obstacle information. In this way, the image information is detected by using the obstacle category detection method to determine the category and the corresponding target position (represented by the coordinate position of the detection frame) of each target (i.e. obstacle) in the image information, so that the specific object category and object position existing in the image information can be determined.

[0052] In another embodiment, in order to improve the clarity of the image information, image optimization operation can also be performed on the image information; the image optimization operation includes at least one of denoising, brightness enhancement and white balance processing, so that the image with higher clarity and accuracy can be obtained, and effective data support can be provided for the subsequent detection steps.

[0053] S303, detecting a plurality of second obstacle information from the image information by using the obstacle contour detection method; the second obstacle information comprises an obstacle contour.

[0054] In this embodiment, the obstacle contour detection method can also use deep learning method, machine learning method, etc. For example, a detection model capable of identifying the obstacle contour in the image is trained by using deep learning method or machine learning method, and the model is used to detect a plurality of second obstacle information from the image information.

[0055] In one embodiment, the obstacle contour detection method is used to detect the obstacle contour of a plurality of obstacles from the image information; the plurality of obstacles are determined as a plurality of second obstacles, and the obstacle contour of the plurality of obstacles is determined as a plurality of second obstacle information. Thus, the obstacle contour detection method is used to detect the obstacles in the image information to determine the obstacle contour (which can be represented by a mask) of each obstacle in the image information, and the contour and position of each obstacle existing in the image information can be determined without distinguishing the obstacle type, that is, without distinguishing the specific object of the obstacle, which is a general obstacle detection method.

[0056] In one example, the obstacle contour can also be subjected to a morphology optimization operation; the morphology optimization operation comprises at least one of erosion, dilation, hole filling, boundary smoothing and contour strengthening. The erosion, dilation, boundary smoothing and contour strengthening can remove the region edge burrs and small false foreground, and can also restore the details lost due to erosion, so that the region edge and the bounding box edge are closer to the real contour; the hole filling operation can fill the small holes in the foreground to make the regions connected. Thus, the morphology optimization operation can make the obtained obstacle contour connected and the boundary clear and smooth.

[0057] S304, fusing the first obstacle information and the second obstacle information located at the same position in the image information to determine third obstacle information of the position; the third obstacle information comprises an obstacle category and an obstacle boundary.

[0058] In this embodiment, the third obstacle information comprises an obstacle category and an obstacle boundary. The obstacle category can be detected by using the obstacle category detection method, or can be determined according to the following process: calculating the intersection-over-union of the detection box in the first obstacle information and the obstacle box in the second obstacle information matched with the detection box, if the intersection-over-union is greater than a preset intersection-over-union threshold, taking the obstacle category in the first obstacle information as the obstacle category included in the third obstacle information; if the intersection-over-union is not greater than the preset intersection-over-union threshold, determining the obstacle category included in the third obstacle information as a general obstacle. The obstacle category detected by the obstacle category detection method can be a specific category of the object, or other unknown category.

[0059] It should be noted that the matching of the first obstacle information and the second obstacle information at the same position in the image information is essentially the mutual complementation and fusion of the two information obtained by the two detection methods for the same obstacle. Specifically, the following detection results can be coped with:

[0060] (1) If the obstacle category detection method detects that the image information includes a stool, a mobile phone and a pet; the obstacle contour detection method detects that the image information includes obstacle 1, obstacle 2 and obstacle 3; and through position comparison, it can be confirmed that the stool is obstacle 1, the mobile phone is obstacle 2, and the pet is obstacle 3, then for the stool, the corresponding third obstacle information includes: obstacle category-stool, obstacle boundary (detected by the obstacle contour detection method); accordingly, then for the mobile phone, the corresponding third obstacle information includes: obstacle category-mobile phone, obstacle boundary (detected by the obstacle contour detection method); and so on for the pet, which will not be described herein.

[0061] (2) If the obstacle category detection method detects that the image information includes a stool, a mobile phone and a pet; the obstacle contour detection method detects that the image information includes obstacle 1 and obstacle 2; and through position comparison, it can be confirmed that the stool is obstacle 1, the mobile phone is obstacle 2, and the pet is not detected by the obstacle contour detection method, then for the stool, the corresponding third obstacle information includes: obstacle category-stool, obstacle boundary (detected by the obstacle contour detection method); accordingly, then for the mobile phone, the corresponding third obstacle information includes: obstacle category-mobile phone, obstacle boundary (detected by the obstacle contour detection method); in this example, for the pet, the corresponding third obstacle information includes: obstacle category-pet, obstacle boundary (determined by the detection box of the pet detected by the obstacle category detection method). When the obstacle contour detection method does not detect the obstacle contour, the corresponding detection box detected by the obstacle category detection method is used to determine the obstacle contour.

[0062] (3) If the obstacle category detection method detects that the image information includes: a stool and a mobile phone; the obstacle contour detection method detects that the image information includes: obstacle 1, obstacle 2 and obstacle 3; and through the position comparison, it can be confirmed that the stool is obstacle 1, the mobile phone is obstacle 2, and obstacle 3 is not detected by the obstacle category detection method, then for the stool, the corresponding third obstacle information includes: obstacle category-stool, obstacle boundary (detected by the obstacle contour detection method); accordingly, then for the mobile phone, the corresponding third obstacle information includes: obstacle category-mobile phone, obstacle boundary (detected by the obstacle contour detection method); for obstacle 3 in this example, the corresponding third obstacle information includes: obstacle category-pet, obstacle boundary (detected by the obstacle contour detection method). As can be seen, the obstacle category detection method and the obstacle contour detection method may detect the same obstacle, or may not detect the same obstacle, but through the matching step of the embodiment, the information of the same obstacle can be complemented and fused, which can ensure the comprehensive extraction of the obstacle information to a certain extent and avoid missing detection.

[0063] In another possible implementation, the first obstacle information and the second obstacle information located at the same position of the image information are fused to determine the third obstacle information at the position, including: fusing the first obstacle information and the second obstacle information located at the same position of the image information, and performing the following steps on the fused information: if the fused information includes an obstacle contour, determining an obstacle boundary based on the obstacle contour; if the fused information only includes an obstacle detection frame, determining an obstacle boundary based on the obstacle detection frame; if the fused information includes an obstacle contour and an obstacle detection frame, determining an obstacle category based on the intersection and union ratio of the obstacle contour and the obstacle detection frame. As can be seen in this implementation, the obstacle contour detected by the obstacle contour detection method is more inclined to be used to determine the obstacle boundary, and if the obstacle contour detection method does not detect the obstacle contour, the obstacle boundary is determined based on the obstacle detection frame detected by the obstacle category detection method. In this way, the information of the same obstacle in the same image information obtained by the two image recognition methods (i.e., the first obstacle information of the same obstacle detected by the obstacle category detection method and the second obstacle information of the same obstacle detected by the obstacle contour detection method) can be complemented and fused, the detection coverage can be improved, and the robot can be accurately and effectively avoided from the obstacle.

[0064] In an embodiment, the intersection-over-union of the obstacle contour and the obstacle bounding box is used to determine the obstacle category, including: if the intersection-over-union is greater than a preset intersection-over-union threshold, the initial category of the obstacle is taken as the obstacle category; otherwise, the obstacle category is determined as a general obstacle. This way, the intersection-over-union is used to measure the coincidence degree of the obstacle contour and the obstacle bounding box. When the coincidence degree is high (i.e., the intersection-over-union is greater than the preset intersection-over-union threshold), it is considered that the initial category output by the obstacle category detection method is more accurate, and thus the initial category of the obstacle is taken as the obstacle category. When the coincidence degree is low (i.e., the intersection-over-union is not greater than the preset intersection-over-union threshold), it is considered that the initial category output by the obstacle category detection method is not highly confident, and thus the obstacle category is determined as a general obstacle. Here, it is set that the confidence of the first obstacle information detected by the obstacle category detection method is lower than the confidence of the second obstacle information detected by the obstacle contour detection method.

[0065] S305, acquiring fourth obstacle information from the moving direction of the cleaning robot by the line laser sensor.

[0066] In this embodiment, the line laser sensor is used to emit: a first line laser beam emitted along the horizontal direction and a second line laser beam emitted downwardly inclined relative to the horizontal plane. Based on the distance and other information measured by the first line laser beam, edge-encircling of non-low obstacle (e.g., obstacle higher than X centimeters) can be realized; based on the distance and other information measured by the second line laser beam, the corresponding fourth obstacle information of low obstacle (e.g., obstacle not higher than X centimeters) can be determined. The value of X is determined according to the installation position of the line laser sensor, specifically according to the height from the ground, and in this embodiment, X = 6.

[0067] In an embodiment, the fourth obstacle information from the moving direction of the cleaning robot is acquired by the line laser sensor, including: the point cloud information corresponding to the second line laser beam emitted by the line laser sensor is determined; and the fourth obstacle information of the low obstacle is determined based on the height difference of the point cloud information. More specifically, since the line laser sensor includes a signal emitting module and a signal receiving module, after the signal emitting module emits the second line laser beam, the laser beam is reflected by the obstacle and then received by the signal receiving module as the point cloud information corresponding to the second line laser beam. Accordingly, the first line laser beam emitted by the line laser sensor can also determine the corresponding point cloud information, and based on the point cloud information, the related information of the non-low obstacle can be obtained. It should be noted that the second line laser beam and the first line laser beam present as horizontal lines on the obstacle and the ground, and the height difference between the horizontal lines on the ground and the horizontal lines on the obstacle is the height difference of the point cloud information.

[0068] Therefore, in an example, the fourth obstacle information is generated according to the distance and the like information measured by the second line laser beam emitted by the line laser sensor at an angle; the fourth obstacle information can include the distance between the self-moving cleaning robot and the obstacle, the obstacle edge point, the boundary line formed, and the like.

[0069] It should be noted that the execution order of S302, S303 and S305 has no priority, and they can be executed simultaneously or sequentially. In addition, the execution order of S305 and S304 also has no priority, and they can be executed simultaneously or sequentially.

[0070] S306, based on the third obstacle information and the fourth obstacle information, generates an obstacle map of the area where the self-moving cleaning robot is located.

[0071] In order to better fuse the third obstacle information and the fourth obstacle information and ensure the accuracy of the fused information, the target obstacle on the carpet and the object obstacle on the non-carpet are distinguished in this embodiment.

[0072] It should be noted that due to the interference of the carpet, the accuracy of the fourth obstacle information determined by the second line laser beam emitted by the line laser sensor may not be high. If such low-accuracy information is fused into the third obstacle information, it will not only affect the accuracy of the third obstacle information, but also may further cause inaccurate obstacle recognition and inaccurate obstacle-avoiding trajectory. Therefore, for the target obstacle on the carpet, the target obstacle is inserted into the instant positioning map of the self-moving cleaning robot according to the obstacle boundary in the third obstacle information, so as to ensure the accuracy of obstacle recognition and obstacle-avoiding trajectory, that is, the fourth obstacle information is ignored to prevent the inaccurate obstacle information on the carpet from affecting the accuracy of the obstacle information determined based on image detection. The instant positioning map is a SLAM map.

[0073] On the other hand, for the object obstacle on the non-carpet or the low obstacle determined based on the second line laser beam, fusing the fourth obstacle information and the third obstacle information can ensure the accuracy of obstacle recognition and obstacle-avoiding trajectory. Therefore, for the low obstacle or the object obstacle on the non-carpet, the obstacle frame in the fourth obstacle information is merged with the obstacle boundary in the third obstacle information to obtain a merged area (i.e. the area surrounded by the obstacle boundary and the maximum coverage range after merging the obstacle frame); the object obstacle or the low obstacle is inserted into the instant positioning map of the self-moving cleaning robot according to the merged area, so as to fuse the obstacle information determined based on the image and the obstacle information determined based on the line laser sensor, to obtain more comprehensive and more accurate obstacle information, which is conducive to realizing accurate edge-avoiding obstacle avoidance of the robot for various obstacles, and at the same time can reduce the cleaning blind area and realize cleaning of the obstacle edge as much as possible.

[0074] Correspondingly, in an embodiment, generating the obstacle map of the area where the self-moving cleaning robot is located based on the third obstacle information and the fourth obstacle information comprises: for the target obstacle on the carpet, inserting the target obstacle into the instant positioning map of the self-moving cleaning robot according to the obstacle boundary in the third obstacle information to obtain the obstacle map; for the object obstacle on the non-carpet or the low obstacle determined based on the second line laser beam, merging the obstacle frame in the fourth obstacle information with the obstacle boundary in the third obstacle information to obtain a merged area; and inserting the object obstacle or the low obstacle into the instant positioning map of the self-moving cleaning robot according to the merged area to obtain the obstacle map.

[0075] It should be noted that when inserting the obstacle into the instant positioning map, a coordinate conversion action is also involved, that is, the coordinates of the profile to be inserted (such as the profile of the merged area or the obstacle boundary in the third obstacle information) need to be converted into the global coordinate system of the instant positioning map, and the insertion action is completed according to the converted coordinates.

[0076] S307, controlling the self-moving cleaning robot to avoid the obstacle according to the obstacle map.

[0077] In this embodiment, the specific process of controlling the self-moving cleaning robot to avoid the obstacle according to the obstacle map can be further introduced for the non-low obstacle and the low obstacle respectively. The edge-following around the obstacle for the non-low obstacle can be performed in cooperation with at least one edge sensor on the side of the main body. The edge-following around the obstacle for the low obstacle can be performed without referring to the distance information measured by the edge sensor.

[0078] Correspondingly, in an embodiment, the side of the main body is further provided with: at least one edge sensor; and accordingly, controlling the self-moving cleaning robot to avoid the obstacle according to the obstacle map comprises: for the non-low obstacle determined based on the first line laser beam, determining an equidistant curve outside the obstacle in the obstacle map based on the safety distance corresponding to the obstacle category; planning an edge-following trajectory for avoiding the obstacle based on the equidistant curve; and controlling the self-moving cleaning robot to move according to the edge-following trajectory, and in the moving process, controlling the moving direction of the self-moving cleaning robot to avoid the obstacle according to the distance data collected by the first line laser beam and the edge sensor. Since the non-low obstacle can not be captured by the camera, in order to realize accurate edge-following around the obstacle, at least one edge sensor on the side of the main body can be cooperated in the process of edge-following around the obstacle to optimize and adjust the edge-following trajectory in real time.

[0079] Correspondingly, for low obstacles, the obstacle avoidance of the self-moving cleaning robot is controlled according to the obstacle map, which comprises: for low obstacles determined based on the second line laser beam, determining the equidistant curve outside the obstacle in the obstacle map based on the safety distance corresponding to the obstacle category; planning the edge travel trajectory for avoiding the obstacle based on the equidistant curve; and controlling the self-moving cleaning robot to move according to the edge travel trajectory, and controlling the travel direction of the self-moving cleaning robot to avoid the obstacle during the movement according to the first line laser beam.

[0080] It also needs to be explained that, in order to realize the correspondence between different categories of obstacles and the corresponding safety distance, a dictionary of obstacle category to safety distance can be established, such as electric wires, fabrics, fragile vases, etc. as dangerous obstacles, which are inflated outward by 5 cm; scales, bases, etc. as safe and touchable obstacles, which are inflated outward by 2 cm. The inflation can be understood as the expansion of the boundary. Correspondingly, the determination of the equidistant curve outside the obstacle in the obstacle map based on the safety distance corresponding to the obstacle category comprises: performing inflation processing on the obstacle boundary based on the safety distance corresponding to the obstacle category, and determining the equidistant curve outside the obstacle based on the boundary line after the inflation processing. This embodiment makes different categories of obstacles correspond to the corresponding safety distance, which can ensure the safety and smoothness of the cleaning process. For example: electric wires, fabrics, etc. are easy to entangle the self-moving parts of the robot, which causes the movement of the robot to be blocked. A certain obstacle avoidance safety distance is maintained for such obstacles, which can avoid the movement of the robot being blocked to a certain extent; and vases, etc. are easy to break due to collision, so a certain obstacle avoidance safety distance is maintained for such fragile obstacles, which can avoid damage to the objects to a certain extent; correspondingly, the safe and touchable obstacles such as scales and bases can be edge-wound, which can ensure the cleaning effect and will not damage the objects. When specifically implemented, the safety distance of the safe and touchable obstacles is greater than that of the entanglement type obstacles and the fragile type obstacles. The safety distances of the entanglement type obstacles and the fragile type obstacles can be equal or not equal.

[0081] It can be seen that, in this embodiment, the image information of the robot travel direction is detected by two different image recognition methods, which can complement and fuse the information of the same obstacle in the image information. Then, the image fusion information of the same obstacle obtained by the complementation and fusion is again integrated with the fourth obstacle information determined by the line laser sensor to generate the obstacle map of the area where the self-moving cleaning robot is located. Finally, the self-moving cleaning robot is controlled to avoid obstacles according to the obstacle map, which can realize the edge obstacle avoidance of the robot to various obstacles. This scheme does not depend on the recognition of the geometric features and categories of objects, and can have good detection effect on various types of objects, with high detection coverage, which can make the robot accurately and effectively avoid obstacles.

[0082] The application also considers that the robot can only see a certain face and part of the boundary of the obstacle at a certain moment, and cannot see the complete edge of the obstacle, so more image information of the view is needed to correspondingly optimize and update the robot travel trajectory. Specifically, after the robot moves based on the calculation at the current moment, the latest image information of the robot travel direction is obtained, and then the relevant steps of the processing of the preceding embodiments are processed for the latest image information to optimize the robot travel trajectory, thereby supplementing the robot field of view in real time and correspondingly optimizing the robot travel trajectory. That is, in the process of avoiding obstacles by the robot, the latest image information of the robot travel direction is obtained in real time; the latest image information is used to optimize the robot travel trajectory. For example: after the robot avoids obstacles to the boundary of the region, if the target obstacle is not within the range, a step back determination is triggered, so that the robot slows down and retreats a certain distance. Through the retreat, the robot expects to again include the target obstacle within the range of the image acquisition component, and re-shoot the latest image information to re-generate the obstacle-avoiding trajectory.

[0083] Since the obstacle-avoiding action performed by the self-moving cleaning robot in the obstacle-avoiding process can include: the self-moving cleaning robot stopping at the edge of the obstacle, then retreating, and then turning to move, etc. Therefore, in order to accurately determine the specific obstacle-avoiding action of the robot in the obstacle-avoiding process, it can be determined whether the self-moving cleaning robot moves to the end of the boundary of the obstacle in the process of avoiding the obstacle by the self-moving cleaning robot; if so, in the case that the obstacle does not fall within the range of the image acquisition component, the self-moving cleaning robot is controlled to retreat, turn, etc.; in the case of retreating to the obstacle falling within the range of the image acquisition component, the latest image information of the travel direction of the self-moving cleaning robot (the latest image information of the obstacle) is obtained by the image acquisition component; the contour completion information is determined based on the latest image information; the contour completion information is filled into the obstacle map to obtain the latest obstacle map; the self-moving cleaning robot is controlled to avoid the obstacle according to the latest obstacle map, and the steps of determining whether the self-moving cleaning robot moves to the end of the boundary of the obstacle in the process of avoiding the obstacle by the self-moving cleaning robot and subsequent other steps are performed until the obstacle contour in the latest obstacle map forms a closed loop, thereby supplementing the robot field of view in real time and correspondingly optimizing and adjusting the robot travel trajectory. The process can be specifically referred to in the foregoing embodiments. Figure 4 , Figure 4 It is illustrated that the obstacle-avoiding is for the water, milk and other liquids forming water accumulation (i.e. transparent obstacle) on the ground, as shown in FIG. 9. Figure 4 When the self-moving cleaning robot first identifies a part of the boundary of the water accumulation and avoids the obstacle along the boundary for a distance, the image acquisition component (such as the camera) is triggered to capture the latest image information of the robot travel direction, as shown in FIG. 10. Figure 4The ellipses shown in the direction of travel of the robot) can have failed to capture the boundary of the puddle, so the self-moving cleaning robot is turned and backed up to enable the image acquisition component to capture other boundaries, and the image acquisition component obtains updated image information of the obstacle when the self-moving cleaning robot backs up to be able to take pictures of the other boundaries; the steps S302-S306 of the foregoing embodiments of the present application are performed for this updated image information, whereby subsequent other boundaries can be supplemented to the original obstacle map, and an updated obstacle map is obtained. The self-moving cleaning robot repeats the above operation according to the updated obstacle map until the obstacle profile in the updated obstacle map forms a closed loop, for example: when the obstacle is Figure 4 a puddle as shown, eventually a closed region as shown Figure 5 in the obstacle map can be formed.

[0084] In an embodiment, the profile completion information is determined based on the updated image information, including: detecting a plurality of first updated obstacle information from the updated image information by an obstacle category detection method; detecting a plurality of second updated obstacle information from the updated image information by an obstacle profile detection method; fusing the first updated obstacle information and the second updated obstacle information located at the same position in the image information to determine third updated obstacle information at the position; obtaining fourth updated obstacle information in the direction of travel of the self-moving cleaning robot by a line laser sensor; and determining the profile completion information based on the third updated obstacle information and the fourth updated obstacle information. The process can refer to the specific implementation of steps S302-S306.

[0085] In order to combine the historical obstacle-avoiding data to optimize the current obstacle-avoiding trajectory, in an embodiment, an edge-following trajectory for avoiding obstacles is planned based on an equidistant curve, including: obtaining a historical obstacle map; and planning an edge-following trajectory based on the equidistant curve and a historical trajectory in the historical obstacle map. It should be noted that the historical obstacle map is obtained by a self-moving cleaning robot for cleaning a specific distance, for example, after the self-moving cleaning robot completes cleaning of a living room, a historical obstacle map corresponding to the living room can be obtained, and then when the self-moving cleaning robot cleans the living room again, the obstacle-avoiding trajectory (i.e., the trajectory) generated for cleaning the living room this time can be corrected with reference to the historical obstacle map (i.e., the historical obstacle map), thereby improving the trajectory generation efficiency and accuracy. Because the positions of obstacles such as sofas and wardrobes in a home environment generally do not change, for such obstacles with unchanged positions, the obstacle-avoiding efficiency and accuracy can be improved according to this method. Of course, for obstacles with unchanged positions, in order to have higher trajectory generation efficiency, the historical trajectory in the historical obstacle map can be directly used for edge-following obstacle avoidance. That is, if the A obstacle is identified in the current cleaning task, if the position of the A obstacle does not change when the next cleaning task is performed in the region, the obstacle-avoiding trajectory can be generated based on the current obstacle map, or the historical obstacle-avoiding trajectory can be directly used for obstacle avoidance. If the position of the A obstacle changes, the boundary of the A obstacle needs to be re-identified, and the obstacle-avoiding trajectory needs to be re-generated accordingly.

[0086] In another possible implementation, in order to improve the obstacle avoidance effect of the self-moving cleaning robot, multi-frame image can be combined for visual segmentation processing, and the position of the obstacle is stabilized through time filtering (such as Kalman / optical flow method) to avoid false avoidance or collision caused by motion blur and segmentation jitter. For example, a plurality of frames of image information are continuously collected, and for the image information, the related steps provided in the foregoing embodiments are performed, and then for the same obstacle, a plurality of third obstacle information can be obtained, and the plurality of third obstacle information is fitted to reduce the detection error of a single frame of image to a certain extent and improve the obstacle avoidance accuracy. Accordingly, the obstacle map of the area where the self-moving cleaning robot is located is generated based on the third obstacle information and the fourth obstacle information, including: fitting the third obstacle information corresponding to the same obstacle in the plurality of frames of image information to obtain fitting information; and generating the obstacle map based on the fitting information and the fitting result of the fourth obstacle information corresponding to the same obstacle. In this way, the detection information of the same obstacle in different frames of image information can be superimposed and fitted, more accurate detection results can be obtained, and more accurate obstacle avoidance path of the self-moving cleaning robot can be obtained. For example, three frames of image information are continuously collected, and the methods S301-S305 in the foregoing embodiments are performed for each frame of image information. For each frame of image information, only the corresponding third obstacle information and the fourth obstacle information can be obtained, and then the three third obstacle information corresponding to the same obstacle is fitted to obtain fitting information, and the three fourth obstacle information corresponding to the same obstacle is fitted to obtain a fitting result. The obstacle map can be generated based on the fitting information and the fitting result. Through the fitting action, the detection error between different frames of image information can be eliminated, and false detection caused by motion blur and other problems can be avoided.

[0087] In the following, a kind of obstacle avoidance scheme for sweeping robot is provided. Sweep robot can be configured along edge sensor, one RGB front view camera and one or more warm light fill light. Image information is collected by RGB front view camera, and semantic segmentation is carried out on the image information to accurately identify the obstacle and obstacle position in the image, and then the obstacle is inserted into global grid map (SLAM map), to realize accurate edge obstacle avoidance of sweeping robot to obstacle. Exemplarily, image information collected by RGB front view camera and corresponding region segmentation result please see Figure 6 .

[0088] In the obstacle avoidance scheme for sweeping robot, the controller in sweeping robot is built-in AI identification function, the AI identification function is the self-moving cleaning robot obstacle avoidance method provided in the application, the line laser sensor in sweeping robot is integrated with camera and fill light, and the relevant information collected by line laser sensor and camera is transmitted to controller, so that the controller executes the self-moving cleaning robot obstacle avoidance method provided in the application.

[0089] See Figure 7 The floor cleaning robot can specifically perform the following steps:

[0090] Step A - image acquisition and preprocessing.

[0091] The front-view camera of the floor cleaning robot acquires an environment image, which can be set to have a frame rate ≥ 8 FPS, and performs operations such as brightness enhancement, white balance, and adaptive exposure on the acquired image to adapt to a dark environment. When it is detected that the image brightness (which can be represented by the BV value or the average gray value) is lower than the preset brightness threshold, the fill light is automatically turned on to realize the detection of obstacles in a dark environment.

[0092] Step B - AI target detection and general obstacle segmentation.

[0093] This step identifies obstacles through a target detection algorithm (i.e., an obstacle category detection method) and a general obstacle segmentation algorithm (an obstacle contour detection method). The target detection algorithm obtains the category labels and bounding boxes (i.e., obstacle detection boxes) of various predefined obstacles. The general obstacle segmentation algorithm obtains the segmentation regions and obstacle boxes (i.e., obstacle contours, which can be represented by a segmentation mask) of obstacles without category distinction. The accuracy of the segmentation can also be optimized through some morphological processing operations, such as erosion (hole filling), dilation, and hole filling, to smooth the region foreground boundary and strengthen the obstacle contour to maintain edge details.

[0094] Specifically, the target detection algorithm can detect fixed categories of objects, and the output information includes obstacle detection boxes and categories. The general obstacle segmentation algorithm identifies obstacles without category limitation, and the output information includes regions and their bounding boxes (generally represented by a mask). Both the target detection algorithm and the general obstacle segmentation algorithm need to be adaptively and specifically trained and optimized according to the actual needs of this embodiment, so that the target detection algorithm is suitable for detecting various objects in an indoor floor scene, and the general obstacle segmentation algorithm is suitable for segmenting and determining the regions and boundaries of general obstacles. The fusion of the output information of the target detection algorithm and the general obstacle segmentation algorithm can specifically include the following cases: the target detection algorithm detects the detection boxes and categories, and the general obstacle segmentation algorithm segments the corresponding regions, so that the segmented regions are used for subsequent edge-avoiding obstacle avoidance, and the specific obstacle categories are output; the target detection algorithm detects the detection boxes and categories, but the general obstacle segmentation algorithm does not segment the corresponding regions, so that the detection boxes are used for subsequent edge-avoiding obstacle avoidance, and the specific obstacle categories are output; the target detection algorithm does not detect the detection boxes and categories, but the general obstacle segmentation algorithm segments the corresponding regions, so that the segmented regions are used for subsequent edge-avoiding obstacle avoidance, and the unknown general obstacle categories are output.

[0095] In an example, the post-processing optimization of the segmentation mask region can include:

[0096] 1) Thresholding and denoising: the purpose is to remove low-probability false foreground points, and the specific operation includes: removing low-probability false foreground points, and median filtering to remove isolated points while maintaining the shape of the region edge.

[0097] 2) Morphological erosion: the purpose is to remove region edge burrs and small false foregrounds. The specific operation includes: using the erosion operator provided by the opencv vision library, and eroding the region with a 3x3 kernel. Opencv is an open source cross-platform computer vision and machine learning software library that can run on Linux, Windows, Android and Mac OS operating systems, and provides interfaces for Python, Ruby, MATLAB and other languages, and implements many general algorithms in image processing and computer vision.

[0098] 3) Morphological dilation: the purpose is to restore the details lost by erosion and make the edge closer to the true contour. The specific operation includes: using the dilation operator provided by the opencv vision library to erode and then dilate, which can effectively denoise and smooth the edge.

[0099] 4) Hole filling: the purpose is to fill small holes in the foreground to make the region connected. The specific operation includes: using the inverse FloodFill algorithm to take the inverse of the region, and filling from the boundary pixels. The filled region corresponds to the external background in the inverse space, and after marking removal, the remaining non-zero connected region is the hole to be filled. The FloodFill algorithm can be used for image filling and region counting, and realizes connected region diffusion by marking adjacent nodes.

[0100] After optimizing the segmentation mask region, the output of the target detection algorithm and the output of the general obstacle segmentation algorithm are fused to supplement the detection results of the target detection algorithm with the segmentation mask to prevent missed detection, and the contour of the segmentation region can also be used instead of the detection frame output by the target detection algorithm to achieve more accurate edge-avoiding obstacle avoidance. Among them, the intersection over union of the detection frame output by the target detection algorithm and the segmentation mask can be calculated, and if the intersection over union is greater than a preset intersection over union threshold, a specific class semantic is assigned, and for the intersection over union less than the threshold, a general obstacle semantic is assigned, so as to determine the obstacle class (i.e. the obstacle class included in the third obstacle information).

[0101] Step C-Generate obstacle avoidance strategy.

[0102] The obstacle position is extracted using the output obstacle class and general segmentation area in step B (i.e. the obstacle class and obstacle boundary included in the third obstacle information), and the boundary inflation distance corresponding to the obstacle class (i.e. the obstacle class included in the third obstacle information) is referred to as a safety distance. The morphological inflation method is used to expand the obstacle position to a safety distance, wherein one class corresponds to one safety distance. Then, the local coordinate system of the obstacle boundary is converted to the global coordinate system to insert the obstacle boundary into the grid map. Then, the obstacle position (expanded obstacle boundary) extracts an outer equidistant curve, and the edge-attached obstacle avoidance trajectory of the sweeping robot is generated accordingly.

[0103] In order to clearly show the safety distance corresponding to each class, a dictionary can be established to map the obstacle class to the inflation safety distance. For example, electric wires and fabrics are dangerous obstacles, and the safety distance outwardly inflated is 5 cm; scales and bases are safe and can be collided obstacles, and the safety distance outwardly inflated is 2 cm.

[0104] In an example, the specific implementation process of inserting the obstacle boundary into the grid map includes: projecting the points on the obstacle position (expanded obstacle boundary) to the camera coordinate system (Xc, Yc, Zc) using the assumption plane, transforming the camera coordinates to the robot base coordinates (Xb, Yb, Zb) using the sensor external parameter (R, t), converting the base coordinates to the world coordinates according to the pose of the robot in the map (global), converting the world coordinates to the grid index (i, j), and writing into the grid map; performing discretization inflation or distance transformation based on the safety distance on the grid map to obtain a binary occupancy or cost map; performing global planning on the cost map (obstacle map) to obtain a rough path, and then using local planning to generate an obstacle avoidance and smooth trajectory.

[0105] Step D - performing edge-attached obstacle avoidance.

[0106] The sweeping robot moves according to the obstacle avoidance trajectory: but considering that the sweeping robot cannot see the whole picture of the obstacle from one angle, when the sweeping robot rounds to the boundary of the obstacle, the sweeping robot performs a local field of view completion strategy so that the obstacle can be completed on the grid map to prevent collision with the obstacle during the rounding process, and thus the complete obstacle avoidance process is completed.

[0107] Specifically, after the sweeping robot rounds to the boundary of the region, if the target obstacle is not within the range, a step back determination is triggered, so that the sweeping robot slows down and retreats a certain distance. Through the retreat, the sweeping robot expects to be able to bring the target obstacle into the range of the camera again, and re-shoot the latest image information to re-generate the obstacle rounding trajectory.

[0108] Further, since the sweeping robot is also provided with a line laser sensor and an edge sensor, the obstacle-avoiding trajectory can also be realized in cooperation with the data collected by the line laser sensor and the edge sensor, and the following cases can be divided:

[0109] (1) Since the navigation line laser beam (first line laser beam) emitted by the line laser sensor in the horizontal direction can play a navigation role in edge-avoiding of non-low obstacles, the obstacle-avoiding process of the sweeping robot can control the robot to stop, retreat, turn, etc., to provide pose data support for the robot, and the edge-avoiding is controlled by the edge sensor to control the edge-avoiding distance.

[0110] (2) The obstacle laser beam (second line laser beam) emitted by the line laser sensor in the inclined direction can identify and detect low obstacles (such as carpets, wires, scales, etc.) with a height of 1.5 cm to 6 cm, so the obstacle map can be generated by combining the obstacles measured by the laser beam (i.e. the fourth obstacle information) and the obstacles measured by the image recognition (i.e. the third obstacle information), and the obstacle-avoiding trajectory can be determined based on the obstacle map. For the edge-avoiding process of low obstacles, the information measured by the edge sensor can be ignored.

[0111] It should be noted that when the information of the obstacles measured by the obstacle laser beam and the obstacles measured by the image recognition is combined, the following process is referred to: if the obstacles measured by the obstacle laser beam and the obstacles measured by the image recognition occupy a certain grid in the grid map, it is considered that the grid is occupied by the obstacles, that is, the maximum coverage range after merging is taken as the final contour of the obstacles. In addition, it is necessary to distinguish between obstacles on the carpet and obstacles on the non-carpet. Due to the interference of the carpet, the accuracy of the fourth obstacle information determined by the second line laser beam emitted by the line laser sensor may not be high. Merging such low-accuracy information into the third obstacle information will not only affect the accuracy of the third obstacle information, but also may further lead to inaccurate obstacle recognition and inaccurate obstacle-avoiding trajectory. Therefore, for the target obstacle on the carpet, only the obstacle boundary in the third obstacle information is used to insert the target obstacle into the grid map to ensure the accuracy of obstacle recognition and obstacle-avoiding trajectory. On the other hand, for the object obstacle on the non-carpet or the low obstacle determined based on the second line laser beam, the fusion of the fourth obstacle information and the third obstacle information can ensure the accuracy of obstacle recognition and obstacle-avoiding trajectory, so for the low obstacle or the object obstacle on the non-carpet, the obstacle frame in the fourth obstacle information is merged with the obstacle boundary in the third obstacle information to obtain a merged area (i.e. the area surrounded by the obstacle boundary and the maximum coverage range after merging the obstacle frame). The object obstacle or the low obstacle is inserted into the grid map according to the merged area.

[0112] It can be seen that in the embodiment, the sweeping robot only needs to be provided with a single camera, a line laser sensor and an edge sensor, thereby saving cost, and realizing a general obstacle avoidance function. The scheme does not depend on the recognition of geometric features and categories of objects, and can have good detection effect on various types of objects, has high detection coverage, and can enable the robot to accurately and effectively avoid obstacles.

[0113] An electronic device provided by an embodiment of the present application is described below. The electronic device described below can be mutually referred to with other embodiments described herein. The electronic device in the embodiment can be a self-moving cleaning robot.

[0114] An electronic device is disclosed by an embodiment of the present application, comprising:

[0115] A memory is configured to store a computer program.

[0116] A processor is configured to execute the computer program to implement the method disclosed in any of the above embodiments.

[0117] Further, an electronic device is also provided by an embodiment of the present application. The electronic device can be a server as shown in Figure 8 , or a terminal as shown in Figure 9 . Figure 8 , and Figure 9 are structural diagrams of electronic devices according to an exemplary embodiment, and the contents in the diagrams should not be considered as any limitation on the use range of the present application.

[0118] Figure 8 A structural diagram of a server provided by an embodiment of the present application is shown. The server can specifically include at least one processor, at least one memory, a power supply, a communication interface, an input / output interface and a communication bus. The memory is configured to store a computer program, and the computer program is loaded and executed by the processor to implement the related steps in the obstacle avoidance of the self-moving cleaning robot disclosed in any of the preceding embodiments.

[0119] In the embodiment, the power supply is configured to provide working voltage for each hardware device on the server; the communication interface can create a data transmission channel between the server and external devices, and the communication protocol followed by the communication interface is any communication protocol applicable to the technical solution of the present application, which is not specifically limited herein; the input / output interface is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not specifically limited herein.

[0120] In addition, the memory as a carrier for storing resources can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon include an operating system, a computer program and data, etc. The storage mode can be temporary storage or permanent storage.

[0121] The operating system is used to manage and control each hardware device on the server and the computer program, so as to realize the operation and processing of the processor on the data in the memory. The operating system can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the self-moving cleaning robot obstacle avoidance method disclosed in any of the preceding embodiments, the computer program can further include a computer program capable of completing other specific work. In addition to the data including the update information of the application program, etc., the data can also include the developer information of the application program, etc.

[0122] Figure 9 A structure schematic diagram of a terminal provided by the embodiment of the present application can include, but is not limited to, a smart phone, a tablet computer, a notebook computer or a desktop computer, etc.

[0123] Generally, the terminal in the embodiment includes a processor and a memory.

[0124] The processor can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor can be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array) and a PLA (Programmable Logic Array). The processor can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor can be integrated with a GPU (Graphics Processing Unit). The GPU is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor can also include an AI (Artificial Intelligence) processor. The AI processor is used to process the computing operation related to machine learning.

[0125] The memory can include one or more computer non-volatile storage media, which can be non-transitory. The memory can also include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices. In the embodiment, the memory is at least used to store the following computer program, wherein the computer program is loaded and executed by the processor, and can realize the related steps in the self-moving cleaning robot obstacle avoidance method disclosed by any of the preceding embodiments. In addition, the resources stored by the memory can also include operating systems, data, etc., and the storage mode can be temporary storage or permanent storage. The operating system can include Windows, Unix, Linux, etc. The data can include but is not limited to application update information.

[0126] In some embodiments, the terminal can also include a display screen, an input / output interface, a communication interface, a sensor, a power supply, and a communication bus.

[0127] Those skilled in the art can understand that, Figure 9 The structure shown in the figure does not constitute a limitation on the terminal, and can include more or fewer components than the figure.

[0128] The following describes a non-volatile storage medium provided by an embodiment of the present application. The non-volatile storage medium described below can be referred to each other with other embodiments described herein.

[0129] A non-volatile storage medium for saving a computer program, wherein the computer program is executed by a processor to implement the self-moving cleaning robot obstacle avoidance method disclosed in the preceding embodiments. The non-volatile storage medium is a computer-readable non-volatile storage medium, which is a carrier for storing resources, and can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc. The resources stored thereon include an operating system, a computer program and data, etc., and the storage mode can be temporary storage or permanent storage.

[0130] The following describes a computer program product provided by an embodiment of the present application. The computer program product described below can be referred to each other with other embodiments described herein.

[0131] A computer program product includes a computer program / instruction, which is executed by a processor to implement the steps of the self-moving cleaning robot obstacle avoidance method disclosed above.

[0132] Embodiments of the present application also provide another computer program product, which includes a non-volatile computer readable storage medium for storing a computer program, and the computer program is executed by a processor to implement the steps in any of the preceding embodiments.

[0133] The various embodiments described in this specification are presented by way of example, and each embodiment describes a specific implementation for the various embodiments. The embodiments are not intended to be exhaustive or to limit the application to the precise form disclosed. The embodiments described herein are presented only to provide the best explanation for the principles and application of the application.

[0134] The descriptions of the methods and algorithms described in this specification are implemented by hardware, software modules executed by a processor, or a combination of both. The software modules can be stored in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of non-volatile storage medium known in the art.

[0135] The principles and implementations of the application are described herein using specific examples. The above descriptions of the embodiments are only intended to help understand the methods and core ideas of the application. For those skilled in the art, the specific implementations and application ranges can be changed according to the ideas of the application. In summary, the content of the specification should not be understood as a limitation of the application.

Claims

1. A method for obstacle avoidance of a self-moving cleaning robot, characterized in that, The self-moving cleaning robot comprises a body main body; at least one edge sensor is arranged on the side of the body main body; a line laser sensor and an image acquisition assembly are arranged on the front side of the body main body along the direction of travel, and the image acquisition assembly and the line laser sensor are integrally arranged; the line laser sensor is used for emitting a first line laser beam emitted in the horizontal direction and a second line laser beam emitted downwardly inclined relative to the horizontal plane; the first line laser beam and the second line laser beam present horizontal lines on the obstacle and the ground; the line laser sensor comprises a base and a mirror assembly; a signal emitting module and a signal receiving module are fixedly arranged on the base; the signal emitting module can emit the first line laser beam and the second line laser beam; the mirror assembly is rotatably arranged on the base, and the mirror assembly comprises a signal reflecting part, wherein the signal reflecting part comprises a first signal reflecting part and a second signal reflecting part; the first signal reflecting part can reflect the first line laser beam in the horizontal direction to perform distance detection in the horizontal direction; the second signal reflecting part can reflect the second line laser beam downwardly inclined relative to the horizontal plane to perform ground obstacle or cliff detection; the obstacle avoidance method comprises: acquiring image information of the direction of travel of the self-moving cleaning robot through the image acquisition assembly; detecting a plurality of first obstacle information from the image information through an obstacle category detection method; the first obstacle information comprises obstacle initial category and obstacle detection frame; detecting a plurality of second obstacle information from the image information through an obstacle contour detection method; the second obstacle information comprises obstacle contour; fusing the first obstacle information and the second obstacle information located at the same position in the image information to determine third obstacle information of the position; the third obstacle information comprises obstacle category and obstacle boundary; determining corresponding point cloud information through the second line laser beam emitted by the line laser sensor; determining fourth obstacle information of a low obstacle based on the height difference of the point cloud information; the fourth obstacle information comprises the distance between the self-moving cleaning robot and the obstacle, obstacle edge point and boundary line formed; generating an obstacle map of the area where the self-moving cleaning robot is located based on the third obstacle information and the fourth obstacle information; determining a 1.5 cm to 6 cm obstacle avoidance trajectory of the low obstacle based on the obstacle map; acquiring point cloud information of the direction of travel of the self-moving cleaning robot based on the first line laser beam emitted by the line laser sensor, performing edge obstacle avoidance on a non-low obstacle according to the point cloud information, providing pose data of the self-moving cleaning robot for navigation, controlling the self-moving cleaning robot to stop, retreat, turn, and performing edge obstacle avoidance on the non-low obstacle through the edge sensor to control the edge distance; the height of the non-low obstacle is greater than 6 cm. Specifically, for the target obstacle on the carpet, the target obstacle is inserted into the real-time positioning map of the self-moving cleaning robot according to the obstacle boundary in the third obstacle information, thus obtaining the obstacle map; For objects or obstacles not on the carpet or low obstacles determined by the second line laser beam, the obstacle border in the fourth obstacle information is merged with the obstacle boundary in the third obstacle information to obtain the merged area with the maximum coverage. According to the merged area, the object obstacle or low obstacle is inserted into the real-time positioning map of the self-moving cleaning robot to obtain the obstacle map; The self-moving cleaning robot is controlled to avoid obstacles based on the obstacle map.

2. The method of claim 1, wherein, The first obstacle information and the second obstacle information located at the same position in the image information are fused to determine the third obstacle information at that position, including: The first obstacle information and the second obstacle information located at the same position in the image information are fused together, and the following steps are performed on the fused information: If the fused information includes the obstacle outline, the obstacle boundary is determined based on the obstacle outline; if the fused information only includes the obstacle detection box, the obstacle boundary is determined based on the obstacle detection box. If the fused information includes the obstacle outline and the obstacle detection box, then the obstacle category is determined based on the intersection-union ratio of the obstacle outline and the obstacle detection box.

3. The method of claim 2, wherein, Determining the obstacle category based on the intersection-union ratio (IUU) of the obstacle contour and the obstacle detection bounding box includes: If the cross-union ratio is greater than a preset cross-union ratio threshold, then the initial obstacle category is taken as the obstacle category; otherwise, the obstacle category is determined as a general obstacle.

4. The method of claim 1, wherein, Controlling the self-moving cleaning robot to avoid obstacles based on the obstacle map includes: For non-low obstacles determined based on the first line laser beam, an equidistant curve is determined on the obstacle map based on the safety distance corresponding to the obstacle category; Based on the equidistant curve, a travel trajectory along the edge is planned to avoid obstacles; The self-moving cleaning robot is controlled to move along the edge trajectory, and during the movement, the direction of travel of the self-moving cleaning robot is controlled to avoid obstacles based on the distance data collected by the edge sensor and the first line laser beam.

5. The method of claim 1, wherein, Controlling the self-moving cleaning robot to avoid obstacles based on the obstacle map includes: For low-lying obstacles identified based on the second laser beam, an equidistant curve is determined on the obstacle map based on the safety distance corresponding to the obstacle category. Based on the equidistant curve, a travel trajectory along the edge is planned to avoid obstacles; The self-moving cleaning robot is controlled to move along the edge trajectory, and during the movement, the direction of travel of the self-moving cleaning robot is controlled according to the first line laser beam to avoid obstacles.

6. The method according to any one of claims 1 to 5, characterized in that, Also includes: In the process of avoiding the obstacle by the self-moving cleaning robot, it is judged whether the self-moving cleaning robot moves to the end of the obstacle boundary; If yes, the self-moving cleaning robot is controlled to retreat in the case that the obstacle does not fall into the range of the image acquisition component; In the case of retreating to the obstacle falling into the range of the image acquisition component, the latest image information of the moving direction of the self-moving cleaning robot is acquired by the image acquisition component; The contour completion information is determined based on the latest image information; The contour completion information is filled into the obstacle map to obtain the latest obstacle map; According to the latest obstacle map, the self-moving cleaning robot is controlled to avoid the obstacle, and the steps of judging whether the self-moving cleaning robot moves to the end of the obstacle boundary in the process of avoiding the obstacle by the self-moving cleaning robot and other subsequent steps are executed until the obstacle contour in the latest obstacle map forms a closed loop.

7. The method of claim 6, wherein, The contour completion information is determined based on the latest image information, including: The first latest obstacle information is detected from the latest image information by the obstacle category detection method; The second latest obstacle information is detected from the latest image information by the obstacle contour detection method; The first latest obstacle information and the second latest obstacle information at the same position of the image information are fused to determine the third latest obstacle information at the position; The fourth latest obstacle information of the moving direction of the self-moving cleaning robot is acquired by the line laser sensor; The contour completion information is determined based on the third latest obstacle information and the fourth latest obstacle information.

8. The method according to claim 4 or 5, characterized in that, The edge-following trajectory for avoiding the obstacle is planned based on the equidistant curve, including: The historical obstacle map is acquired; The edge-following trajectory is planned based on the equidistant curve and the historical trajectory in the historical obstacle map.

9. A self-moving cleaning robot, characterized in that, It includes: A memory for storing a computer program; A processor for executing the computer program to realize the method of any one of claims 1 to 8.

Images