Self-moving robot and system thereof

Abstract

A distance sensor is used for emitting a first target light surface and a second target light surface which have a first target included angle, the first target light surface is parallel to the plane where a robot body is located, and the second target light surface faces the plane where the robot body is located. The obstacle avoidance sensor is used for acquiring feature recognition information of an object in the moving direction of the machine body; the control assembly is connected with the distance sensor and the obstacle avoidance sensor, and is configured to obtain a first-class point cloud frame of a target obstacle irradiated by the first target light surface and the second target light surface; constructing and updating a target two-dimensional map according to the first-class point cloud frame; determining current position information of the self-moving robot according to a matching result of the first-class point cloud frame and the feature recognition information with a pre-acquired space map; and generating a target motion plan according to the target two-dimensional map, the feature recognition information and the current position information. And the cleaning coverage rate and the accuracy of the obstacle avoidance strategy are improved.

Description

Technical Field

[0001] This application relates to the field of cleaning equipment technology, and in particular to a self-moving robot and its system. Background Technology

[0002] With the development of cleaning equipment technology, self-propelled robots have become a commonly used cleaning device in people's lives.

[0003] Autonomous mobile robots typically use sensors to identify obstacles during their movement, but existing autonomous mobile robots have difficulty identifying and avoiding obstacles during their movement.

[0004] Furthermore, existing self-propelled robots are relatively tall, making it difficult for them to enter low-lying areas, resulting in low cleaning coverage and potential blind spots. Summary of the Invention

[0005] Based on this, it is necessary to address the problems in the aforementioned background technology by providing a self-moving robot and its system, which can at least reduce the overall height of the robot body through a sensor module composed of a distance sensor and an obstacle avoidance sensor, accurately identify low-lying objects in the near range using the distance sensor to avoid blind spots, and perform real-time mapping, localization, and path planning for the self-moving robot using the identification information from the distance sensor and the obstacle avoidance sensor, thereby improving the obstacle avoidance effect of the self-moving robot.

[0006] To address the aforementioned technical problems and other issues, according to some embodiments, a first aspect of this application provides a self-moving robot, including a main body and a distance sensor, an obstacle avoidance sensor, and a control component embedded in the main body; the distance sensor is used to emit a first target light surface and a second target light surface having a first target angle, the first target light surface being parallel to the plane where the main body is located, and the second target light surface facing the plane where the main body is located; the first target light surface is used to detect one type of obstacle that is a first target distance from the main body in the direction of movement of the main body, and the second target light surface is used to detect two types of obstacles that are a second target distance from the main body in the direction of movement. The target distance is greater than or equal to the second target distance; the obstacle avoidance sensor is used to acquire feature recognition information of objects in the direction of movement of the robot body; the control component is connected to both the distance sensor and the obstacle avoidance sensor, and is configured to: acquire a type of point cloud frame illuminating the target obstacle by the first target light surface and the second target light surface; construct and update the target two-dimensional map based on the type of point cloud frame, the target two-dimensional map including the distance information between the target obstacle and the robot body; determine the current position information of the self-moving robot based on the matching result of the type of point cloud frame and feature recognition information with the pre-acquired spatial map; and generate the target motion plan based on the target two-dimensional map, feature recognition information, and current position information.

[0007] The self-moving robot in the above embodiments detects a type of obstacle with a first target distance from the robot body in the direction of movement based on a first target light surface emitted by a distance sensor parallel to the plane where the robot body is located. The first target distance mainly depends on the light intensity in the first target light surface. Based on a second target light surface emitted by a distance sensor toward the plane where the robot body is located, it detects two types of obstacles with a second target distance from the robot body in the direction of movement. The first target distance is greater than or equal to the second target distance. For example, by comprehensively utilizing the second target light surface, it is possible to effectively detect low-lying obstacles at close range in the direction of movement of the self-moving robot, avoiding the formation of blind spots in recognition. Furthermore, based on a type of point cloud frame illuminating the target obstacle using the first and second target light surfaces, a target two-dimensional map of the self-moving robot is constructed and updated according to the type of point cloud frame. The target two-dimensional map includes the distance information between the target obstacle and the robot body. For example, the target obstacle may include at least one small object such as a small toy, building block, or thin thread, so as to realize real-time mapping and localization of the self-moving robot. Based on the type of point cloud frame and the matching result of the feature recognition information obtained based on the obstacle avoidance sensor and the pre-acquired spatial map, the current position information of the self-moving robot is determined. This can avoid the point cloud sparsity problem caused by a single distance sensor or a single obstacle avoidance sensor, and can also avoid the problem of not being able to effectively identify objects with poor light reflectivity. Thus, a target motion plan can be generated based on the target two-dimensional map, feature recognition information, and current position information, improving the cleaning coverage and the accuracy of the obstacle avoidance strategy.

[0008] In some embodiments, the control component is further configured to: perform height calculation based on the detected light in the first target light surface that is reflected back, and perform distance calculation based on the light rays in the second target light surface that is reflected back; and perform three-dimensional coordinate reconstruction based on the height calculation result information and the distance calculation result information to obtain a type of point cloud frame.

[0009] In some embodiments, the feature recognition information includes 3D feature point landmark information; the obstacle avoidance sensor includes a left-eye RGB camera and a right-eye RGB camera, the left-eye RGB camera being mounted on a first side of the main body; the right-eye RGB camera being mounted on a second side of the main body; the first side and the second side are opposite sides of the distance sensor along the direction intersecting the movement direction; the control component is connected to both the left-eye RGB camera and the right-eye RGB camera and is configured to: obtain a binocular depth point cloud based on the binocular RGB images jointly acquired by the left-eye RGB camera and the right-eye RGB camera; and obtain 3D feature point landmark information based on the binocular depth point cloud and a type of point cloud frame.

[0010] In some embodiments, the control component is further configured to perform distortion correction and epipolar alignment processing on the binocular RGB images before acquiring the binocular depth point cloud, so that the image planes of the left and right RGB cameras are coplanar and aligned.

[0011] In some embodiments, the feature recognition information includes 3D coordinates; the control component is also configured to perform the following steps to acquire feature recognition information: extracting feature points and finding matching pairs based on the binocular RGB images jointly acquired by the left and right RGB cameras after distortion correction and epipolar alignment processing; for each pair of matching feature points, calculating their 3D coordinates in the world coordinate system using the projection matrices of the processed left and right RGB cameras through triangulation.

[0012] In some embodiments, the feature recognition information includes 3D coordinates; the control component is also configured to perform the following steps to acquire feature recognition information: extracting feature points and finding matching pairs based on the binocular RGB images jointly acquired by the left and right RGB cameras after distortion correction and epipolar alignment processing; for each pair of matching feature points, calculating their 3D coordinates in the world coordinate system using the projection matrices of the processed left and right RGB cameras through triangulation.

[0013] In some embodiments, the control component is further configured to: determine the rotation matrix and translation matrix between the distance sensor and the left and right RGB cameras based on the point cloud corresponding to multiple sets of calibration images acquired by the distance sensor and multiple sets of calibration images acquired by the left and right RGB cameras; correct the left RGB camera based on the rotation and translation matrices, convert the point cloud corresponding to the multiple sets of calibration images acquired by the distance sensor to the left RGB camera's viewpoint, and project the point cloud converted to the left RGB camera's viewpoint onto the left RGB camera's viewpoint based on the intrinsic parameter matrix, thereby obtaining an aligned depth image and a left RGB image.

[0014] In some embodiments, the control component is further configured to: control the distance sensor and the left-eye RGB camera to simultaneously capture images of spatial objects of different shapes in space; fit the point cloud projection onto the left-eye RGB camera based on the initial extrinsic matrix of the left eye and the extrinsic matrix of the right eye based on the initial extrinsic matrix of the left eye and the extrinsic matrix of the right eye based on the extrinsic matrix of the right eye; adjust the rotation angle and offset variable of the left eye RGB camera until the point cloud projection onto the RGB camera best fits the outline of the captured object; and determine the rotation matrix and translation matrix between the distance sensor and the left-eye RGB camera and the right eye RGB camera.

[0015] In some embodiments, the distance sensor is further configured to emit a third target light surface and a fourth target light surface having a second target angle, both of which are perpendicular to the plane of the fuselage body. The third target light surface is used to detect obstacles at a first height within the field of view of the distance sensor, and the fourth target light surface is used to detect obstacles at a second height within the field of view of the distance sensor. The first height and the second height are both heights relative to the plane.

[0016] In some embodiments, the feature recognition information includes at least one of point cloud, image frame, and image sequence.

[0017] In some embodiments, both the first target light surface and the second target light surface are laser beam scanning surfaces.

[0018] In some embodiments, the obstacle avoidance sensor includes at least one of a monocular RGB camera and a ground-to-ground optical flow sensor.

[0019] In some embodiments, the included angle of the first target is 100 degrees to 150 degrees.

[0020] In some embodiments, the angle between the second target light surface and the plane is related to the second target spacing.

[0021] In some embodiments, the self-moving robot includes a robotic vacuum cleaner.

[0022] Secondly, the present invention also provides a self-moving robot system, the self-moving robot system including a base station and the self-moving robot as described above; the base station is adapted to use with the self-moving robot, and the base station is provided with a placement area for the self-moving robot to recharge.

[0023] The self-moving robot and its system described in the above embodiments can effectively detect low-lying obstacles at close range in the robot's direction of movement using a distance sensor, avoiding blind spots in recognition. Based on a type of point cloud frame illuminating the target obstacle using a first and second target light surface, a target two-dimensional map of the self-moving robot can be constructed and updated. This target two-dimensional map includes the distance information between the target obstacle and the robot body. For example, the target obstacle may include at least one small object such as a toy, building block, or thin thread, enabling real-time mapping and localization of the self-moving robot. Based on the point cloud frame and the matching results between the feature recognition information obtained from the obstacle avoidance sensor and the pre-acquired spatial map, the current position information of the self-moving robot can be determined. This avoids the point cloud sparsity problem caused by a single distance sensor or a single obstacle avoidance sensor, and also avoids the problem of not being able to effectively identify objects with poor light reflectivity. Therefore, a target motion plan can be generated based on the target two-dimensional map, feature recognition information, and current position information, improving cleaning coverage and the accuracy of obstacle avoidance strategies. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram illustrating the working principle of a self-moving robot provided in one embodiment of this application;

[0026] Figure 2 This is a schematic diagram showing the distribution of distance sensors and obstacle avoidance sensors of a self-moving robot provided in one embodiment of this application;

[0027] Figure 3 This is a schematic diagram of the detection light surface of a distance sensor in a self-moving robot provided in one embodiment of this application;

[0028] Figure 4 for Figure 3 A schematic diagram of the field of view of a mid-range sensor in a plane parallel to the plane where the self-moving robot is located;

[0029] Figure 5 This is a schematic diagram illustrating the process of a control component generating a target motion plan in a self-moving robot according to one embodiment of this application;

[0030] Figure 6 This is a schematic diagram of the structure of a self-moving robot system provided in one embodiment of this application.

[0031] Explanation of reference numerals in the attached figures:

[0032] 10. Self-moving robot system; 101. Control components; 21. Distance sensor; 22. Obstacle avoidance sensor; 20. Self-moving robot; 30. Base station; 31. Placement area; 221. Left eye RGB camera; 222. Right eye RGB camera. Detailed Implementation

[0033] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings. Preferred embodiments of this application are shown in the drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of this application.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0035] When using the terms “including,” “having,” and “comprising” as described herein, another component may be added unless explicitly qualifying terms such as “only,” “consisting of,” etc. are used. Unless otherwise stated, singular terms may include plural forms and should not be construed as having a quantity of one.

[0036] It should be understood that although the terms “first,” “second,” etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of this application, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0037] In this application, unless otherwise expressly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a direct connection or an indirect connection through an intermediate medium, or they can refer to the internal connection of two elements or the interaction between two elements. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0038] With the development of cleaning equipment technology, self-propelled robots have become a commonly used cleaning device in people's lives.

[0039] Autonomous mobile robots typically use sensors to identify obstacles during their movement, but existing autonomous mobile robots have difficulty identifying and avoiding small obstacles during their movement, which can easily create blind spots in cleaning.

[0040] Furthermore, existing self-propelled robots are relatively tall, making it difficult for them to enter low-lying areas, resulting in low cleaning coverage.

[0041] Most navigation and sensing solutions in related technologies use 360° laser-direct-structuring (LDS) technology with a raised top. The raised top means that the robot is too tall to enter low areas, such as under furniture, which ultimately results in a low household cleaning coverage rate for robot vacuums.

[0042] To reduce the robot's height and achieve maximum cleaning coverage, leading robot vacuum manufacturers are trying to eliminate Laser-Direct-Structured (LDS) technology and use other sensors for navigation and perception. For example, a dual Time-of-Flight (TOF) distance measurement system (FOV) eliminates the overhead 360° LDS and replaces it with embedded dual TOF sensors. This provides a horizontal field of view (FOV) coverage greater than 200 degrees. However, this approach is costly, and TOF devices themselves have physical limitations such as multipath interference, low-reflection holes, high-reflection bumps, and point cloud expansion, which can lead to poor navigation and perception performance. Another approach is the embedded LDS solution, which embeds the overhead LDS sensor in front of the robot. However, the effective coverage angle of the embedded single-line radar is reduced (only 18°). For example, a single scan cannot achieve 360° panoramic coverage, requiring the body to rotate to better locate the spatial position of the body. Moreover, the forward space needs to be hollowed out, which occupies a large space, is prone to dirt, and is difficult to maintain. Another example is other lifting LDS solutions, which use a raised 360° LDS solution in normal environments and lower the LDS when the machine enters a low-lying area. The disadvantage is that the pure visual positioning in low-lying areas results in poor navigation. At the same time, the lifting structure occupies a large internal longitudinal space and is prone to damage. Another example is the Direct Time-of-Flight (dToF) and monocular RGB fusion solution. When ToF has large areas of low-reflection or high-reflection scenes with large holes in the point cloud or many missing points in fine line scenes, without the guidance of ToF depth, even if monocular RGB information is fused, it may not be able to accurately recover the depth of the holes. This will have a negative impact on Simultaneous Localization and Mapping (SLAM) and obstacle avoidance.

[0043] Specifically, dToF (direct Time of Flight) measures distance by measuring the time difference between the emission and reception of laser light; iToF (Indirect Time-of-Flight) is a 3D visual perception technology that calculates target distance by measuring the phase difference between the modulated light signal and the reflected light; and structured light depth mapping determines three-dimensional depth information by projecting a specifically coded light pattern (such as speckle, stripe, or dot matrix) onto the target object and analyzing the deformation of the reflected light.

[0044] This application employs a fusion scheme of distance sensor and obstacle avoidance sensor, capable of outputting point clouds with high-precision ranging at both near and far distances. The 3D point clouds can then be used for more precise navigation and obstacle avoidance. Compared to dual ToF, it improves the accuracy of near-range point clouds and the overall density of the point cloud; compared to embedded radar, it offers better navigation and obstacle avoidance performance; compared to laser-direct-structuring (LDS) technology, it exhibits stronger stability and superior navigation and obstacle avoidance performance; and compared to digital time of flight (dToF) and monocular RGB, it outputs complete and accurate point clouds in low-reflection and high-reflection void regions, resulting in better navigation and obstacle avoidance outcomes.

[0045] For example, a distance sensor is used to emit a first target light surface and a second target light surface with a first target angle. The first target light surface is parallel to the plane where the fuselage body is located, and the second target light surface faces the plane where the fuselage body is located, for example, the plane where the fuselage body is located is the ground or floor. The second target light surface is tilted downwards, and the angle between the second target light surface and the plane where the fuselage body is located is an acute angle. The first target light surface is used to detect one type of obstacle that is a first target distance from the fuselage body in the direction of movement of the fuselage body. The second target light surface is used to detect two types of obstacles that are a second target distance from the fuselage body in the direction of movement of the fuselage body. The first target distance is greater than or equal to the second target distance.

[0046] For example, the detection light in the first target light surface and the second target light surface can be a laser. Both the first target light surface and the second target light surface are laser beam scanning surfaces.

[0047] For example, the distance sensor can be installed directly in front of the robot vacuum, while the obstacle avoidance sensor can be installed on the same side of the robot body as the distance sensor, or on a different side.

[0048] Please refer to Figure 1In some embodiments, a self-moving robot is provided, including a body (not shown) and a distance sensor 21, an obstacle avoidance sensor 22, and a control component 101 embedded in the body. The distance sensor 21 is used to emit a first target light surface and a second target light surface with a first target angle α. The first target light surface is parallel to the plane where the body is located, and the second target light surface faces the plane where the body is located. The angle between the second target light surface and the first target light surface can be b. The first target light surface is used to detect one type of obstacle that is a first target distance from the body in the direction of movement of the body. The second target light surface is used to detect two types of obstacles that are a second target distance from the body in the direction of movement. The first target distance is greater than or equal to the second target distance. The first target distance can work together with the second target light surface to detect the height of the two types of obstacles. The obstacle avoidance sensor 22 is used to acquire feature recognition information of objects in the direction of movement of the robot body; the control component 101 is connected to both the distance sensor 21 and the obstacle avoidance sensor 22, and is configured to perform the following steps: acquire a type of point cloud frame illuminating the target obstacle by the first target light surface and the second target light surface; construct and update the target two-dimensional map based on the type of point cloud frame, the target two-dimensional map including the distance information between the target obstacle and the robot body; determine the current position information of the self-moving robot based on the matching result of the type of point cloud frame and feature recognition information with the pre-acquired spatial map; generate the target motion plan based on the target two-dimensional map, feature recognition information, and current position information.

[0049] For example, please continue to refer to Figure 1 Based on a first target light surface emitted by distance sensor 21 parallel to the plane of the robot body, obstacles of a type that are a first target distance from the robot body in the direction of movement are detected. The first target distance mainly depends on the light intensity in the first target light surface. Based on a second target light surface emitted by distance sensor 21 toward the plane of the robot body, two types of obstacles that are a second target distance from the robot body in the direction of movement are detected. The first target distance is greater than or equal to the second target distance. For example, by comprehensively utilizing the second target light surface, low obstacles at close range in the direction of movement of the mobile robot can be effectively detected, avoiding the formation of blind spots in recognition.

[0050] Furthermore, based on a type of point cloud frame illuminating the target obstacle using the first and second target light surfaces, a target two-dimensional map of the self-moving robot is constructed and updated according to the type of point cloud frame. The target two-dimensional map includes the distance information between the target obstacle and the robot body. For example, the target obstacle may include at least one small object such as a small toy, building block, or thin thread, so as to realize real-time mapping and localization of the self-moving robot. Based on the type of point cloud frame and the matching result of the feature recognition information obtained by the obstacle avoidance sensor 22 and the pre-acquired spatial map, the current position information of the self-moving robot is determined. This can avoid the point cloud sparsity problem caused by a single distance sensor 21 or a single obstacle avoidance sensor 22, and can also avoid the problem of not being able to effectively identify objects with poor light reflectivity. Thus, a target motion plan can be generated based on the target two-dimensional map, feature recognition information, and current position information, improving the cleaning coverage and the accuracy of the obstacle avoidance strategy.

[0051] Please refer to Figure 1 In some embodiments, the control component 101 is further configured to: perform height calculation based on the detected light in the first target light surface that is reflected back, and perform distance calculation based on the light in the second target light surface that is reflected back; and perform three-dimensional coordinate reconstruction based on the height calculation result information and the distance calculation result information to obtain a type of point cloud frame.

[0052] In some embodiments, the distance sensor may include a two-dimensional lidar that scans a horizontal plane parallel to the plane of the fuselage to form a first target light surface, obtaining obstacle information on the horizontal plane. This plane can be used to detect the horizontal contour of the obstacle, providing distance and angle in the horizontal direction, and can be used to calculate the projected position (x, y) coordinates of the obstacle on the horizontal plane. The distance sensor may also include another two-dimensional lidar that scans a horizontal plane perpendicular to the plane of the fuselage to form a second target light surface facing the plane of the fuselage, providing distance and angle in the vertical direction, and can be used to calculate the height (z coordinate) and horizontal distance of the obstacle. By combining the data from the two lidar surfaces, a point cloud in three-dimensional space can be obtained, thereby enabling three-dimensional environment modeling and localization. Information on low-lying obstacles perpendicular to the horizontal plane can be acquired, such as small toys, building blocks, and at least one small object like thin wire.

[0053] In some embodiments, we can obtain two sets of point clouds using the aforementioned two laser surfaces. Since the horizontal laser surface provides a horizontally annular point cloud, while the vertical laser surface provides a vertically fan-shaped point cloud, we can fuse them as follows: convert the point cloud of the horizontal laser surface into three-dimensional points (all points have the same height, H), and then merge it with the point cloud of the vertical laser surface. Note that the data from the two laser surfaces may have different timestamps, requiring time synchronization.

[0054] Please refer to Figures 2-3 In some embodiments, the feature recognition information includes 3D feature point landmark information; the obstacle avoidance sensor 22 includes a left-eye RGB camera 221 and a right-eye RGB camera 222, with the left-eye RGB camera 221 mounted on the first side of the main body and the right-eye RGB camera 222 mounted on the second side of the main body; the first side and the second side are opposite sides of the distance sensor 21 along the direction intersecting the movement direction; the control component 101 is connected to both the left-eye RGB camera 221 and the right-eye RGB camera 222 and is configured to: obtain a binocular depth point cloud based on the binocular RGB images jointly collected by the left-eye RGB camera 221 and the right-eye RGB camera 222; and obtain 3D feature point landmark information based on the binocular depth point cloud and a type of point cloud frame.

[0055] Please refer to Figures 2-4 In some embodiments, the first target angle α of the first target light surface emitted by the distance sensor 21 is 100-150 degrees, for example, the first target angle is 100 degrees, 120 degrees, 130 degrees, 140 degrees, or 150 degrees, etc. The field of view (FOV) of the distance sensor 21 in a plane parallel to the plane of the fuselage body is equal to the first target angle α.

[0056] Please continue to refer to this. Figure 3 In some embodiments, the angle b between the second target light surface and the first target light surface is related to the second target distance. For example, if you want to detect small obstacles within a closer distance in the direction of movement of the self-moving robot, you can increase the angle b accordingly; conversely, in order to further reduce the probability of colliding with obstacles, you can increase the angle b accordingly.

[0057] For example, the included angle b can be greater than or equal to 15 degrees and less than or equal to 60 degrees. For example, the included angle b can be 15 degrees, 20 degrees, 30 degrees, 40 degrees, 50 degrees, 60 degrees, etc.

[0058] Please refer to Figure 5 In some embodiments, the control component is also configured to perform the following steps:

[0059] Step S20: Acquire binocular RGB images jointly collected by the left eye RGB camera 221 and the right eye RGB camera 222, as well as a type of point cloud frame of the target obstacle illuminated by the first target light surface and the second target light surface;

[0060] Step S31: Obtain feature recognition information based on the binocular RGB images;

[0061] Step S32: Obtain the stereo depth point cloud based on the stereo RGB image;

[0062] Step S41: Construct and update a target two-dimensional map based on a type of point cloud frame. The target two-dimensional map includes the distance information between the target obstacle and the fuselage body.

[0063] Step S42: Obtain 3D feature point landmark information based on the binocular depth point cloud;

[0064] Step S50: Determine the current position information of the self-moving robot based on the matching results of a type of point cloud frame and feature recognition information with the pre-acquired spatial map; generate a target motion plan based on the target 2D map, feature recognition information, and current position information.

[0065] Please note that, unless otherwise implied by the order of signal flows, the order in which the steps appear does not imply a specific limitation on the order in which they are executed. In practical applications, the specific settings can be adjusted according to the order of signal flows and actual needs.

[0066] In some embodiments, before acquiring the binocular depth point cloud in step S32, distortion correction and epipolar alignment processing are performed on the binocular RGB image to make the image planes of the left RGB camera 221 and the right RGB camera 222 coplanar and aligned.

[0067] In some embodiments, the feature recognition information includes 3D coordinates; the control component is also configured to perform the following steps to acquire feature recognition information: extracting feature points and finding matching pairs based on the binocular RGB images jointly acquired by the left RGB camera 221 and the right RGB camera 222 after distortion correction and epipolar alignment processing; for each pair of matching feature points, calculating their 3D coordinates in the world coordinate system using the projection matrices of the processed left RGB camera 221 and the right RGB camera 222 through triangulation.

[0068] For example, the core principle of the technology for locating and recognizing objects based on binocular 3D feature points is to reconstruct the three-dimensional structure of the object through stereo vision geometry and feature point matching, thereby achieving localization and recognition.

[0069] For example, at least one of the algorithms SIFT (Scale-Invariant Feature Transform), SURF (SpeededUp Robust Features), and ORB (Oriented Fast and Rotated Brief) can be used to extract feature points from the binocular RGB images of the left-eye RGB camera 221 and the right-eye RGB camera 222. Then, corresponding feature point pairs in the left and right images are found, and feature point matching is performed. For each matched feature point pair, their horizontal coordinate difference (disparity) in the image is calculated. Based on the disparity and the camera's intrinsic parameters (focal length, baseline, etc.), the three-dimensional coordinates of the feature points are calculated through triangulation.

[0070] For example, the relationship between disparity and depth is: Z = (f × B) / d; where Z is the depth value of the feature point (vertical distance from the camera), f is the camera focal length (in pixels), B is the baseline length (actual physical distance), and d is the disparity = x1 - x2.

[0071] For example, the formula for calculating three-dimensional coordinates (X, Y) can be:

[0072] X = (x1 × Z) / f;

[0073] Y = (y1 × Z) / f;

[0074] (x1, y1) are the coordinates of the feature point in the left image.

[0075] In some embodiments, the control component is further configured to perform disparity matching using one or more algorithms such as PSMNet (Pyramid Stereo Matching Network), AANet (Adaptive Aggregation Network for Efficient Stereo Matching), or RAFT (Multilevel Recurrent Field Transforms for Stereo Matching) to determine the initial depth for both binoculars. The depth refinement optimization process inputs the left-eye RGB image, the initial depth for both binoculars, and a class of point cloud frames into a pre-designed deep learning model. The model is optimized through a data-driven approach, ultimately outputting a jointly optimized depth map with high accuracy at both near and far distances.

[0076] In some embodiments, the control component is further configured to: determine the rotation matrix and translation matrix between the distance sensor 21 and the left and right RGB cameras 221 and 222 based on the point cloud corresponding to the multiple sets of calibration images acquired by the distance sensor 21 and the multiple sets of calibration images acquired by the left and right RGB cameras 221 and 222; correct the left RGB camera 221 based on the rotation and translation matrices; convert the point cloud corresponding to the multiple sets of calibration images acquired by the distance sensor 21 to the view of the left RGB camera 221; and project the point cloud converted to the view of the left RGB camera 221 onto the view of the left RGB camera 221 based on the intrinsic parameter matrix to obtain the aligned depth image and the left RGB image.

[0077] Please refer to Figures 2-3 In some embodiments, the control component 101 is further configured to: control the distance sensor 21 and the left-eye RGB camera 221 to simultaneously capture spatial objects of different shapes in space; fit the point cloud onto the left-eye RGB camera 221 based on the left-eye initial extrinsic matrix and the right-eye extrinsic matrix of the left-eye RGB camera 221 and the right-eye RGB camera 222; adjust the rotation angle and offset variable of the left-eye RGB camera 221 until the point cloud projection onto the RGB best fits the outline of the captured object; and determine the rotation matrix and translation matrix between the distance sensor 21 and the left-eye RGB camera 221 and the right-eye RGB camera 222.

[0078] For example, based on the point cloud corresponding to multiple sets of calibration images acquired by the distance sensor 21, and the multiple sets of calibration images acquired by the left eye RGB camera 221 and the right eye RGB camera 222, the rotation matrix and translation matrix between the distance sensor 21 and the left eye RGB camera 221 and the right eye RGB camera 222 are determined according to the Zhang Zhengyou calibration method.

[0079] As an example, the calibration of the left-eye RGB camera 221 and the right-eye RGB camera 222 can be performed using a checkerboard pattern. The left-eye RGB camera 221 and the right-eye RGB camera 222 simultaneously capture multiple images of the calibration board. The Zhang Zhengyou calibration method is used to calculate the extrinsic matrix, the left and right eye intrinsic matrix, the rotation matrix R1, and the translation matrix T1 between the two cameras in OpenCV.

[0080] As an example, in the step of performing binocular RGB camera correction, the original RGB image can be distorted and aligned with the horizontal epipolar line based on the intrinsic and extrinsic parameters calibrated in the previous step. After correction, the same point in the binocular RGB image will be on the same horizontal line in both the left and right images.

[0081] In some embodiments, the distance sensor is further configured to emit a third target light surface and a fourth target light surface having a second target angle c. Both the third and fourth target light surfaces are perpendicular to the plane of the fuselage. The third target light surface is used to detect obstacles at a first height within the distance sensor's field of view, and the fourth target light surface is used to detect obstacles at a second height within the distance sensor's field of view. The first and second heights are both relative to the plane. This further reduces the obstacle avoidance blind spot and improves object height detection during the obstacle avoidance phase.

[0082] For example, the detection light in the third and fourth target optical surfaces can be laser beams. Both the third and fourth target optical surfaces are laser beam scanning surfaces.

[0083] In some embodiments, the feature recognition information includes at least one of point cloud, image frame, and image sequence.

[0084] In some embodiments, the obstacle avoidance sensor includes at least one of a monocular RGB camera and a ground-to-ground optical flow sensor.

[0085] In some embodiments, a monocular RGB camera working in conjunction with a distance sensor can also achieve real-time mapping, localization, and path planning, thereby improving the obstacle avoidance performance of the self-moving robot.

[0086] In some embodiments, the obstacle avoidance sensor includes a ground-based optical flow sensor, which provides spatial 3D feature point information instead of a binocular RGB camera. The optical flow sensor can rely on the camera to capture ground texture images and deduce the motion speed by calculating pixel displacement changes between consecutive frames. By detecting changes in ground texture, the relative motion trend between the robot and obstacles is determined, and the travel path is adjusted to avoid collisions. Combined with other sensors (such as LiDAR and infrared), an environmental map is constructed, and the cleaning route is optimized. This effectively improves cleaning coverage and the accuracy of the obstacle avoidance strategy.

[0087] In some embodiments, the self-moving robot includes a robotic vacuum cleaner. Compared to the top-mounted LDS solution, the distance sensor and binocular RGB camera fusion solution can reduce the robot's height, allowing it to enter more low-lying areas and improve cleaning coverage; moreover, the 3D area array perception solution provides more accurate positioning and obstacle avoidance; compared to the front and rear dual ToF solution, the distance sensor and binocular RGB camera fusion solution can overcome the inherent limitations of distance sensors, improving the SLAM and obstacle avoidance performance of the robotic vacuum cleaner; compared to the binocular solution, the distance sensor and binocular RGB camera fusion solution has significant advantages in weak textures and long-distance depth accuracy, thus resulting in better SLAM performance; compared to the embedded solution... The in-body LDS solution, a fusion of distance sensor and binocular RGB camera, is a 3D area array SLAM solution that achieves a low profile while providing better SLAM and obstacle avoidance performance. Compared to the pop-up LDS solution, the distance sensor and binocular RGB camera fusion solution has no mechanical lifting structure, resulting in a longer lifespan, and the SLAM performance in low-profile areas remains unaffected. Compared to the distance sensor and monocular RGB fusion solution, the distance sensor and binocular RGB camera fusion solution can handle scenes with large areas of low reflectivity, large areas of high reflectivity, or lack of depth such as fine lines, achieving better SLAM and obstacle avoidance performance.

[0088] Please refer to Figure 6 In some embodiments, this application also provides a self-moving robot system, which includes a base station 30 and a self-moving robot 20 as described above; the base station 30 is adapted to use with the self-moving robot 20, and the base station 30 is provided with a placement area 31 for the self-moving robot 20 to be recharged.

[0089] For example, a distance sensor can effectively detect low-lying obstacles at close range along the movement direction of a self-propelled robot, avoiding blind spots in recognition. Based on a type of point cloud frame illuminating the target obstacle using a first and second target light surface, a target 2D map of the self-propelled robot can be constructed and updated. This target 2D map includes the distance information between the target obstacle and the robot body; for example, the target obstacle can include at least one small object such as a toy, building block, or thin thread, enabling real-time mapping and localization of the self-propelled robot. Based on the point cloud frame and the matching results of feature recognition information obtained from an obstacle avoidance sensor with a pre-acquired spatial map, the current position information of the self-propelled robot can be determined, avoiding the point cloud sparsity problem caused by a single distance sensor or obstacle avoidance sensor, and also avoiding the inability to effectively identify objects with poor light reflectivity. Therefore, a target motion plan can be generated based on the target 2D map, feature recognition information, and current position information, improving cleaning coverage and the accuracy of obstacle avoidance strategies.

[0090] Here, the deep learning model of this application fully integrates the advantages of high close-range accuracy from a distance sensor and high long-range accuracy from a binocular RGB camera. For scenes with large areas of low reflectivity, large areas of high reflectivity, or thin lines where depth is lacking, this application, combined with binocular depth estimation, can output accurate and complete point clouds. This can significantly improve the navigation and obstacle avoidance performance of robotic vacuum cleaners.

[0091] In some embodiments, the deep learning model is ultimately implemented on an NPU (Neural Processing Unit) chip, and through the acceleration of the NPU, the actual running frame rate reaches more than 10 frames per second. During use, a type of point cloud frame from the distance sensor and a stereo RGB image are used as input, and the model outputs an optimized depth map.

[0092] In some embodiments, the binocular depth point cloud obtained from binocular RGB images can help the self-moving robot effectively identify small objects within the blind spot of the distance sensor, such as small toys, building blocks, thin lines, and black objects with low laser reflectivity. The distance sensor and the binocular RGB camera work together to complement each other, significantly improving the obstacle avoidance performance of the robot vacuum. The embedded, low-profile design, while maintaining the robot's ultra-thin form, combined with obstacle avoidance sensors such as the binocular RGB camera, achieves accurate and efficient navigation, with better obstacle avoidance capabilities, ensuring navigation and positioning while preventing collisions.

[0093] It should be understood that, although the flowcharts involved in the embodiments described above are not identical, Figure 5 The steps in the code are displayed sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows.

[0094] Unless otherwise expressly stated herein, there is no strict order in which these steps are performed; they can be performed in other orders. Furthermore, the flowcharts described in the embodiments above... Figure 5 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0095] In some embodiments, a self-moving robot is provided, the self-moving robot including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the method steps performed by the control components described above.

[0096] In some embodiments, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method steps of the control component described above.

[0097] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these. The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combinations of these technical features are not contradictory, they should be considered within the scope of this specification.

[0098] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A self-moving robot, characterized by, Includes the fuselage body and a distance sensor, obstacle avoidance sensor and control components embedded in the fuselage body; The distance sensor is used to emit a first target light surface and a second target light surface with a first target angle. The first target light surface is parallel to the plane where the fuselage body is located, and the second target light surface faces the plane. The first target light surface is used to detect a type of obstacle that is a first target distance from the fuselage body in the direction of movement of the fuselage body. The second target light surface is used to detect a type of obstacle that is a second target distance from the fuselage body in the direction of movement of the fuselage body. The first target distance is greater than or equal to the second target distance. The obstacle avoidance sensor is used to acquire feature recognition information of objects in the direction of movement of the fuselage body; The control component is connected to both the distance sensor and the obstacle avoidance sensor, and is configured as follows: Acquire a type of point cloud frame of the target obstacle illuminated by the first target light surface and the second target light surface; A target two-dimensional map is constructed and updated based on the aforementioned point cloud frames, wherein the target two-dimensional map includes the distance information between the target obstacle and the fuselage body; Based on the matching results of the point cloud frame and the feature recognition information with the pre-acquired spatial map, the current position information of the self-moving robot is determined; A target motion plan is generated based on the target 2D map, the feature recognition information, and the current location information.

2. The self-moving robot according to claim 1, characterized in that, The control component is also configured to: The height is calculated based on the light detected in the first target light surface that is reflected back, and the distance is calculated based on the light rays detected in the second target light surface that is reflected back. Based on the height calculation results and the distance calculation results, three-dimensional coordinate reconstruction is performed to obtain the first type of point cloud frame.

3. The self-moving robot according to claim 1, characterized in that, The feature recognition information includes 3D feature point road sign information; The obstacle avoidance sensor includes: The left-eye RGB camera is mounted on the first side of the main body of the camera. The right-eye RGB camera is mounted on the second side of the main body; the first side and the second side are opposite sides of the distance sensor along the direction intersecting the direction of movement; The control component is connected to both the left and right RGB cameras and is configured as follows: Based on the binocular RGB images jointly acquired by the left and right RGB cameras, obtain the binocular depth point cloud; 3D feature point landmark information is obtained based on the binocular depth point cloud and the type of point cloud frame.

4. The self-moving robot according to claim 3, characterized in that, The control component is also configured to: Before acquiring the stereo depth point cloud, distortion correction and epipolar alignment processing are performed on the stereo RGB images to make the image planes of the left RGB camera and the right RGB camera coplanar and aligned.

5. The self-moving robot according to claim 4, characterized in that, The feature recognition information includes 3D coordinates; The control component is also configured to perform the following steps to acquire the feature recognition information: Based on the binocular RGB images jointly acquired by the left and right RGB cameras after distortion correction and epipolar alignment processing, feature points are extracted and matching pairs are found. For each pair of matched feature points, the 3D coordinates in the world coordinate system are calculated using the projection matrices of the processed left and right RGB cameras through triangulation.

6. The self-moving robot according to claim 4, characterized in that, The feature recognition information includes 3D coordinates; The control component is also configured to perform the following steps to acquire the feature recognition information: Based on the binocular RGB images jointly acquired by the left and right RGB cameras after distortion correction and epipolar alignment processing, feature points are extracted and matching pairs are found. For each pair of matched feature points, the 3D coordinates in the world coordinate system are calculated using the projection matrices of the processed left and right RGB cameras through triangulation.

7. The self-moving robot according to claim 4, characterized in that, The control component is also configured to: Based on the point cloud corresponding to multiple sets of calibration images acquired by the distance sensor and multiple sets of calibration images acquired by the left and right RGB cameras, the rotation and translation matrices between the distance sensor and the left and right RGB cameras are determined. According to the rotation matrix and translation matrix, the left eye RGB camera is corrected, and the point cloud corresponding to the multiple sets of calibration images collected by the distance sensor is converted to the view of the left eye RGB camera. According to the intrinsic parameter matrix, the point cloud converted to the view of the left eye RGB camera is projected onto the view of the left eye RGB camera to obtain the aligned depth image and the left eye RGB image.

8. The self-moving robot according to claim 7, characterized in that, The control component is also configured to: The distance sensor and the left-eye RGB camera are controlled to simultaneously capture images of spatial objects of different shapes in space. Based on the initial extrinsic matrix of the left eye RGB camera and the extrinsic matrix of the right eye RGB camera, the point cloud is fitted and projected onto the left eye RGB camera; Adjust the rotation angle and offset variables of the left RGB camera until the point cloud projection onto the RGB best matches the outline of the object being photographed. Then determine the rotation matrix and translation matrix between the distance sensor, the left RGB camera, and the right RGB camera.

9. The self-moving robot according to claim 1, characterized in that, The distance sensor is further configured to emit a third target light surface and a fourth target light surface having a second target angle. The third target light surface and the fourth target light surface are both perpendicular to the plane on which the fuselage body is located. The third target light surface is used to detect obstacles at a first height within the field of view of the distance sensor, and the fourth target light surface is used to detect obstacles at a second height within the field of view of the distance sensor. The first height and the second height are both heights relative to the plane.

10. The self-moving robot according to claim 1, characterized in that, Includes at least one of the following features: The feature recognition information includes at least one of point cloud, image frame, and image sequence; Both the first target light surface and the second target light surface are laser beam scanning surfaces; The obstacle avoidance sensor includes at least one of a monocular RGB camera and a ground-to-ground optical flow sensor; The included angle of the first target is 100 degrees to 150 degrees; The angle between the second target light surface and the plane is related to the second target spacing; The self-moving robot includes a floor sweeper.

11. A self-moving robot system, characterized in that, include: The self-moving robot as described in any one of claims 1 to 10; A base station is adapted for use with the self-moving robot, and the base station is provided with a placement area for the self-moving robot to recharge.

Images