Autonomous mobile device, control method and device thereof and storage medium

Abstract

The present disclosure relates to an autonomous mobile device, a control method and apparatus thereof, and a storage medium, the method comprising: a shooting step of shooting an image of a working environment of the autonomous mobile device; a distance measurement step: obtaining point cloud data of a working environment of the autonomous mobile device; an identification step: inputting the image into an image object identification model for identification to obtain an object in the image and a space range of a detection frame where the object is located; an obtaining step: obtaining the number of the point cloud data in the space range of the detection frame; a judgment step: judging whether the quantity of the point cloud data exceeds a preset quantity threshold value or not; and an object positioning step of calculating the spatial position of the detection frame according to the point cloud data in the spatial range of the detection frame under the condition that the number of the point cloud data exceeds a preset number threshold. Therefore, wrong object recognition or wrong object position judgment can be avoided, the accuracy of obstacle recognition and obstacle avoidance of the autonomous mobile equipment is improved, and thus the intelligent degree of the autonomous mobile equipment is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of smart home technology, and in particular to an autonomous mobile device, a control method and device thereof, and a storage medium. Background Art

[0002] Autonomous mobile devices refer to devices that move autonomously within a set work area to perform preset tasks. Currently, autonomous mobile devices generally include but are not limited to cleaning robots (such as smart sweepers, smart floor scrubbers, and window cleaning robots), companion mobile robots (such as smart electronic pets and nanny robots), service mobile robots (such as reception robots in hotels, inns, and meeting places), industrial inspection intelligent equipment (such as power inspection robots, smart forklifts, etc.), and security robots (such as household or commercial intelligent guard robots). As autonomous mobile devices become more and more intelligent, some autonomous mobile devices can use deep learning models to identify objects in the environment. However, object recognition based on existing deep learning models often results in misidentification, such as incorrect identification of the category of an object, which leads to incorrect judgment of whether the object actually exists; if the object actually exists, the object's location is incorrectly judged.

[0003] Therefore, how to improve the accuracy of object recognition on autonomous mobile devices is one of the problems that autonomous mobile devices need to solve in the future. Summary of the invention

[0004] In view of this, the present disclosure proposes an autonomous mobile device, a control method and device thereof, and a storage medium, so as to improve the accuracy of object recognition.

[0005] According to a first aspect of the present disclosure, a control method for an autonomous mobile device is provided, the method comprising: a shooting step of shooting an image of a working environment of the autonomous mobile device; a ranging step of acquiring point cloud data of the working environment of the autonomous mobile device; an identification step of inputting the image into an image object recognition model for identification to obtain an object in the image and a spatial range of a detection frame where the object is located; an acquisition step of acquiring the number of point cloud data within the spatial range of the detection frame; a judgment step of judging whether the number of point cloud data exceeds a preset number threshold; and an object positioning step of calculating the spatial position of the detection frame based on the point cloud data within the spatial range of the detection frame when the number of point cloud data exceeds the preset number threshold.

[0006] According to a second aspect of the present disclosure, a control device for an autonomous mobile device is provided, the device comprising: a shooting unit, for shooting an image of a working environment of the autonomous mobile device; a ranging unit, for acquiring point cloud data of the working environment of the autonomous mobile device; a recognition unit, for inputting the image into an image object recognition model for recognition, and obtaining an object in the image and a spatial range of a detection frame where the object is located; an acquisition unit, for acquiring the number of point cloud data within the spatial range of the detection frame; a judgment unit, for judging whether the number of point cloud data exceeds a preset number threshold; and an object positioning unit, for calculating the spatial position of the detection frame based on the point cloud data within the spatial range of the detection frame when the number of point cloud data exceeds the preset number threshold.

[0007] According to a third aspect of the present disclosure, a control device for an autonomous mobile device is provided, the device comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the control method of the first aspect above when executing the instructions stored in the memory.

[0008] According to a fourth aspect of the present disclosure, an autonomous mobile device is provided, comprising: the control device of the second aspect or the third aspect; and a motion unit for moving the autonomous mobile device on a working surface in a working area.

[0009] According to a fifth aspect of the present disclosure, a non-temporary computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor, the processor is enabled to execute the control method of the first aspect.

[0010] Compared with the prior art which only improves the recognition accuracy of the image object recognition model for objects in the image, the present invention combines the image object recognition model with point cloud data, and uses the point cloud data to perform secondary confirmation on the objects recognized by the image object recognition model, thereby avoiding misidentification of objects or misjudgment of the object's position, improving the accuracy of the autonomous mobile device in identifying and avoiding obstacles, and thereby improving the intelligence of the autonomous mobile device.

[0011] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0013] Figure 1 A flow chart illustrating a control method of an autonomous mobile device according to an exemplary embodiment.

[0014] Figure 2a A schematic diagram is shown of an autonomous mobile device according to an exemplary embodiment.

[0015] Figure 2b~2c A schematic diagram showing an application scenario of an autonomous mobile device according to an exemplary embodiment.

[0016] Figure 3a to 3e Angles involved in the spatial range of a detection frame according to an exemplary embodiment are shown.

[0017] Figures 4a to 4d An example diagram illustrating an image displayed with a detection frame according to an exemplary embodiment.

[0018] Figure 5 A flow chart illustrating a control method of an autonomous mobile device according to an exemplary embodiment.

[0019] Figure 6 A block diagram illustrating a control apparatus of an autonomous mobile device according to an exemplary embodiment. DETAILED DESCRIPTION

[0020] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0021] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0022] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0023] In the related art, image object recognition models (such as deep learning models) are often used for intelligent object recognition. Specifically, the captured image can be input into a trained deep learning model, and the deep learning model outputs the object or object category in the image. However, the deep learning model itself cannot obtain the distance information or coordinate information of the object in the image. Therefore, when the deep learning model is set on the autonomous mobile device and the environmental objects in the working area (usually indoor space) are identified as the autonomous mobile device moves, misidentification will occur, resulting in the identification of the passable area as an obstacle (such as identifying the texture of the floor as an obstacle, or identifying the interface between the floor and the carpet as an obstacle), or even if the environmental object is accurately identified, the position or distance of the environmental object is misjudged, causing the autonomous mobile device to still collide with the environmental object.

[0024] For example, when using a deep learning model to identify environmental objects, a threshold may be identified at a certain location in the working area. However, the object at that location is actually not a threshold but a carpet pattern. For an autonomous mobile device, a carpet pattern is not an obstacle. If only a deep learning model is used for object recognition, misidentification may occur, and the autonomous mobile device may perform inappropriate processing such as obstacle avoidance and crossing based on the result of the misidentification.

[0025] This disclosure proposes Figure 1 The control method of the autonomous mobile device shown in the figure is executed by the autonomous mobile device. The present disclosure does not impose any specific restrictions on the execution time of the control method of the autonomous mobile device disclosed in the present disclosure. In one possible implementation, when the autonomous mobile device is just turned on, the autonomous mobile device may execute Figure 1 Of course, the control method of this embodiment can also be executed after a period of time has passed since the autonomous mobile device was turned on, when the execution condition is met, or when the user actively sends an execution instruction.

[0026] It should be understood that autonomous mobile devices refer to intelligent mobile devices that perform predetermined tasks on the working surface in the working area, and may include but are not limited to: cleaning robots (Robot Vacuum Cleaner, RVC), such as: intelligent sweeping machines, intelligent floor cleaning machines, window cleaning robots, etc.; logistics robots, handling robots, etc.; weeding robots, ice scraping robots, etc.; companion mobile robots, such as: intelligent electronic pets, nanny robots; service mobile robots, such as: reception robots in hotels, inns, and meeting places; industrial inspection intelligent devices, such as power inspection robots, intelligent forklifts, etc.; security robots, such as: household or commercial intelligent guard robots, etc. It should be noted that autonomous mobile devices are not limited to the aforementioned types, and any intelligent device with autonomous mobile functions should belong to the autonomous mobile devices described in this embodiment.

[0027] exist Figure 1 In the illustrated embodiment, the control method of the autonomous mobile device may include the following steps:

[0028] Step S110 (photographing step): photographing an image of the working environment of the autonomous mobile device.

[0029] In this embodiment, an autonomous mobile device equipped with a camera runs in a working environment and obtains images in the working environment through the camera. Figure 2a As shown, the camera C1 can be fixedly mounted on the autonomous mobile device 100 and shoot in a certain direction (such as the front of the forward direction of the autonomous mobile device) to obtain an image of the working environment corresponding to the direction. The forward direction of the autonomous mobile device mentioned above refers to the forward direction of movement of the autonomous mobile device during normal operation, such as Figure 2b and Figure 2c The positive x-axis direction (the direction from left to right as shown by the arrow in the figure); the so-called "normal operation" refers to the movement of the autonomous mobile device when performing a task, which is different from the abnormal normal operation of the autonomous mobile device such as retreating or swinging in the obstacle avoidance or escape mode. The above-mentioned working environment image sometimes includes objects or parts of objects in the working environment (such as Figure 2b or Figure 2cThe objects shown in the figure 200) are, for example, furniture, walls, electrical appliances, shoes, socks, clothes, trash cans, cartons, wires, thresholds, and other objects in the working area in front of the autonomous mobile device that are within the camera shooting range, but the objects in the image mentioned in the present disclosure generally do not include the ground, that is, even if there is a ground in the image, the ground is not an obstacle that hinders the operation of the autonomous mobile device, so the ground is not regarded as an object in the image, and the ground is not recognized by the subsequent image object recognition model. In other words, since the technical solution of the present disclosure is to identify the object in the image so that the autonomous mobile device avoids the object in the working environment, the object in the image mentioned in the present disclosure is actually an obstacle, that is, an object that can hinder or limit the operation of the autonomous mobile device, and therefore does not include the ground.

[0030] It should be understood that the camera used in the shooting step S110 can be just an ordinary camera without using a depth camera, so a cheap monocular camera can be selected instead of a complex and expensive binocular camera or structured light camera. Of course, the present disclosure can also use a depth camera (such as a binocular camera or a structured light camera, etc.) or other types of cameras. The present disclosure does not limit the type of camera used.

[0031] Step S120 (distance measurement step): obtaining point cloud data of the working environment of the autonomous mobile device.

[0032] During the operation of the autonomous mobile device, point cloud data is collected and updated in real time as the autonomous mobile device moves. When the autonomous mobile device is moving, the spatial coordinates of the autonomous mobile device in the absolute coordinate system can be obtained through a positioning system or positioning algorithm (such as a Simultaneous Localization And Mapping (SLAM) algorithm), and the autonomous mobile device can calculate the spatial coordinates of the point cloud data around it (which can be relative coordinates or absolute coordinates).

[0033] In some embodiments, a distance measuring device such as a laser distance sensor (LDS) or a laser radar (LIDAR) may be used (e.g. Figure 2a and 2cThe rotating laser radar 300 that emits a non-horizontal laser beam as shown emits a laser detection signal to the working environment of the autonomous mobile device. When the laser detection signal irradiates an object in the working environment (the object in this disclosure refers to an obstacle), a light spot is formed on the surface of the object (i.e., a laser point generated on the surface of the object when it is irradiated on the object in the working environment and blocked by the object in the working environment), and the spatial coordinates of each laser point are calculated to obtain point cloud data, that is, the point cloud data is a light spot / laser point with known spatial coordinates. The spatial coordinates of each laser point in the point cloud data in the absolute coordinate system can be calculated based on the spatial coordinates (relative coordinates) of each laser point in the relative coordinate system of the autonomous mobile device, combined with the coordinates of the autonomous mobile device in the absolute coordinate system, as shown in FIG. Figure 2a and Figure 2c shown.

[0034] In some embodiments, in addition to using the above-mentioned laser rangefinder to obtain point cloud data, for example, the following can also be used: Figure 2a and Figure 2b The line laser L1 and / or L2 shown as being arranged approximately perpendicular to the ground emits a line laser approximately perpendicular to the ground in front of the autonomous mobile device 100 to obtain point cloud data: when the line laser irradiates the obstacle in front, a light spot will be formed on the surface of the obstacle (i.e., a laser point generated on the surface of the obstacle when irradiating the obstacle in front and being blocked by the obstacle in front), and then an image acquisition unit arranged on the autonomous mobile device takes an image of the working environment in front of the autonomous mobile device 100 to collect an image containing the light spot; then the processing unit calculates the spatial coordinates of each light spot (i.e., laser point) from the image to obtain point cloud data, i.e., the point cloud data is a light spot or laser point with known spatial coordinates. The spatial coordinates of the point cloud data here can be relative coordinates or absolute coordinates. The distance measuring device composed of the above-mentioned line laser and image acquisition unit in this example can be called a 3D laser scanning device (3D Laser Scanner). For specific calculation methods, please refer to Laser Scanner Calibration Dependency on the Line Detection Method (Martens, et al. Elektronika Ir Elektrotechnika (2015).), Low-Cost 3d Laser Scanning In Air Or Water Using Self-Calibrating Structured Light (Bleier M, A Nüchter. 3D ARCH 2017.2017.).

[0035] The present disclosure does not limit the device and method for acquiring point cloud data.

[0036] It should be noted that the image acquisition unit for acquiring point cloud data in the distance measurement step S120 and the camera in the shooting step S110 can be the same device or different devices; if the two are the same device, its parameters need to be adjusted in different steps to capture different images, thereby achieving different uses in different steps. Figure 2a As shown, in the shooting step S110, the camera C1 shoots an image of the working environment of the autonomous mobile device (a normal image within the visible light wavelength range) and provides it to the image object recognition model for object recognition. In the ranging step S120, the image acquisition unit, which is also the camera C1 as part of the 3D laser scanning device in the above embodiment, changes its parameters and filters the visible light band through a filter so that it can only shoot the infrared band light emitted by the line laser L1 and / or L2 to calculate the spatial coordinates of the point cloud data.

[0037] Step S130 (recognition step): the above image is input into the image object recognition model for recognition, and the object in the image and the spatial range of the detection frame where the object is located are obtained.

[0038] In the related art, image object recognition models are generally used for target detection. The target detection involved in the present disclosure is object detection or obstacle detection, that is, the detection of obstacles in the working environment that may block or limit the operation of autonomous mobile devices. Exemplarily, the target detection model can be a YOLO model. The YOLO model is one of the basic network models and is a neural network model that realizes the positioning and recognition of targets through convolutional neural networks (English: Convolutional Neural Networks, abbreviated as: CNN). In one possible implementation, the image object recognition model may include but is not limited to a deep learning model. Exemplarily, the image object recognition model is a YOLOv5 model. For details of the above-mentioned related technologies, please refer to Redmon J, Farhadi A. YOLOv3: An Incremental Improvement [J]. arXiv e-prints, 2018. DOI: 10.48550 / arXiv.1804.02767, or refer to https: / / zhuanlan.zhihu.com / p / 150646196, https: / / blog.csdn.net / yegeli / article / details / 109861867, https: / / zhuanlan.zhihu.com / p / 94986199.

[0039] The detection frame of the present disclosure refers to the image object recognition model identifying the image input therein. If the image includes an identifiable object (an obstacle in the present disclosure), a detection frame will be used to frame the object in the output image, so that the object in the image is completely contained in the detection frame, such as Figure 4a , 4b The white frame shown in the image. The spatial range of the detection frame is related to the coordinates of the detection frame in the image coordinate system. The coordinates of the detection frame in the image coordinate system can be expressed as (x, y) in the Cartesian coordinate system. Using the intrinsic and extrinsic matrix of the camera, the coordinate position (x, y) of the detection frame in the image coordinate system can be converted to the spatial coordinates of the detection frame in the three-dimensional space coordinate system, such as the three-dimensional space coordinates (α, β, r) in the spherical coordinate system. For details on coordinate conversion, please refer to Lecture 3 "Rigid Body Motion in Three-Dimensional Space" and Lecture 5 "Camera and Image" in "Fourteen Lectures on Visual SLAM: From Theory to Practice" (Gao Xiang et al.; Publishing House of Electronics Industry; 2017.4.1), or refer to https: / / zhuanlan.zhihu.com / p / 144307108. The spatial range of the detection frame can be determined by the range of the lateral angle α and the range of the longitudinal angle β.

[0040] In a first implementation of this embodiment, the spatial range of the detection frame includes a first angle range consisting of a horizontal lower limit angle θ1, a horizontal upper limit angle θ2, and a vertical lower limit angle θ3. Figure 3a and Figure 3b As shown in , the range of the horizontal angle α of the detection frame is (θ1, θ2), that is, θ1<α<θ2, then the horizontal lower limit angle θ1 and the horizontal upper limit angle θ2 limit the range of the detection frame in the horizontal direction (horizontal direction or in the xy plane in the Cartesian three-dimensional space coordinate system), as shown in Figure 3a As shown; the range of the longitudinal angle β of the detection frame is (θ3, +∞), that is, β>θ3, then the longitudinal lower limit angle θ3 limits the range of the detection frame in the longitudinal direction (the direction perpendicular to the horizontal plane or the direction where the z-axis is located in the Cartesian three-dimensional space coordinate system), as shown Figure 3b As shown. It can be seen that the first angle range of the detection frame in this implementation is a U-shaped range defined by the horizontal lower limit coordinate θ1, the horizontal upper limit coordinate θ2 and the vertical lower limit coordinate θ3, as shown in FIG. Figure 4a As shown. For example, in Figure 4aIn the image shown, the angle range defined by the white frame line on the left, the white frame line on the right, and the white frame line on the bottom of the detection frame corresponds to the first angle range. Since the monocular camera cannot obtain the distance information (i.e., depth information) of the object in the image through the image, the spatial range of the detection frame defined by the horizontal lower limit coordinate θ1, the horizontal upper limit coordinate θ2, and the vertical lower limit coordinate θ3 can only be an angle range, and the definite spatial position of the detection frame or the object in it cannot be obtained.

[0041] Based on the three-dimensional space coordinate transformation between the spherical coordinate system and the Cartesian coordinate system, as a variation of the first implementation method mentioned above, that is, the second implementation method of this embodiment, the longitudinal angle β and the longitudinal lower limit angle θ3 of the detection frame in the spherical coordinate system are respectively transformed into the height coordinate Z and the longitudinal lower limit height H1 in the Cartesian three-dimensional space coordinate system, that is, Z>H1, as shown in Figure 3c As shown, the lower limit height H1 in the longitudinal direction defines the range of the detection frame in the longitudinal direction (the direction perpendicular to the horizontal plane or the direction where the z-axis is located in the Cartesian three-dimensional space coordinate system). Figure 3c , H1<0. Therefore, in this implementation, the spatial range of the detection frame includes a first width range (θ1, θ2) defined by a lateral lower limit angle θ1 and a lateral upper limit angle θ2, and a first height range (H1, +∞) defined by a longitudinal lower limit height H1. Since a monocular camera cannot obtain the distance information (i.e., depth information) of an object therein through an image, the spatial range of the detection frame consisting of the first width range (θ1, θ2) defined by the lateral lower limit coordinate θ1 and the lateral upper limit coordinate θ2 and the first height range (H1, +∞) defined by the longitudinal lower limit height H1 can still only be an angle range, and the definite spatial position of the detection frame or the object therein cannot be obtained.

[0042] In a third implementation of this embodiment, the spatial range of the detection frame includes a second angle range consisting of a horizontal lower limit angle θ1, a horizontal upper limit angle θ2, a vertical lower limit angle θ3, and a vertical upper limit angle θ4. Figure 3d and Figure 3e As shown in , the range of the horizontal angle α of the detection frame is (θ1, θ2), that is, θ1<α<θ2, then the horizontal lower limit angle θ1 and the horizontal upper limit angle θ2 limit the range of the detection frame in the horizontal direction (horizontal direction or in the xy plane in the Cartesian three-dimensional space coordinate system), as shown in Figure 3d As shown in the figure, the range of the longitudinal angle β of the detection frame is (θ3, θ4), that is, θ3<β<θ4, then the longitudinal lower limit angle θ3 and the longitudinal upper limit angle θ4 limit the range of the detection frame in the longitudinal direction (the direction perpendicular to the horizontal plane or the direction where the z-axis is located in the Cartesian three-dimensional space coordinate system), as shown in Figure 3eAs shown. It can be seen that the second angle range of the detection frame in this implementation is a rectangular frame consisting of the horizontal lower limit coordinate θ1, the horizontal upper limit coordinate θ2, the vertical lower limit coordinate θ3, and the vertical upper limit coordinate θ4, as shown in FIG. Figure 4b As shown in the white box in FIG. Figure 4b In the image shown, the angle range defined by the white frame line on the left, the white frame line on the right, the white frame line on the top, and the white frame line on the bottom of the detection frame corresponds to the second angle range. Since the monocular camera cannot obtain the distance information (i.e., depth information) of the object in the image through the image, the spatial range of the detection frame defined by the horizontal lower limit angle θ1, the horizontal upper limit angle θ2, the vertical lower limit angle θ3, and the vertical upper limit angle θ4 can only be an angle range, and the definite spatial position of the detection frame or the object in the detection frame cannot be obtained.

[0043] It should be noted that the present disclosure requires that the photographing step S110 be performed first and then the identification step S130 be performed, but the execution order of the ranging step S120, the photographing step S110 and the identification step S130 is not limited. Figure 1 As shown, the shooting step S110, the ranging step S120, and the identification step S130 are performed in sequence. Alternatively, the ranging step S120 may be performed first and then the shooting step S110 and the identification step S130. Alternatively, the shooting step S110 and the ranging step S120 may be performed simultaneously (in this case, the image acquisition unit for acquiring point cloud data described in the ranging step S120 and the camera in the shooting step S110 should be different devices) and then the identification step S130 is performed.

[0044] Step S140 (acquisition step): Acquire the number of point cloud data within the spatial range of the above detection frame.

[0045] In the above-mentioned ranging step 120, the point cloud data of the working environment of the autonomous mobile device has been obtained, that is, the laser points irradiated on the obstacles in the working environment and their spatial coordinates (spherical coordinate system (α, β, r) or Cartesian coordinate system (x, y, z), and the two can be interchanged through coordinate transformation). Then, through the above-mentioned recognition step 130, the spatial range of the detection frame that frames the object in the image is obtained from the image input into the image object recognition model. Based on the above description, it can be seen that the spatial range of the detection frame obtained from the image can only provide an angle range in a certain direction due to the lack of depth information of the detection frame, and the specific spatial position of the detection frame cannot be determined. However, since the point cloud data includes the spatial coordinates of the point cloud, it is possible to determine which point cloud data are within the spatial range of the detection frame based on the spatial coordinates of the point cloud data, and therefore the number of point cloud data within the spatial range of the detection frame can be counted.

[0046] In the first implementation of the present embodiment, Figure 3a , 3b As shown in Figure 4a, the spatial range of the detection frame includes a first angle range defined by a horizontal lower limit angle θ1, a horizontal upper limit angle θ2, and a vertical lower limit angle θ3. Based on the spatial coordinates of the point cloud data, a first point cloud data set whose spatial coordinates meet the horizontal angle range (θ1, θ2) and the vertical angle range (θ3, +∞) of the detection frame is selected from the point cloud data of the working environment of the autonomous mobile device. The number of point cloud data in the first point cloud data set is counted to obtain the number of point cloud data within the spatial range of the detection frame.

[0047] In the above second implementation of this embodiment, Figure 3c and 4a As shown, the spatial range of the detection frame includes a first width range (θ1, θ2) defined by the lateral lower limit angle θ1 and the lateral upper limit angle θ2, and a first height range (H1, +∞) defined by the longitudinal lower limit height H1. Although this implementation is only the first implementation from coordinate transformation, it has practical significance. In practical applications, for the convenience of calculation, the ground height is usually set as the z-axis zero point of the Cartesian coordinate system, that is, the ground height Z0 = 0. Considering that obstacles in the actual working environment must be higher than the ground (therefore, they can hinder autonomous mobile devices running on the ground in the working environment), the longitudinal lower limit height can be set to H1 = 0, and only the point cloud data with its height coordinate Z>H1 can be screened, that is, only the laser points reflected by objects above the ground will be considered as point cloud data that meet the detection frame conditions (of course, the premise is that the spatial coordinates of the point cloud data must also be within the first width range of the detection frame). Of course, this implementation method is not limited to setting H1=0. The maximum height that the autonomous mobile device can cross can also be used as the longitudinal lower limit height H1. For example, if the autonomous mobile device can cross an obstacle with a height of 5 cm, the longitudinal lower limit height H1 can be set to 5 cm, and only point cloud data with a Z value greater than 5 cm can be filtered, because only point cloud data greater than H1 can block the autonomous mobile device.

[0048] For example, Figure 4d As shown, the white horizontal line is the line where H1=0. Although yellow laser points are displayed in the image, it can be seen from the figure that these yellow laser points are located below the white horizontal line, that is, their height coordinates Z are all less than 0. Therefore, these yellow laser points do not meet the condition that the height coordinate of the point cloud data is greater than the longitudinal lower limit height H1 of the detection frame. Therefore, these yellow laser points are not point cloud data that meet the detection frame.

[0049] For example, point cloud data that meets the spatial range of the above detection frame can also be screened by other ground segmentation algorithms. For details, see M. Himmelsbach, Fv Hundelshausen and H.-J. Wuensche, "Fast segmentation of 3D point clouds for ground vehicles," 2010 IEEE Intelligent Vehicles Symposium, La Jolla, CA, USA, 2010, pp. 560-565, doi: 10.1109 / IVS.2010.5548059.

[0050] Therefore, by detecting whether the height coordinates of the point cloud data meet the first height range, the objects identified in the image can be screened for a second time, and the longitudinal lower limit coordinate θ3 or the longitudinal lower limit height H1 can be used to screen out the misjudgment of the image object recognition model caused by floor patterns, thereby improving the reliability of obstacle recognition. If it is found that the obstacle on the ground identified by the image object recognition model is a misidentification rather than a real obstacle, the autonomous mobile device can ignore the image and continue to operate normally, so the autonomous mobile device does not need to execute the obstacle avoidance mode or escape mode for the identified carpet pattern.

[0051] Independent of the step of determining the point cloud data within the first height range (H1, +∞) of the detection frame, the point cloud data within the first angle range (θ1, θ2) of the detection frame can be determined according to the spatial coordinates of the point cloud data before, after, or simultaneously with the step. Then the point cloud data within both the first height range (H1, +∞) of the detection frame and the first angle range (θ1, θ2) of the detection frame is the first point cloud data set within the spatial range of the detection frame. The point cloud data in the first point cloud data set is counted to obtain the number of point cloud data within the spatial range of the detection frame.

[0052] In the third implementation of this embodiment, Figure 3d , 3e As shown in FIG4b, based on the first angle range of the first implementation, the upper limit coordinate θ4 in the longitudinal direction can be further limited, thereby limiting the range of the frame eyebrow on the detection frame. That is, the spatial range of the detection frame includes a second angle range consisting of a horizontal lower limit angle θ1, a horizontal upper limit angle θ2, a vertical lower limit angle θ3, and a vertical upper limit angle θ4, as shown in FIG4b. Figure 3d and 3eBased on the spatial coordinates of the point cloud data, a second point cloud data set whose spatial coordinates meet the horizontal angle range (θ1, θ2) and the vertical angle range (θ3, θ4) of the detection frame is selected from the point cloud data of the working environment of the autonomous mobile device. The number of point cloud data in the second point cloud data set is the number of point cloud data within the spatial range of the detection frame, as shown in Figure 4b The pink laser point is contained in the detection box in the image.

[0053] Similar to the second implementation of this embodiment, the lateral lower limit angle, lateral upper limit angle, longitudinal lower limit angle, and / or longitudinal upper limit angle in the third implementation can also be transformed to transform the spatial coordinates of the point cloud data between the spherical coordinate system and the Cartesian coordinate system, and the spatial coordinates of the point cloud data or the spatial range of the detection frame are displayed with appropriate spatial coordinates according to the principle of convenient processing, and the spatial coordinates of the point cloud data are compared with the spatial range of the detection frame, so as to filter out the point cloud data within the spatial range of the detection frame and count the number of these point cloud data. The specific method is referred to the aforementioned implementation method, which will not be repeated here.

[0054] It should be understood that the present disclosure requires performing the photographing step S110 and the identifying step S130 before the acquiring step S140 .

[0055] After the number of point cloud data within the spatial range is obtained, the following step S150 may be performed.

[0056] Step S150 (judgment step): judge whether the amount of point cloud data exceeds a preset amount threshold. If the amount of point cloud data exceeds the preset amount threshold, it is judged that the object actually exists in the working environment, and the following step S160 is continued.

[0057] In practice, if there are too few point cloud data within the spatial range of the detection frame of a certain image, it may be caused by the error of the ranging device itself, and it does not necessarily mean that there is a real object in the detection frame. Therefore, a certain amount of point cloud data is required to confirm that there is a real object in the detection frame identified by the image object recognition model. Therefore, a quantity threshold can be set in advance (for example, according to the statistical law of system errors or based on experience), and whether there is a real object (obstacle) in the working environment corresponding to the detection frame can be judged according to whether the number of point cloud data within the spatial range of the detection frame exceeds the preset quantity threshold. If the number of point cloud data within the spatial range of the detection frame does not exceed the preset quantity threshold, it can be considered that the image object recognition model recognizes the object incorrectly, that is, the object recognized by the image object recognition model does not really exist, but is misidentified, such as identifying the floor texture or carpet pattern as an obstacle; at this time, the object recognized by the image object recognition model is ignored, or the image is deleted. Accordingly, the autonomous mobile device is not misled by the misidentified object and does not perform subsequent response operations (such as the avoidance steps and marking steps introduced later). Only when the number of point cloud data within the spatial range of the detection frame exceeds the preset threshold, the image object recognition model is considered to have correctly identified the object. In other words, the object recognized by the image object recognition model actually exists in the working environment. The autonomous mobile device will then treat the object as a real obstacle and perform subsequent response steps (such as avoidance steps, marking steps, etc.).

[0058] In some embodiments, illustratively, different quantity thresholds a may be preset for different objects or object categories. That is, the preset quantity threshold should correspond to the object category. In a possible implementation, objects are divided into different object categories according to the area and shape of the objects in the image. For example, for two-dimensional surface objects such as clothing, shoes, and walls (two-dimensional surface objects can be understood as objects that usually present a two-dimensional shape and have a certain area in the image taken from the perspective of the autonomous mobile device), the corresponding quantity threshold may be preset to a1=100; for strip-shaped objects such as thresholds (different from two-dimensional surface objects, thresholds, fan seats, etc. usually present strips with a certain width in the image taken from the perspective of the autonomous mobile device, so they are defined as strip-shaped objects), the corresponding quantity threshold may be preset to a2=40; for linear objects such as wires, cables, and curtain tassels (different from two-dimensional surface objects and strip-shaped objects, linear objects such as wires and cables usually present unclear and intermittent thin lines in the image taken from the perspective of the autonomous mobile device, so they are defined as linear objects), the corresponding quantity threshold may be preset to a3=10. For example, if the image object recognition model recognizes that the object in the detection frame of the image is a strip object, such as a threshold, and the number of point cloud data A within the spatial range of the detection frame is 50, since 50>40 (corresponding to a2 of the strip object in the above example), it can be considered that the image object recognition model correctly recognizes the object in the detection frame, and the object needs to be treated as an obstacle of the strip object for subsequent response operations. In the above example, if the image object recognition model recognizes that the object in the detection frame of the image is still a strip object, and the number of point cloud data A within the spatial range of the detection frame is 30, since 30<40 (corresponding to a2 of the strip object in the above example), it is considered that the image object recognition model recognizes the object in the detection frame incorrectly, even if 30>10 here (corresponding to a3 of the linear object in the above example), the object recognized by the image object recognition model is still not considered to be real, and the image will be ignored or deleted, and the autonomous mobile device will not perform subsequent response operations based on the incorrectly recognized object.In the above example, if the image object recognition model recognizes that the object in the detection frame of the image is still a strip-shaped object, and the number of point cloud data A within the spatial range of the detection frame is 110, since 110>40 (corresponding to a2 of the strip-shaped object in the above example), it can be considered that the image object recognition model is correct in recognizing the object in the detection frame. Even if 110>100 here (corresponding to a1 of the two-dimensional surface object in the above example), it can still be considered that the object recognized by the image object recognition model is a strip-shaped object (that is, as long as the number of point cloud data exceeds the preset number threshold corresponding to the category of the object being recognized, the image object recognition model recognizes the category of the object; this is only an example and should not be regarded as a limitation of the present disclosure), and the object is treated as an obstacle to the strip-shaped object for subsequent response operations.

[0059] For example, Figure 4a , 4b As shown, the image object recognition model recognizes that the object in the white detection frame in the image is clothing, and its object category is a two-dimensional surface object. It is calculated that the number of point cloud data within the spatial range of the detection frame in the figure is 200, and 200>100 (corresponding to a1 of the two-dimensional surface object in the above example). Therefore, the number of point cloud data within the spatial range of the detection frame exceeds the corresponding quantity threshold a1. It can be considered that the image object recognition model is correct for the object in the detection frame, and it is judged that the recognized clothing actually exists in the working environment of the autonomous mobile device.

[0060] For example, Figure 4c As shown, the image object recognition model recognizes that the object in the white detection frame in the image is a threshold, and its object category is a strip object. The number of point cloud data within the spatial range of the detection frame in the figure is calculated to be 3, 3<40 (corresponding to a2 of the strip object in the above example). Therefore, the number of point cloud data within the spatial range of the detection frame does not exceed the corresponding number threshold a2. Therefore, it is considered that the image object recognition model is wrong in recognizing the object in the detection frame, and it is judged that the recognized threshold does not actually exist in the working environment of the autonomous mobile device.

[0061] It should be understood that the setting rules of the quantity threshold of the present disclosure should not be limited to setting the quantity threshold corresponding to the object category, and the quantity threshold can be set according to the actual application situation, for example, the quantity threshold can be set according to the object size, object material, etc. In addition, the present disclosure does not specifically limit the execution order of the step of presetting the quantity threshold and other steps. In fact, the step for presetting the quantity threshold can be executed first.

[0062] Step S160 (object positioning step): when the amount of the point cloud data exceeds a preset amount threshold, the spatial position of the detection frame is calculated based on the point cloud data within the spatial range of the detection frame. The spatial position of the detection frame represents the spatial position of the object in the detection frame in the working environment, that is, the spatial position of the object (obstacle) is the spatial position of the detection frame where the object is located.

[0063] As mentioned above, since the image has no depth information, it is difficult to obtain the accurate spatial position of the object in the detection frame from the image captured in the shooting step S110. When the number of point cloud data exceeds the preset number threshold, that is, it is recognized that the object in the detection frame is an obstacle that actually exists in the working environment. At this time, exemplarily, the spatial position of the point cloud data within the spatial range of the detection frame represents the spatial position of the detection frame in the image, and also represents the spatial position of the object in the detection frame in the working environment. The representative points of the detection frame (points that can represent the key positions of the detection frame, such as the geometric centroid and symmetry center of the detection frame) are calculated by statistical methods, or some / some qualified point cloud data are selected as representative points of the detection frame, and the distance between the spatial positions of the representative points of these detection frames and the autonomous mobile device (referred to as the representative point distance) is calculated, and the distance between the detection frame and the autonomous mobile device (referred to as the estimated distance) is calculated based on the representative point distance, and the spatial position of the detection frame is calculated based on the estimated distance and the spatial range of the detection frame.

[0064] It should be noted that, considering that the object recognized by the image object recognition model does not actually exist in the working environment of the autonomous mobile device, the recognized object does not actually exist, and the autonomous mobile device will not perform subsequent response operations based on the wrongly recognized object, and there is no need to calculate the spatial position of the wrongly recognized object in the working environment, and naturally there is no need to calculate the spatial position of the detection frame that frames the wrongly recognized object. Therefore, the object positioning step S160 can be executed to calculate the spatial position of the detection frame only when it is determined in the judgment step S150 that the number of point cloud data exceeds the preset number threshold, thereby reducing the operation time and saving computing power.

[0065] The following describes how to calculate the spatial position of the detection frame based on the point cloud data: For detection frames whose number of point cloud data within the spatial range exceeds a preset threshold, the laser point with the smallest distance from the center of the camera can be selected from these point cloud data as the representative point of the detection frame, and the representative point distance between the representative point of the detection frame and the center of the camera C1 is calculated, and the projection of the representative point distance on the optical axis of the camera is used as the distance r between the detection frame and the camera (which can be used as the estimated distance between the detection frame and the autonomous mobile device). The spatial position of the detection frame can be calculated based on the above estimated distance r and the spatial range of the detection frame. By way of example, the third implementation method described above is combined with Figure 3d and Figure 3e For example, the spatial position of the detection frame can be calculated based on the estimated distance r, the angles θ1 and θ2 involved in the above-mentioned horizontal angle range, and the angles θ3 and θ4 involved in the vertical angle range. For example, the following formula can be used to calculate Figure 3d and Figure 3e The three-dimensional spatial coordinates of the four corner points (b, c, d, e) of the detection frame in the Cartesian coordinate system shown in the figure are: b(x, y, z) = r×(1, tan(θ2), tan(θ3)); c(x, y, z) = r×(1, tan(θ1), tan(θ3)); d(x, y, z) = r×(1, tan(θ1), tan(θ4)); e(x, y, z) = r×(1, tan(θ2), tan(θ4)). Determining the spatial coordinates of the four corner points (b, c, d, e) of the detection frame also determines the spatial position of the detection frame.

[0066] Of course, the representative point of the detection frame can also be selected in other ways, such as calculating the average spatial coordinates of all the point cloud data within the spatial range of the detection frame according to their spatial coordinates, and using the average spatial coordinates as the representative point of the detection frame, or using the spatial coordinates of the actual point cloud data closest to the average spatial coordinates as the representative point of the detection frame; or averaging the lateral coordinates (such as the x-axis coordinates) of the two point cloud data with the farthest lateral span within the spatial range of the detection frame as the lateral coordinates of the representative point of the detection frame, averaging the longitudinal coordinates (such as the z-axis coordinates) of the two point cloud data with the farthest longitudinal span within the spatial range of the detection frame as the longitudinal coordinates of the representative point of the detection frame, averaging the depth coordinates (such as the z-axis coordinates) of the two point cloud data with the farthest longitudinal span within the spatial range of the detection frame as the longitudinal coordinates of the representative point of the detection frame, and averaging the depth coordinates (such as the z-axis coordinates) of the two point cloud data with the farthest depth span within the spatial range of the detection frame (i.e., the direction of the camera optical axis) as the longitudinal coordinates of the representative point of the detection frame. Figure 3a , Figure 3b , Figure 3d , Figure 3e The r coordinates in the detection box are averaged as the depth coordinates of the representative points of the detection box to obtain the spatial coordinates of the representative points of the detection box. The present disclosure does not limit the method of selecting the representative points of the detection box.

[0067] It should be noted that although the spatial position of the detection frame is used in this embodiment to represent the spatial position of the object, it should not be limited to using the spatial position of the detection frame to represent the spatial position of the object. This is because the object can also be represented by the point cloud data in the detection frame that is close to the autonomous mobile device, that is, a part of the valid point cloud data in the detection frame represents the object. In fact, the spatial position of the detection frame itself is not important, and its purpose is to find the object in it and determine its position.

[0068] Therefore, considering that the autonomous mobile device can collect and update point cloud data, and calculate the information of objects in the surrounding environment based on the point cloud data (such as calculating the relative coordinates and absolute coordinates of the object, and calculating the distance between the autonomous mobile device and each object in the surrounding environment, etc.), according to the present disclosure, the autonomous mobile device uses the image object recognition model to recognize the object in the image and the spatial range of the detection frame where the object is located, and judges whether the object actually exists in the working environment based on whether the number of point cloud data in the spatial range exceeds a preset number threshold. If the object actually exists in the working environment, the spatial position of the detection frame is calculated based on the point cloud data in the spatial range of the detection frame. Therefore, the present disclosure combines the image object recognition model with the point cloud data, and uses the point cloud data to perform a secondary confirmation on the object recognized by the image object recognition model, thereby avoiding the misidentification of the object or the misjudgment of the object position, improving the accuracy of the autonomous mobile device in identifying obstacles and avoiding obstacles, and thus improving the intelligence of the autonomous mobile device.

[0069] In some embodiments, such as Figure 5 As shown, after determining the spatial position of the detection frame (i.e., the spatial position of the object in the detection frame), the control method of the autonomous mobile device may further include step S170 (avoidance step): controlling the autonomous mobile device to avoid the spatial position of the detection frame during operation. At this time, the autonomous mobile device may be controlled to operate in an avoidance mode, illustratively, for example, it may first retreat a set distance, then rotate a set angle, change the running direction, and then run forward. Exemplarily, the avoidance mode may also be to first retreat a set distance, and then run along a forward arc in a curve to bypass the obstacles identified in front.

[0070] In some embodiments, such as Figure 5As shown, after calculating the spatial position of the detection frame (i.e., the spatial position of the object in the detection frame), the control method of the autonomous mobile device may further include step S180 (marking step): marking the information of the object in the map of the working environment, the information of the object including the name or category of the object and the spatial position of the object; the spatial position of the object is the spatial position of the detection frame where the object is located. Of course, other information such as the distance of the object from the autonomous mobile device, the size of the object, etc. may also be marked, so that the user can obtain information about the objects in the working environment based on the map.

[0071] It should be noted that the present disclosure does not limit the execution order of the avoidance step S170 and the marking step S180. Figure 5 As shown, the avoidance step S170 is performed first and then the marking step S180 is performed. Alternatively, the marking step S180 may be performed first and then the avoidance step S170. Alternatively, the avoidance step S170 and the marking step S180 may be performed simultaneously.

[0072] Figure 6 A block diagram showing a control device of an autonomous mobile device according to an exemplary embodiment is shown. Figure 6 As shown, the control device 600 may include a shooting unit 610, a distance measuring unit 620, a recognition unit 630, an acquisition unit 640, a judgment unit 650 and an object positioning unit 660. The shooting unit 610 is used to shoot an image of the working environment of the autonomous mobile device. The distance measuring unit 620 is used to obtain point cloud data of the working environment of the autonomous mobile device. The recognition unit 630 is connected to the shooting unit 610, and is used to input the image into the image object recognition model for recognition, and obtain the object in the image and the spatial range of the detection frame where the object is located. The acquisition unit 640 is connected to the recognition unit 630 and the distance measuring unit 620, and is used to obtain the number of point cloud data within the spatial range of the detection frame. The judgment unit 650 is connected to the acquisition unit 640, and is used to judge whether the number of point cloud data exceeds a preset number threshold. The object positioning unit 660 is connected to the judgment unit 650 and the distance measuring unit 620, and is used to calculate the spatial position of the detection frame according to the point cloud data within the spatial range of the detection frame when the number of point cloud data exceeds the preset number threshold.

[0073] In a possible implementation, the control device 600 may also include a marking unit (not shown) for marking the information of the object in the map of the working environment, the information of the object including the name or category of the object and the spatial position of the object; the spatial position of the object is the spatial position of the detection box where the object is located.

[0074] In a possible implementation, the control device 600 may further include an avoidance unit (not shown) configured to control the autonomous mobile device to avoid the spatial position of the detection frame during operation.

[0075] In a possible implementation, the spatial range of the detection frame includes a first angle range consisting of a horizontal lower limit angle, a horizontal upper limit angle, and a vertical lower limit angle.

[0076] In a possible implementation, the spatial range of the detection frame includes a first width range defined by a lower horizontal limit angle and an upper horizontal limit angle, and a first height range defined by a lower vertical limit height.

[0077] In a possible implementation, the spatial range of the detection frame includes a second angle range consisting of a horizontal lower limit angle, a horizontal upper limit angle, a vertical lower limit angle, and a vertical upper limit angle.

[0078] In one possible implementation, the preset quantity threshold corresponds to the category of the object recognized by the image object recognition model; the judgment unit 650 can be configured to: determine whether the quantity of the point cloud data exceeds the preset quantity threshold corresponding to the category of the object.

[0079] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0080] Therefore, compared with the prior art that only improves the recognition accuracy of the image object recognition model for objects in the image, the present invention combines the image object recognition model with point cloud data, and uses the point cloud data to perform secondary confirmation of the objects recognized by the image object recognition model, thereby avoiding misidentification of objects or misjudgment of the object's position, improving the accuracy of autonomous mobile devices in identifying and avoiding obstacles, and thereby improving the intelligence of autonomous mobile devices.

[0081] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A control method for an autonomous mobile device, characterized in that: include: A photographing step of photographing an image of a working environment of the autonomous mobile device; A ranging step of obtaining point cloud data of the working environment of the autonomous mobile device; A recognition step of inputting the image into an image object recognition model for recognition to obtain the object in the image and the spatial range of the detection frame where the object is located; An acquisition step of acquiring the amount of point cloud data within the spatial range of the detection frame; A judging step, judging whether the amount of the point cloud data exceeds a preset amount threshold; The object positioning step calculates the spatial position of the detection frame based on the point cloud data within the spatial range of the detection frame when the amount of the point cloud data exceeds the preset amount threshold.

2. The control method according to claim 1, characterized in that: After the object positioning step, the method further includes: The marking step marks the information of the object in the map of the working environment, wherein the information of the object includes the name or category of the object and the spatial position of the object; the spatial position of the object is the spatial position of the detection frame where the object is located.

3. The control method according to claim 1, characterized in that: After the object positioning step, the method further includes: The avoidance step controls the autonomous mobile device to avoid the spatial position of the detection frame during operation.

4. The control method according to claim 1, characterized in that: The spatial range of the detection frame includes a first angle range consisting of a horizontal lower limit angle, a horizontal upper limit angle, and a vertical lower limit angle.

5. The control method according to claim 1, characterized in that: The spatial range of the detection frame includes a first width range defined by a horizontal lower limit angle and a horizontal upper limit angle, and a first height range defined by a vertical lower limit height.

6. The control method according to claim 1, characterized in that: The spatial range of the detection frame includes a second angle range consisting of a horizontal lower limit angle, a horizontal upper limit angle, a vertical lower limit angle, and a vertical upper limit angle.

7. The control method according to claim 1, characterized in that: The preset quantity threshold corresponds to the category of the object recognized by the image object recognition model; The judging step includes judging whether the quantity of the point cloud data exceeds a preset quantity threshold corresponding to the category of the object.

8. A control device for an autonomous mobile device, characterized in that: include: A photographing unit, used to photograph an image of a working environment of the autonomous mobile device; A distance measuring unit, used to obtain point cloud data of the working environment of the autonomous mobile device; A recognition unit, used for inputting the image into an image object recognition model for recognition, and obtaining the object in the image and the spatial range of the detection frame where the object is located; An acquisition unit, used to acquire the amount of point cloud data within the spatial range of the detection frame; A judging unit, used to judge whether the amount of the point cloud data exceeds a preset amount threshold; The object positioning unit is used to calculate the spatial position of the detection frame according to the point cloud data within the spatial range of the detection frame when the amount of the point cloud data exceeds the preset amount threshold.

9. A control device for an autonomous mobile device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the control method according to any one of claims 1 to 7 when executing the instructions stored in the memory.

10. An autonomous mobile device, characterized in that: include: a motion unit for moving the autonomous mobile device on a work surface in a work area; as well as A control device according to claim 8 or 9.

11. A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor, enables the processor to perform the control method according to any one of claims 1 to 7.

Images