Object detection method and apparatus, cleaning device control method and apparatus, and cleaning device

Abstract

An object detection method, a cleaning device control method, a cleaning device control apparatus, an object detection apparatus, and a cleaning device. The object detection method is applied to a cleaning device, the cleaning device comprises a robotic arm and a gripper, the gripper is arranged on the end of the robotic arm distant from a body of the cleaning device, and the robotic arm is used for cooperating with the gripper to grab an object. The object detection method comprises: acquiring a target image comprising the gripper; and on the basis of the target image, detecting whether there is a grabbed target object in the gripper.

Description

Object detection method and device, cleaning equipment control method and device, and cleaning equipment Cross-reference to Related Applications

[0001] This application claims priority to Chinese Patent Application No. 2024113225952, filed on September 20, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to the technical field of cleaning equipment control, and in particular, relates to an object detection method, a cleaning equipment control method and device, an object detection device, a program product, a medium, and a cleaning equipment. BACKGROUND

[0003] Cleaning equipment (such as a sweeper or a mop) has been widely used in family life today, which can replace users to complete housework, bringing great convenience to users. In addition to cleaning the ground, the cleaning equipment can also arrange the sundries, and in the process of arranging the sundries, the cleaning equipment is usually configured with a gripper to grab objects. It can be understood that ensuring the accurate detection of the cleaning equipment on the grabbed objects is the key to the cleaning equipment grabbing the objects. However, there is currently no mature object detection scheme for the accurate detection of the cleaning equipment on the grabbed objects. Therefore, how to improve the accuracy of the cleaning equipment in detecting the grabbed objects is a technical problem to be solved. SUMMARY

[0004] Embodiments of the present disclosure provide an object detection method, a cleaning equipment control method and device, an object detection device, a program product, a medium, and a cleaning equipment, thereby at least to some extent improving the accuracy of the cleaning equipment in detecting the grabbed objects.

[0005] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0006] According to a first aspect of embodiments of the present disclosure, an object detection method is provided, the method being applied to a cleaning equipment, the cleaning equipment comprising a mechanical arm and a gripper, the gripper being arranged at an end of the mechanical arm away from a machine body of the cleaning equipment, the mechanical arm being configured to cooperate with the gripper to grab objects, the method comprising: acquiring a target image containing the gripper; and based on the target image, detecting whether a grabbed target object exists in the gripper.

[0007] In some embodiments, the cleaning equipment further comprises an image acquisition device, and the acquiring the target image containing the gripper comprises: acquiring the target image containing the gripper acquired by the image acquisition device.

[0008] In some embodiments, the detecting, based on the target image, whether the target object is grabbed in the gripper, comprises: obtaining a mechanical arm position parameter of the mechanical arm, and determining foreground data in the target image according to the mechanical arm position parameter, wherein the foreground data is image data that moves with the gripper; and detecting, based on the foreground data, whether the target object is grabbed in the gripper.

[0009] In some embodiments, the detecting, based on the foreground data, whether the target object is grabbed in the gripper, comprises: determining first pose data of the gripper in the foreground data; obtaining second pose data of the gripper in a closed state that is predetermined; and comparing the first pose data and the second pose data to detect whether the target object is grabbed in the gripper.

[0010] In some embodiments, the detecting, based on the target image, whether the target object is grabbed in the gripper, comprises: segmenting each target object in the target image by an image segmentation algorithm, the target object at least including the gripper; and detecting, according to a pose relationship between the target object other than the gripper and the gripper, whether the target object is grabbed in the gripper.

[0011] According to the technical solution provided in the present disclosure, since the target image obtained contains the gripper, the image analysis on the target image can accurately detect whether the target object is grabbed in the gripper.

[0012] According to a second aspect of the embodiments of the present disclosure, a cleaning device control method is provided, the cleaning device comprising a mechanical arm and a gripper, the gripper being arranged at an end of the mechanical arm away from a machine body of the cleaning device, the mechanical arm being configured to cooperate with the gripper to grab an object, the method comprising: obtaining an initial position of a target object, and controlling the cleaning device to grab the target object based on the initial position; detecting whether the target object is grabbed in the gripper; and if the target object is grabbed in the gripper, controlling the cleaning device to carry the target object to a target position.

[0013] In some embodiments, the detecting whether the target object is grabbed in the gripper comprises: detecting whether the target object is grabbed in the gripper according to the method as described in the first aspect of the embodiments of the present disclosure.

[0014] In some embodiments, the method further comprises: determining a change value of the relative pose between the target object and the gripper according to target images containing the gripper acquired at different times during the process in which the cleaning device carries the target object, the change value being used to represent a degree of slippage of the target object in the gripper; and detecting whether the target object in the gripper has fallen or is at risk of falling according to the change value.

[0015] In some embodiments, the determining of the change value of the relative pose between the target object and the gripper according to target images containing the gripper acquired at different times comprises: determining foreground data in the target images acquired at different times to obtain a plurality of frames of foreground data; and determining the change value of the relative pose between the target object and the gripper according to the plurality of frames of foreground data.

[0016] In some embodiments, the determining of the change value of the relative pose between the target object and the gripper according to target images containing the gripper acquired at different times comprises: segmenting the target object and the gripper in each target image by an image segmentation algorithm; and determining the change value of the relative pose between the target object and the gripper according to the pose relationship between the target object and the gripper in each target image.

[0017] In some embodiments, the detecting of whether the target object in the gripper has fallen or is at risk of falling according to the change value comprises: if the change value is greater than a first change threshold and less than a second change threshold, determining a risk value of the target object falling from the gripper according to the change value, and / or a distance between the cleaning device and the target location, and / or a category of the target object; and detecting whether the target object in the gripper is at risk of falling according to the risk value.

[0018] In some embodiments, the detecting of whether the target object in the gripper is at risk of falling according to the risk value comprises: if the risk value is less than or equal to a preset risk threshold, determining that the target object in the gripper is not at risk of falling; and if the risk value is greater than the preset risk threshold, determining that the target object in the gripper is at risk of falling.

[0019] In some embodiments, the method further comprises: if the change value is less than or equal to the first change threshold, determining that the target object in the gripper is not at risk of falling.

[0020] In some embodiments, the method further comprises: if the change value is greater than or equal to the second change threshold, determining that the target object in the gripper has fallen.

[0021] In some embodiments, the method further comprises: if the target object in the gripper is at risk of falling, controlling the cleaning device to perform an object re-grab action; if the target object in the gripper falls, controlling the cleaning device to perform an object search action.

[0022] In some embodiments, the controlling the cleaning device to perform the object re-grab action comprises: controlling the cleaning device to stop moving and place the target object grabbed by the gripper on the ground; and controlling the cleaning device to re-grab the target object.

[0023] In some embodiments, the cleaning device further comprises a photoelectric sensor, or a force feedback sensor, or a current detection sensor, and the detecting whether the target object is grabbed in the gripper comprises: acquiring first sensing data collected by the photoelectric sensor, or the force feedback sensor, or the current detection sensor; and detecting whether the target object is grabbed in the gripper according to the first sensing data.

[0024] In some embodiments, the method further comprises: during the cleaning device carrying the target object, acquiring second sensing data collected by the photoelectric sensor, or the force feedback sensor, or the current detection sensor; detecting whether the target object in the gripper falls according to the second sensing data; and if the target object in the gripper falls, controlling the cleaning device to perform an object search action.

[0025] In some embodiments, the controlling the cleaning device to perform the object search action comprises: acquiring a first position of the cleaning device when the falling of the target object is detected; estimating a falling area of the target object according to the first position; controlling the cleaning device to search for the target object in the falling area; and if the cleaning device searches for the target object in the falling area, controlling the cleaning device to re-grab the target object and carry the re-grabbed target object to a target position.

[0026] In some embodiments, the estimating the falling area of the target object according to the first position comprises: acquiring a second position of the cleaning device when the falling of the target object is last detected; and estimating the falling area of the target object according to the first position and the second position.

[0027] In some embodiments, the controlling the cleaning device to search for the target object in the drop area comprises: determining at least one target search point in the drop area; controlling the cleaning device to move to each target search point and to observe a circle around each target search point respectively to search for the target object.

[0028] In some embodiments, the determining at least one target search point in the drop area comprises: determining a plurality of groups of search points in the drop area, wherein each group of search points comprises at least one search point; calculating a search efficiency of each group of search points, and determining a search point in a group of search points with the highest search efficiency as the at least one target search point.

[0029] In some embodiments, the controlling the cleaning device to search for the target object in the drop area comprises: if the cleaning device searches for an object in the drop area, acquiring second appearance attribute information of the searched object; acquiring first appearance attribute information of the target object recorded by the cleaning device when the target object is first picked up; calculating a similarity between the searched object and the target object based on the first appearance attribute information and the second appearance attribute information; and determining an object with the highest similarity and exceeding a preset similarity threshold as the target object.

[0030] In some embodiments, the method further comprises: if the cleaning device does not search for the target object in the drop area, controlling the cleaning device to perform other tasks and caching first appearance attribute information of the target object and current task information of the target object, wherein the first appearance attribute information and the task information are used for the cleaning device to search for the target object during performance of the other tasks.

[0031] In some embodiments, the method further comprises: if the cleaning device searches for the target object during performance of the other tasks, controlling the cleaning device to pick up the target object again and to carry the target object picked up again to a target position; and if the cleaning device does not search for the target object during performance of the other tasks, triggering an error prompt.

[0032] The cleaning equipment control scheme based on the present disclosure first detects whether the target object is present in the gripper before controlling the cleaning equipment to move the target object from the initial position to the target position. It can be understood that if the target object is not present in the gripper, the cleaning equipment will not be controlled to perform the walking action from the initial position to the target position. It can be seen that the cleaning equipment control scheme based on the present disclosure can ensure that the target object is moved from the initial position to the target position, thereby improving the reliability of the cleaning equipment during the moving task.

[0033] According to a third aspect of the embodiments of the present disclosure, an object detection device is provided, which is applied to a cleaning equipment, the cleaning equipment comprising a mechanical arm and a gripper, the gripper being arranged at one end of the mechanical arm away from a machine body of the cleaning equipment, the mechanical arm being configured to cooperate with the gripper to grasp an object, the device comprising: a first acquisition unit configured to acquire a target image containing the gripper; and a first detection unit configured to detect, based on the target image, whether a target object is present in the gripper.

[0034] According to a fourth aspect of the embodiments of the present disclosure, a cleaning equipment control device is provided, the cleaning equipment comprising a mechanical arm and a gripper, the gripper being arranged at one end of the mechanical arm away from a machine body of the cleaning equipment, the mechanical arm being configured to cooperate with the gripper to grasp an object, the device comprising: a second acquisition unit configured to acquire an initial position of a target object, and control the cleaning equipment to grasp the target object based on the initial position; a second detection unit configured to detect whether the target object is present in the gripper; and a control unit configured to control the cleaning equipment to move the target object from the initial position to a target position if the target object is present in the gripper.

[0035] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, the computer program product comprising computer instructions stored in a computer readable storage medium and adapted to be read and executed by a processor to cause a computer device having the processor to perform operations performed by the method according to any one of the first aspect and the second aspect.

[0036] According to a sixth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, comprising at least one computer program instruction stored thereon, the at least one computer program instruction being loaded and executed by a processor to implement operations performed by the method according to any one of the first aspect and the second aspect.

[0037] According to a seventh aspect of the embodiments of the present disclosure, there is provided a cleaning device, comprising one or more processors and one or more memories, wherein the one or more memories store at least one computer program instruction, and when the at least one computer program instruction is loaded and executed by the one or more processors, the one or more processors are caused to implement the operations performed by the method according to any one of the first aspect and the second aspect of the embodiments of the present disclosure.

[0038] It should be understood that the general description above and the detailed description below are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. It should be apparent to those skilled in the art that the accompanying drawings are only some embodiments of the present disclosure, and other drawings can be obtained from these drawings without creative labor. In the drawings:

[0040] FIG. 1 shows an application scenario diagram of a cleaning device according to some embodiments of the present disclosure;

[0041] FIG. 2 shows a flowchart of an object detection method according to some embodiments of the present disclosure;

[0042] FIG. 3 shows a flowchart of a cleaning device control method according to some embodiments of the present disclosure;

[0043] FIG. 4 shows a demonstration diagram of a cleaning device performing a carrying task according to some embodiments of the present disclosure;

[0044] FIG. 5 shows a demonstration diagram of a cleaning device searching for a target object according to some embodiments of the present disclosure;

[0045] FIG. 6 shows a block diagram of an object detection apparatus according to some embodiments of the present disclosure;

[0046] FIG. 7 shows a block diagram of a cleaning device control apparatus according to some embodiments of the present disclosure;

[0047] FIG. 8 shows a structural schematic diagram of a cleaning device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0048] With reference to the drawings and the embodiments of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, but not all the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of the present disclosure.

[0049] In addition, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the present disclosure. One skilled in the relevant art will recognize, however, that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the present disclosure.

[0050] The block diagrams shown in the drawings are only functional entities, and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices. It should be noted that in the drawings, in order to ensure the simplicity of the drawings, some components in the drawings that do not affect the explanation of the technical solutions of the present disclosure are omitted.

[0051] The flowcharts shown in the drawings are only exemplary illustrations, and do not necessarily include all contents and operations / steps, nor do they necessarily have to be executed in the order described. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to the actual situation.

[0052] In the description of the present disclosure, it should be understood that the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more.

[0053] In order for those skilled in the art to better understand the present disclosure, first, the application scenario related to the present disclosure will be briefly described in conjunction with FIG. 1.

[0054] FIG. 1 shows an application scenario diagram of a cleaning device according to some embodiments of the present disclosure.

[0055] As shown in FIG. 1, the cleaning device 101 can be a sweeper, a mop, or a sweeper-mop combination, which is not limited in the present disclosure. In some embodiments, the cleaning device 101 can include a mechanical arm 101 and a gripper 102, the gripper 102 is arranged at the end of the mechanical arm 101 away from the body of the cleaning device 100, and the mechanical arm 101 is used to cooperate with the gripper 102 to grasp objects. It should be noted that the mechanical arm 101 shown in FIG. 1 is a two-degree-of-freedom mechanical arm, but in other embodiments, the degree of freedom of the mechanical arm can also be three degrees of freedom, or four degrees of freedom, which is not limited in the present disclosure.

[0056] In some embodiments, the cleaning device 101 can also include an image acquisition device 103, which can be used to acquire images containing the gripper 102. In the present disclosure, the image acquisition device 103 can be an RGB camera, a depth camera, or a 3D imaging sensor, etc. Two-dimensional images or three-dimensional point cloud images can be acquired by using the image acquisition device 103. As shown in FIG. 1, the image acquisition device 103 can be installed at the front of the chassis of the cleaning device 100, and in other embodiments, the image acquisition device can also be installed on the top of the chassis of the cleaning device or directly installed at the end of the gripper, which is not limited in the present disclosure.

[0057] In an embodiment of the present disclosure, the cleaning device 100 can be used to perform object carrying tasks on the ground, for example, the cleaning device 100 can carry the object 104 from position A to position B. The cleaning device 100 can control the mechanical arm 101 and the gripper 102 to grasp the object 104 at position A, and then move to the vicinity of position B to place the object 104 at position B, thereby completing the carrying task for the object 104.

[0058] In the present disclosure, it can be understood that when the cleaning device performs object carrying tasks on the ground, the object needs to be grasped by the gripper, and in order to achieve successful grasping of the object by the gripper of the cleaning device, it is necessary to ensure accurate detection of the grasped object by the cleaning device. In this case, the present disclosure first proposes an object detection method to improve the accuracy of object detection by the cleaning device.

[0059] FIG. 2 shows a flowchart of an object detection method according to some embodiments of the present disclosure, which can be applied to the cleaning device shown in FIG. 1 and can be executed by a device with computing processing function. Referring to FIG. 2, the cleaning device control method includes at least steps 210 to 220, which are described in detail as follows.

[0060] In step 210, a target image containing the gripper is acquired.

[0061] In some embodiments, acquiring the target image containing the gripper can comprise:

[0062] In step 211, a target image containing the gripper is acquired by the image acquisition device.

[0063] In some embodiments, since the image acquisition device is installed at the front of the chassis of the cleaning device, or on the top of the chassis of the cleaning device, or directly at the end of the gripper, when the gripper of the cleaning device grasps an object, the gripper will move into the field of view of the image acquisition device under the cooperation of the mechanical arm, so that the image acquisition device can acquire a target image (which can be a two-dimensional image or a three-dimensional point cloud image) containing the gripper.

[0064] It should be noted that during the process of grasping the target object, the gripper may have a blind area for the image acquisition device. Only when the gripper moves into the field of view of the image acquisition device under the cooperation of the mechanical arm, the image acquisition device can acquire a target image containing the gripper.

[0065] It should be further noted that in the present disclosure, the acquisition of images by the acquisition device can be continuous, i.e. the image acquisition device can acquire an image every certain time interval, such as every 1 second. After the gripper moves into the field of view of the image acquisition device under the cooperation of the mechanical arm, the image acquisition device can acquire multiple frames of target images containing the gripper.

[0066] Continuing to refer to FIG. 2, in step 220, based on the target image, it is detected whether the gripper contains a grasped target object.

[0067] In some embodiments, based on the target image, it is detected whether the gripper contains a grasped object, which can comprise:

[0068] In step 221, a mechanical arm position parameter of the mechanical arm is acquired, and based on the mechanical arm position parameter, foreground data in the target image is determined, wherein the foreground data is image data that moves with the gripper.

[0069] In step 222, based on the foreground data, it is detected whether the gripper contains a grasped target object.

[0070] In the present disclosure, while the image acquisition device acquires the target image containing the gripper, the mechanical arm position parameter of the mechanical arm can be recorded; the mechanical arm position parameter can include the size of each link in the mechanical arm, the included angle between adjacent links, and the relative position relationship between the mechanical arm and the image acquisition device.

[0071] In the present disclosure, the foreground data and the background data in the target image can be distinguished according to the mechanical arm position parameter of the mechanical arm. It should be noted that the foreground data is image data that moves with the movement of the gripper, such as image data of the mechanical arm, image data of the gripper, and image data of the object gripped by the gripper. The background data is image data that does not move with the movement of the gripper, such as image data of walls, furniture, household appliances, etc. After determining the foreground data in the target image, whether the gripper has a gripped target object can be detected based on the foreground data.

[0072] In the present disclosure, it can be understood that if the target image is captured by an RGB camera, the foreground data can actually include the contour information of the gripper; if the target image is captured by a depth camera, the foreground data can actually include the pose point cloud information of the gripper.

[0073] In the present disclosure, since the background data in the target image has no reference effect on detecting whether the gripper has a gripped target object, by determining the foreground data in the target image, the noise data including the background data in the target image is actually removed; in this way, the analysis efficiency and accuracy of the image data can be improved, and thus the detection efficiency and accuracy of the gripped object in the gripper can be improved.

[0074] In some embodiments, detecting whether the gripper has a gripped target object based on the foreground data can include:

[0075] Step 2221, determining first pose data of the gripper in the foreground data;

[0076] Step 2222, obtaining second pose data of the gripper in a closed state determined in advance;

[0077] Step 2223, comparing the first pose data and the second pose data to detect whether the gripper has a gripped target object.

[0078] In the present disclosure, the pose data of the gripper can be the opening degree of the components in the gripper, i.e., the opening degree of the components in the gripper. It can be understood that the opening degree of the components in the gripper in the state of gripping an object is greater than the opening degree of the components in the gripper in the closed state (i.e., the state of not gripping an object). Therefore, by comparing the first pose data of the gripper determined in the foreground data and the second pose data of the gripper in the closed state, whether the gripper has a gripped target object can be accurately detected.

[0079] In some embodiments, detecting whether there is a grabbed object in the gripper based on the target image can include:

[0080] Step 223, segmenting each target object in the target image by an image segmentation algorithm, the target object at least including the gripper;

[0081] Step 224, detecting whether there is a grabbed target object in the gripper according to the pose relationship between the target object other than the gripper and the gripper.

[0082] In the present disclosure, each target object in the target image can be segmented by an image segmentation algorithm, such as the gripper in the cleaning device, such as the mechanical arm in the cleaning device, such as other object objects such as furniture and home appliances in the environment; it can be understood that if the gripper grabs an object, the object grabbed in the gripper can also be segmented by the image segmentation algorithm. In some embodiments, according to the pose relationship between the target object other than the gripper and the gripper, if there is a segmented object in the gripper, it can be determined that there is a grabbed target object in the gripper. It can be seen that by the image segmentation algorithm, whether there is a grabbed target object in the gripper can be accurately detected.

[0083] Based on the object detection scheme proposed in the present disclosure, since the target image obtained contains the gripper, by image analysis on the target image, whether there is a grabbed target object in the gripper can be accurately detected.

[0084] The cleaning device proposed in the present disclosure can be used to arrange sundries, that is, it can perform a carrying task for objects; based on this, the present disclosure also proposes a cleaning device control method to improve the reliability of the cleaning device in performing the carrying task.

[0085] Referring to FIG. 3, a flowchart of a cleaning device control method according to some embodiments of the present disclosure is shown, which can be executed by a device with computing processing function. Referring to FIG. 3, the cleaning device control method at least includes:

[0086] Step 310, obtaining an initial position of a target object, and controlling the cleaning device to grab the target object based on the initial position;

[0087] Step 320, detecting whether there is a grabbed target object in the gripper;

[0088] Step 330, if there is a grabbed target object in the gripper, controlling the cleaning device to carry the target object to a target position.

[0089] In the present disclosure, during the process in which the cleaning device arranges the sundries, the cleaning device can detect the area to be arranged. If a target object to be carried is detected in the area to be arranged, the initial position of the target object can be determined based on the sensing data (such as point cloud data) of the target object collected by the sensor (such as a radar sensor, a laser sensor, etc.) on the cleaning device. After the initial position of the target object is obtained, the cleaning device can be controlled to move to the vicinity of the initial position, and then the mechanical arm of the cleaning device can be controlled to move the gripper to the initial position, and the target object can be grabbed by the gripper to carry the grabbed target object to a target position.

[0090] It should be noted that, during the process in which the cleaning device grabs the target object, and during the process in which the target object is carried to the target position, the cleaning device only controls the mechanical arm and the gripper to perform the grabbing action, and controls the body of the cleaning device to perform the walking action from the initial position to the target position, and controls the mechanical arm and the gripper to perform the object placing action at the target position. However, it cannot be guaranteed in practice that the gripper of the cleaning device has really grabbed the target object. If the gripper of the cleaning device does not grab the target object in practice, the purpose of carrying the target object from the initial position to the target position cannot be achieved.

[0091] However, based on the cleaning device control scheme proposed in the present disclosure, before the cleaning device is controlled to carry the target object to the target position, it is first detected whether the target object grabbed exists in the gripper. If the target object grabbed exists in the gripper, the cleaning device is controlled to carry the target object to the target position. It can be understood that, if the target object grabbed does not exist in the gripper, the cleaning device will not be controlled to perform the walking action from the initial position to the target position. It can be seen that, based on the cleaning device control scheme proposed in the present disclosure, it can be ensured that the target object is carried from the initial position to the target position, and thus the reliability of the cleaning device in performing the carrying task can be improved.

[0092] It should be further noted in the present disclosure that, if it is detected that the target object grabbed exists in the gripper, it can be considered that the gripper is successful in this object grabbing. At this time, the first appearance attribute information of the grabbed object (i.e., the target object) in the target image can be recorded. In some embodiments, the appearance attribute information of the object can include the category, size, shape contour, pose relative to the gripper, color, texture, etc. of the object.

[0093] In step 320 as shown in FIG. 3, whether the target object grabbed exists in the gripper can be detected according to the object detection method as described above.

[0094] Since the object detection method as described above can accurately detect whether the target object is present in the gripper, detection of whether the target object is present in the gripper by the object detection method as described above can further ensure that the target object is transported from the initial position to the target position, thereby improving the reliability of the cleaning device in performing the transport task.

[0095] Next, the disclosure will be based on the object detection method as described above, and the control logic of the cleaning device in transporting the target object will be described.

[0096] In some embodiments, the cleaning device control method can further include:

[0097] Step 341, during the process of transporting the target object by the cleaning device, determining a change value of the relative pose between the target object and the gripper according to target images containing the gripper acquired at different times, the change value being used to represent the degree of sliding of the target object in the gripper;

[0098] Step 342, detecting whether the target object in the gripper has fallen or has a risk of falling according to the change value.

[0099] In the disclosure, the image acquisition device can continuously acquire target images, such as acquiring a target image every 1 second. Thus, during the process of transporting the target object by the cleaning device, multiple target images containing the gripper can be acquired, and the relative pose between the target object and the gripper can be analyzed according to the target images acquired at different times. It can be understood that if the target object slides in the gripper, the relative pose between the target object and the gripper in different target images is different. Therefore, the disclosure can detect whether the target object in the gripper has fallen or has a risk of falling according to the determined change value of the relative pose between the target object and the gripper.

[0100] It should be noted that in the disclosure, the change value of the relative pose can be a change value of the relative distance between the target object and the gripper, or a change value of the relative angle between the target object and the gripper, or a change value calculated based on the relative distance and the relative angle. For this purpose, the disclosure is not limited in particular.

[0101] Next, the disclosure will combine two specific embodiments to describe the determination step of the change value of the relative pose between the target object and the gripper.

[0102] In some embodiments, the determining the change value of the relative pose between the target object and the gripper according to the target images containing the gripper acquired at different times can comprise:

[0103] Step 3411, determining foreground data in the target images acquired at different times to obtain multiple frames of foreground data; and

[0104] Step 3412, determining the change value of the relative pose between the target object and the gripper according to the multiple frames of foreground data.

[0105] Based on the object detection scheme as described above, the foreground data in the target images acquired at different times can be distinguished from the background data according to the mechanical arm position parameters of the mechanical arm at different times. Then, multiple frames of foreground data moving with the gripper can be obtained, and the change value of the relative pose can be determined according to the change of the relative pose between the target object and the gripper in the multiple frames of foreground data.

[0106] In other embodiments, the determining the change value of the relative pose between the target object and the gripper according to the target images containing the gripper acquired at different times can further comprise:

[0107] Step 3413, segmenting the target object and the gripper in each target image by an image segmentation algorithm;

[0108] Step 3414, determining the change value of the relative pose between the target object and the gripper according to the pose relationship between the target object and the gripper in each target image.

[0109] Based on the object detection scheme as described above, the segmentation of the target object and the gripper in the target images acquired at different times can be implemented based on an image segmentation algorithm. After the target object and the gripper in each target image are segmented, the change value of the relative pose can be determined according to the change of the relative pose between the target object and the gripper in each target image.

[0110] In some embodiments, the detecting whether the target object in the gripper falls or has a risk of falling according to the change value can comprise:

[0111] Step 3421, if the change value is greater than a first change threshold and less than a second change threshold, determining a risk value of the target object falling from the gripper according to the change value, and / or the distance between the cleaning device and the target position, and / or the category of the target object; and detecting whether the target object in the gripper has a risk of falling according to the risk value.

[0112] Step 3422, if the change value is less than or equal to the first change threshold value, it is determined that the target object in the gripper does not have a risk of falling off;

[0113] Step 3423, if the change value is greater than or equal to the second change threshold value, it is determined that the target object in the gripper falls off.

[0114] In some embodiments, the detecting whether the target object in the gripper has a risk of falling off according to the risk value can include:

[0115] Step 34211, if the risk value is less than or equal to a preset risk threshold value, it is determined that the target object in the gripper does not have a risk of falling off;

[0116] Step 34212, if the risk value is greater than the preset risk threshold value, it is determined that the target object in the gripper has a risk of falling off.

[0117] In the present disclosure, if the change value is greater than the first change threshold value and less than the second change threshold value, it means that the target object grabbed in the gripper has a certain sliding situation during the execution of the carrying task by the cleaning device; at this time, the risk value of the target object falling off from the gripper can be determined according to the change value (i.e., considering the sliding degree of the target object in the gripper; the greater the sliding degree, the higher the risk of the target object falling off from the gripper), and / or the distance between the cleaning device and the target position (i.e., considering whether the cleaning device is currently far away from the target position; the farther the distance, the higher the risk of the target object falling off from the gripper), and / or the category of the target object (i.e., considering whether the category (such as the shape of the object) of the target object is easy to fall off; the higher the risk of the target object falling off from the gripper if the shape of the object is a regular strip shape). It can be understood that the higher the risk value, the greater the possibility of the target object falling off from the gripper during the carrying process.

[0118] In some embodiments, when detecting whether the target object in the gripper has a risk of falling off according to the risk value, it can be detected by comparing the size relationship between the risk value and a preset risk threshold value. That is, if the risk value is less than or equal to a preset risk threshold value, it is determined that the target object in the gripper does not have a risk of falling off; if the risk value is greater than the preset risk threshold value, it is determined that the target object in the gripper has a risk of falling off.

[0119] In some embodiments, if the change value is less than or equal to the first change threshold, it indicates that the degree of sliding of the target object in the gripper during the execution of the carrying task is weak; at this time, it can be determined that the target object in the gripper does not have a risk of falling.

[0120] In some embodiments, if the change value is greater than or equal to the second change threshold, it indicates that the target object in the gripper has fallen. It should be noted that the second change threshold can be pre-set to a larger value, such as the length of the target object. If the target object in the gripper has fallen, it can be considered that the change value of the relative position between the target object and the gripper is a larger value (at least greater than the second change threshold).

[0121] Based on the cleaning device control scheme proposed in the present disclosure, the change value of the relative pose between the target object and the gripper is determined based on the target images obtained at different times; by comparing the size relationship between the change value and the pre-set first change threshold and second change threshold, it is determined whether the target object in the gripper of the cleaning device falls or has a risk of falling during the execution of the carrying task. In this way, during the execution of the object carrying task by the cleaning device, a more accurate reference basis can be provided for the subsequent control strategy of the cleaning device, so as to ensure that the target object is carried to the target position and improve the reliability of the cleaning device during the execution of the carrying task.

[0122] In some embodiments, the cleaning device control method can further include:

[0123] Step 351, if the target object in the gripper has a risk of falling, controlling the cleaning device to perform an object re-grabbing action;

[0124] Step 352, if the target object in the gripper has fallen, controlling the cleaning device to perform an object searching action;

[0125] In some embodiments, in the case where the target object in the gripper has a risk of falling, in order to avoid the target object from falling from the gripper during the carrying process, the cleaning device can be controlled to perform an object re-grabbing action.

[0126] In some embodiments, the control of the cleaning device to perform an object re-grabbing action can include:

[0127] Step 3521, controlling the cleaning device to stop moving and placing the target object grabbed by the gripper on the ground;

[0128] Step 3522, controlling the cleaning device to re-grab the target object.

[0129] In the present disclosure, after controlling the cleaning device to re-grab the target object, the cleaning device can continue to be controlled to perform a carrying task of carrying the target object to a target position.

[0130] In the present disclosure, the cleaning device can further include a photoelectric sensor, or a force feedback sensor, or a current detection sensor.

[0131] Based on the cleaning device including the photoelectric sensor, or the force feedback sensor, or the current detection sensor, in step 320 as shown in FIG. 3, the detection of whether the target object is grabbed in the gripper can further include:

[0132] Step 321, acquiring first sensing data collected by the photoelectric sensor, or the force feedback sensor, or the current detection sensor;

[0133] Step 322, detecting whether the target object is grabbed in the gripper according to the first sensing data.

[0134] In some embodiments, the photoelectric sensor can be installed on the gripper, and then whether the target object is grabbed by the gripper can be determined according to whether the photoelectric sensor detects a changed photoelectric signal (i.e., first sensing data) (if the target object is grabbed by the gripper, the photoelectric signal emitted by the photoelectric sensor will be blocked by the target object).

[0135] In the present disclosure, the force feedback sensor can be installed on the gripper or the mechanical arm, and then whether the target object is grabbed by the gripper can be determined according to whether the force feedback sensor detects a changed force signal (i.e., first sensing data) (if the target object is grabbed by the gripper, the force signal detected by the force feedback sensor will become larger).

[0136] In the present disclosure, the current detection sensor can be installed on the gripper or the driving motor of the mechanical arm, and then whether the target object is grabbed by the gripper can be determined according to whether the current detection sensor detects a changed electric signal (i.e., first sensing data) (if the target object is grabbed by the gripper, the electric signal detected by the current detection sensor will become larger).

[0137] In some embodiments, the cleaning device control method can further include:

[0138] Step 351, acquiring second sensing data collected by the photoelectric sensor, or the force feedback sensor, or the current detection sensor during the carrying of the target object by the cleaning device;

[0139] Step 352, detecting whether the target object in the gripper falls according to the second sensing data;

[0140] Step 353, if the target object in the gripper falls, controlling the cleaning device to perform an object retrieval action.

[0141] In some embodiments, if a photoelectric sensor is installed on the gripper, when the target object on the gripper falls, the photoelectric signal (i.e., the second sensing data) emitted by the photoelectric sensor will be detected by the photoelectric sensor itself; if a force feedback sensor is installed on the gripper or the mechanical arm, when the target object on the gripper falls, the force signal (i.e., the second sensing data) detected by the force feedback sensor will become smaller; if a current detection sensor is installed on the gripper or the driving motor of the mechanical arm, when the target object on the gripper falls, the electric signal (i.e., the second sensing data) detected by the current detection sensor will become smaller. In this way, whether the target object in the gripper falls can be detected according to the second sensing data collected by the photoelectric sensor, or the force feedback sensor, or the current detection sensor.

[0142] Next, further description will be made on how to control the cleaning device to perform an object retrieval action:

[0143] In some embodiments, the control of the cleaning device to perform an object retrieval action can include:

[0144] Step 361, obtaining a first position of the cleaning device when the falling of the target object is detected;

[0145] Step 362, estimating a falling area of the target object according to the first position;

[0146] Step 363, controlling the cleaning device to search for the target object in the falling area;

[0147] Step 364, if the cleaning device searches for the target object in the falling area, controlling the cleaning device to re-grab the target object and carry the re-grabbed target object to a target position.

[0148] In some embodiments, the control of the cleaning device to perform an object retrieval action first needs to estimate the falling area of the target object, so as to facilitate the cleaning device to search for the target object in the falling area.

[0149] In some embodiments, the estimation of the falling area of the target object according to the first position can include:

[0150] Step 3621, obtaining a second position of the cleaning device when the cleaning device last detected that the target object did not fall off;

[0151] Step 3622, estimating a falling area of the target object according to the first position and the second position;

[0152] In order for those skilled in the art to better understand the present disclosure, the following is an embodiment in conjunction with FIG. 4.

[0153] FIG. 4 shows a demonstration diagram of a cleaning device performing a carrying task according to some embodiments of the present disclosure.

[0154] As shown in FIG. 4, the object carrying task performed by the cleaning device 100 is to carry the target object from the initial position A to the target position B; in this process, the cleaning device 100 is first controlled to grasp the target object at the initial position A; after grasping the target object, the cleaning device 100 is controlled to travel in the object carrying direction; in the process of traveling of the cleaning device 100, the target object actually falls off from the gripper at position D2; the cleaning device can detect that the target object has not fallen off from the gripper before position D2 (i.e., at position D1); when the cleaning device detects the target object at position D3, it can be detected that the target object has fallen off from the gripper. It can be understood that there is a certain time delay between the detection of the cleaning device that the target object has fallen off and the actual falling off of the target object, and any historical position in this period of time can be the position where the target object falls off from the gripper.

[0155] Based on this, in the present disclosure, the second position D1 of the cleaning device when the cleaning device last detected that the target object did not fall off can be obtained, and the falling area C of the target object can be estimated based on the first position D3 of the cleaning device when the cleaning device detected that the target object fell off and the second position D1. For example, the xy coordinates of the second position can be taken as the minimum xy coordinates of the falling area, and the xy coordinates of the first position can be taken as the maximum xy coordinates of the falling area, and then the falling area C of the target object is determined.

[0156] In other embodiments, considering that the target object may roll to other positions when it falls off at a certain position; therefore, the initially determined falling area can also be expanded by a certain proportion to obtain the expanded falling area C; the advantage of this is that it can increase the possibility of the cleaning device searching for the target object.

[0157] In the present disclosure, the falling area of the target position can also be determined according to the first position, that is, only the first position is taken as the reference position, and the falling area is determined in the ground according to certain rules, and the present disclosure does not make specific limitations on this.

[0158] In the present disclosure, after determining the falling area of the target object, the cleaning device can be controlled to search for the target object in the falling area.

[0159] In some embodiments, the controlling the cleaning device to search for the target object in the falling area can include:

[0160] Step 3631, determining at least one target search point in the falling area;

[0161] Step 3632, controlling the cleaning device to move to each target search point and respectively observe a circle at each target search point to search for the target object.

[0162] In order to make the person skilled in the art better understand the present disclosure, the following will be described in conjunction with a specific embodiment of FIG. 5.

[0163] Referring to FIG. 5, a demonstration diagram of a cleaning device searching for a target object according to some embodiments of the present disclosure is shown.

[0164] As shown in subgraph (a) of FIG. 5, after determining target search points F, G, and H in the falling area C, the cleaning device 100 can be controlled to move to the target search points F, G, and H in sequence and respectively observe a circle to search for the target object in a certain area centered on the search point (for example, the area f centered on the search point F).

[0165] As also shown in subgraph (b) of FIG. 5, after determining target search points I, J, K, and L in the falling area C, the cleaning device 100 can be controlled to move to the target search points I, J, K, and L in sequence and respectively observe a circle to search for the target object in a certain area centered on the search point.

[0166] In some embodiments, the determining at least one target search point in the falling area can include:

[0167] Step 36311, determining a plurality of groups of search points in the falling area, wherein each group of search points includes at least one search point;

[0168] Step 36312, calculating the search efficiency of each group of search points, and determining the search point in the group of search points with the highest search efficiency as the at least one target search point.

[0169] In the present disclosure, continuing to refer to FIG. 5, a set of search points F, G, H as shown in subgraph (a) of FIG. 5 and a set of search points I, J, K, L as shown in subgraph (b) of FIG. 5 can be determined in the drop area, and then the search efficiencies of the two sets of search points are calculated respectively. When calculating the search efficiencies, factors such as required search time, covered search range, etc. can be considered. After calculating the search efficiencies of the sets of search points, the search point in the set of search points with the highest search efficiency can be determined as the at least one target search point. In the present disclosure, the algorithm for calculating the search efficiency can be designed according to actual conditions, and the present disclosure does not make specific limitations thereon.

[0170] In the present disclosure, by determining multiple sets of search points in the drop area and determining the set of search points with the highest search efficiency as the at least one target search point, the search efficiency of searching for the target object in the drop area can be improved.

[0171] In some embodiments, the controlling the cleaning device to search for the target object in the drop area can further include:

[0172] Step 3633, if the cleaning device searches for an object in the drop area, acquiring second appearance attribute information of the searched object;

[0173] Step 3634, acquiring first appearance attribute information of the target object recorded by the cleaning device when the target object is first grasped;

[0174] Step 3635, calculating a similarity between the searched object and the target object based on the first appearance attribute information and the second appearance attribute information;

[0175] Step 3636, determining the object with the highest similarity and exceeding a preset similarity threshold as the target object.

[0176] In the present disclosure, since the cleaning device can search for multiple objects in the drop area, in this case, the first appearance attribute information of the searched objects (such as the category, size, shape, pose relative to the gripper, color, texture, etc. of the object) can be obtained, and the first appearance attribute information of the target object recorded when the cleaning device first grasps the target object can be obtained. Then, based on the first appearance attribute information and the second appearance attribute information, the similarity between the searched objects and the target object is calculated, and the object with the highest similarity and exceeding a preset similarity threshold is determined as the target object. It can be understood that, according to the appearance attribute information of the object, the similarity between the searched objects and the target object is calculated, the target object can be accurately determined from the searched objects, so that the target object can be accurately found back, so as to ensure that the target object is transported from the initial position to the target position, and thus the reliability of the cleaning device in performing the transportation task is improved.

[0177] In some embodiments, the cleaning device control method can further include:

[0178] Step 365, if the cleaning device does not search for the target object in the drop area, controlling the cleaning device to perform other tasks, and caching the first appearance attribute information of the target object and the current transportation task information, the first appearance attribute information and the task information being used for the cleaning device to search for the target object in the process of performing other tasks.

[0179] In some embodiments, the cleaning device control method can further include:

[0180] Step 366, if the cleaning device searches for the target object in the process of performing other tasks, controlling the cleaning device to re-grasp the target object, and transporting the re-grasped target object to the target position.

[0181] Step 367, if the cleaning device does not search for the target object in the process of performing other tasks, triggering an error prompt.

[0182] In actual application, for a cleaning device configured with a mechanical arm and a gripper, after an object is grabbed, an object carrying action is performed. Since the gripper itself may be unstable when grabbing the object, and there are problems such as acceleration and deceleration, jolting or collision between the grabbed object and the environment during the carrying process, the grabbed object may fall during the carrying process. The cleaning device control scheme based on the present disclosure first detects whether the grabbed target object exists in the gripper before controlling the cleaning device to carry the target object to the target position. If the grabbed target object exists in the gripper, the cleaning device is controlled to carry the target object to the target position. It can be understood that if the grabbed target object does not exist in the gripper, the cleaning device will not perform the walking action from the initial position to the target position. During the object carrying task of the cleaning device, by detecting whether the target object in the gripper of the cleaning device falls, a more accurate reference basis can be provided for the cleaning device to execute a control strategy in the subsequent process, so as to ensure that the target object is carried to the target position, thereby improving the reliability of the cleaning device in the carrying task.

[0183] The following describes a device embodiment of the present disclosure, which can be used to execute the object detection method in the above-mentioned embodiments of the present disclosure. For details not disclosed in the device embodiment of the present disclosure, please refer to the above-mentioned embodiments of the object detection method of the present disclosure.

[0184] Referring to FIG. 6, a block diagram of an object detection device according to some embodiments of the present disclosure is shown. The device is applied to a cleaning device. The cleaning device includes a mechanical arm and a gripper. The gripper is arranged at one end of the mechanical arm away from the body of the cleaning device. The mechanical arm is used to cooperate with the gripper to grab an object.

[0185] As shown in FIG. 6, the object detection device 600 according to the embodiments of the present disclosure includes a first acquisition unit 601 and a first detection unit 602.

[0186] In some embodiments, the first acquisition unit 601 is configured to acquire a target image containing the gripper; and the first detection unit 602 is configured to detect whether a grabbed target object exists in the gripper based on the target image.

[0187] In some embodiments, the cleaning device further includes an image acquisition device, and the first acquisition unit 601 is configured to acquire a target image containing the gripper acquired by the image acquisition device.

[0188] In some embodiments, the first detection unit 602 is configured to: acquire a mechanical arm position parameter of the mechanical arm, and determine foreground data in the target image according to the mechanical arm position parameter; the foreground data is image data that moves with the gripper; and detect whether the gripper contains a target object to be grabbed based on the foreground data.

[0189] In some embodiments, the first detection unit 602 is further configured to: determine first pose data of the gripper in the foreground data; acquire second pose data of the gripper in a closed state that is predetermined; and compare the first pose data and the second pose data to detect whether the gripper contains a target object to be grabbed.

[0190] In some embodiments, the first detection unit 602 is configured to: segment each target object in the target image by an image segmentation algorithm, the target object at least including the gripper; and detect whether the gripper contains a target object to be grabbed according to a pose relationship between the target object other than the gripper and the gripper.

[0191] Another device embodiment of the present disclosure is introduced below, which can be used to execute the cleaning equipment control method in the above-mentioned embodiments of the present disclosure. For details not disclosed in the device embodiment of the present disclosure, please refer to the above-mentioned embodiments of the cleaning equipment control method of the present disclosure.

[0192] Referring to FIG. 7, a block diagram of a cleaning equipment control device according to some embodiments of the present disclosure is shown, the cleaning equipment including a mechanical arm and a gripper, the gripper being arranged at an end of the mechanical arm away from a machine body of the cleaning equipment, and the mechanical arm being used to cooperate with the gripper to grab objects.

[0193] As shown in FIG. 7, the cleaning equipment control device 700 according to the embodiments of the present disclosure includes a second acquisition unit 701, a second detection unit 702, and a control unit 703.

[0194] In some embodiments, the second acquisition unit 701 is configured to acquire an initial position of a target object, and control the cleaning equipment to grab the target object based on the initial position; the second detection unit 702 is configured to detect whether the gripper contains a target object to be grabbed; and the control unit 703 is configured to control the cleaning equipment to carry the target object to a target position if the gripper contains a target object to be grabbed.

[0195] In some embodiments, the second detection unit 702 is configured to detect whether the gripper contains a target object to be grabbed according to the object detection method as described above.

[0196] In some embodiments, the apparatus further comprises a detection unit configured to determine a change value of the relative pose between the target object and the gripper during the process that the cleaning device carries the target object according to target images containing the gripper acquired at different times, the change value being used to represent a degree of slippage of the target object in the gripper; and detect whether the target object in the gripper falls or has a risk of falling according to the change value.

[0197] In some embodiments, the detection unit is configured to determine foreground data in the target images acquired at different times to obtain a plurality of frames of foreground data; and determine the change value of the relative pose between the target object and the gripper according to the plurality of frames of foreground data.

[0198] In some embodiments, the detection unit is configured to segment the target object and the gripper in each target image by an image segmentation algorithm; and determine the change value of the relative pose between the target object and the gripper according to the pose relationship between the target object and the gripper in each target image.

[0199] In some embodiments, the detection unit is configured to, if the change value is greater than a first change threshold and less than a second change threshold, determine a risk value of the target object falling from the gripper according to the change value, and / or a distance between the cleaning device and the target position, and / or a category of the target object; and detect whether the target object in the gripper has a risk of falling according to the risk value.

[0200] In some embodiments, the detection unit is configured to, if the risk value is less than or equal to a preset risk threshold, determine that the target object in the gripper does not have a risk of falling; and if the risk value is greater than the preset risk threshold, determine that the target object in the gripper has a risk of falling.

[0201] In some embodiments, the detection unit is configured to, if the change value is less than or equal to the first change threshold, determine that the target object in the gripper does not have a risk of falling.

[0202] In some embodiments, the detection unit is configured to, if the change value is greater than or equal to the second change threshold, determine that the target object in the gripper falls.

[0203] In some embodiments, the control unit 703 is configured to, if the target object in the gripper has a risk of falling, control the cleaning device to perform an object re-grabbing action; and if the target object in the gripper falls, control the cleaning device to perform an object search action.

[0204] In some embodiments, the control unit 703 is configured to control the cleaning device to stop moving and place the target object gripped by the gripper on the ground, and control the cleaning device to re-grab the target object.

[0205] In some embodiments, the cleaning device further comprises a photoelectric sensor, or a force feedback sensor, or a current detection sensor, and the second detection unit 702 is configured to acquire first sensing data collected by the photoelectric sensor, or the force feedback sensor, or the current detection sensor, and detect whether the target object is gripped in the gripper according to the first sensing data.

[0206] In some embodiments, the control unit 703 is configured to acquire second sensing data collected by the photoelectric sensor, or the force feedback sensor, or the current detection sensor during the process that the cleaning device carries the target object, and detect whether the target object in the gripper falls according to the second sensing data, and control the cleaning device to perform an object search action if the target object in the gripper falls.

[0207] In some embodiments, the control unit 703 is configured to acquire a first position of the cleaning device when the falling of the target object is detected, estimate a falling area of the target object according to the first position, control the cleaning device to search for the target object in the falling area, and control the cleaning device to re-grab the target object and carry the re-grabbed target object to a target position if the cleaning device searches for the target object in the falling area.

[0208] In some embodiments, the control unit 703 is configured to acquire a second position of the cleaning device when the falling of the target object is last detected, and estimate the falling area of the target object according to the first position and the second position.

[0209] In some embodiments, the control unit 703 is configured to determine at least one target search point in the falling area, control the cleaning device to move to each target search point and observe a circle at each target search point to search for the target object.

[0210] In some embodiments, the control unit 703 is configured to determine a plurality of groups of search points in the falling area, each group of search points comprising at least one search point, calculate a search efficiency of each group of search points, and determine a search point in a group of search points with the highest search efficiency as the at least one target search point.

[0211] In some embodiments, the control unit 703 is configured to: if the cleaning device searches for an object in the drop area, acquire second appearance attribute information of the searched object; acquire first appearance attribute information of the target object recorded when the cleaning device first grasps the target object; calculate a similarity between the searched object and the target object based on the first appearance attribute information and the second appearance attribute information; and determine the object with the highest similarity and exceeding a preset similarity threshold as the target object.

[0212] In some embodiments, the control unit 703 is configured to: if the cleaning device does not search for the target object in the drop area, control the cleaning device to perform other tasks, and cache first appearance attribute information of the target object and current task information of the carried task, the first appearance attribute information and the task information being used for the cleaning device to search for the target object in the process of performing other tasks.

[0213] In some embodiments, the control unit 703 is configured to: if the cleaning device searches for the target object in the process of performing other tasks, control the cleaning device to re-grasp the target object and carry the re-grasped target object to a target position; and if the cleaning device does not search for the target object in the process of performing other tasks, trigger an error prompt.

[0214] Based on the same inventive concept, an embodiment of the present disclosure provides a computer program product, which includes computer instructions stored in a computer readable storage medium and adapted to be read and executed by a processor to enable a computer device having the processor to implement operations performed by the object detection method and the cleaning device control method as described above.

[0215] Based on the same inventive concept, an embodiment of the present disclosure provides a computer readable storage medium, which includes at least one computer program instruction stored thereon and loaded and executed by a processor to implement operations performed by the object detection method and the cleaning device control method as described above.

[0216] Based on the same inventive concept, an embodiment of the present disclosure further provides a cleaning device, referring to FIG. 8, which shows a structural schematic diagram of a cleaning device according to some embodiments of the present disclosure, the cleaning device including one or more memories 804, one or more processors 802, and at least one computer program (computer program instructions) stored in the memory 804 and executable on the processor 802, the processor 802 implementing the object detection method and the cleaning device control method as described above when executing the computer program.

[0217] In FIG. 8, a bus architecture (represented by bus 800) can include any number of interconnected buses and bridges. The bus 800 links together various circuits including the processor 802, represented by one or more processors, and the memory 804, represented by memory. The bus 800 can also link various other circuits such as peripheral devices, voltage stabilizers, and power management circuits, which are well known in the art and thus, not further described herein. The bus interface 805 provides an interface between the bus 800 and the receiver 801 and the transmitter 803. The receiver 801 and the transmitter 803 can be the same element, i.e., a transceiver, providing a means for communicating with various other apparatuses over a transmission medium. The processor 802 is responsible for managing the bus 800 and general processing, while the memory 804 can be used for storing data used by the processor 802 in executing operations.

[0218] The functions described herein can be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions can be stored on or transferred over as one or more instructions or code on a computer-readable medium. Other examples and implementations are within the scope and spirit of the disclosure and appended claims. For example, due to the nature of software, functions described above can be implemented using software executed by a processor, hardware, firmware, hardwiring, or combinations of any of these. Also, each of the functions can be implemented as a separate means or some of the functions can be combined into a single means. In an embodiment, the functions of the various elements described herein can be provided through the use of dedicated hardware as well as hardware capable of executing software. When provided by a processor, the processor can be a single processor or can be

[0219] In several embodiments provided in the present disclosure, it should be understood that the disclosed technology can be implemented in other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between units or modules, which can be electrical or other forms.

[0220] The units described as separate components can or can not be physically separate, and the components of the control device can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0221] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present disclosure, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer program instructions.

[0222] The above only describes the embodiments of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. A method for detecting an object, applied to a cleaning device, the cleaning device comprising a mechanical arm and a gripper, the gripper being arranged at an end of the mechanical arm away from a body of the cleaning device, the mechanical arm being configured to cooperate with the gripper to grasp an object, the method comprising: obtaining a target image containing the gripper; and detecting, based on the target image, whether a grasped target object exists in the gripper. The cleaning device further comprises an image acquisition device, and the obtaining the target image containing the gripper comprises: obtaining the target image containing the gripper acquired by the image acquisition device. The detecting, based on the target image, whether a grasped target object exists in the gripper comprises: obtaining a mechanical arm position parameter of the mechanical arm, and determining foreground data and background data in the target image according to the mechanical arm position parameter, wherein the foreground data is image data that moves with the gripper, and the background data is image data that does not move with the gripper; and detecting, based on the foreground data, whether a grasped target object exists in the gripper. The detecting, based on the foreground data, whether a grasped target object exists in the gripper comprises: determining first pose data of the gripper in the foreground data; obtaining second pose data of the gripper in a closed state that is predetermined in advance; and comparing the first pose data and the second pose data to detect whether a grasped target object exists in the gripper.

2. The method of claim 1, wherein, The detecting, based on the target image, whether a grasped target object exists in the gripper comprises: segmenting each target object in the target image by using an image segmentation algorithm, the target object at least including the gripper; and detecting, according to a pose relationship between the target object other than the gripper and the gripper, whether a grasped target object exists in the gripper. 6.A method for controlling a cleaning device, the cleaning device comprising a mechanical arm and a gripper, the gripper being arranged at an end of the mechanical arm away from a body of the cleaning device, the mechanical arm being configured to cooperate with the gripper to grasp an object, the method comprising: obtaining an initial position of a target object, and controlling the cleaning device to grasp the target object based on the initial position; detecting whether a grasped target object exists in the gripper; and if a grasped target object exists in the gripper, controlling the cleaning device to carry the target object to a target position.

3. The method of claim 1, wherein, The detecting whether a grasped target object exists in the gripper comprises: detecting, according to the method of any one of claims 1 to 5, whether a grasped target object exists in the gripper. 8.The method of claim 7, further comprising: during the cleaning device carrying the target object, determining a change value of a relative pose between the target object and the gripper according to target images containing the gripper obtained at different times, the change value being used to represent a degree of sliding of the target object in the gripper. ​ 4. The method of claim 3, wherein, ​ ​ ​ ​ 5. The method of claim 1, wherein, ​ ​ ​ ​ ​ ​ ​ ​ 7. The method of claim 6, wherein, ​ ​ ​ ​ According to the change value, it is detected whether the target object in the gripper falls or has a risk of falling.

9. The method of claim 8, wherein, The change value of the relative pose between the target object and the gripper is determined according to target images containing the gripper acquired at different times, and the change value of the relative pose between the target object and the gripper includes: The foreground data is determined in the target images acquired at different times to obtain multiple frames of foreground data; and According to the multiple frames of foreground data, the change value of the relative pose between the target object and the gripper is determined.

10. The method of claim 8, wherein, The change value of the relative pose between the target object and the gripper is determined according to target images containing the gripper acquired at different times, and the change value of the relative pose between the target object and the gripper includes: The target object and the gripper in each target image are segmented by an image segmentation algorithm; and According to the pose relationship between the target object and the gripper in each target image, the change value of the relative pose between the target object and the gripper is determined.

11. The method of claim 8, wherein, The change value of the relative pose between the target object and the gripper is determined according to target images containing the gripper acquired at different times, and the change value of the relative pose between the target object and the gripper includes: If the change value is greater than a first change threshold and less than a second change threshold, a risk value of the target object falling from the gripper is determined according to the change value, and / or the distance between the cleaning device and the target position, and / or the category of the target object; According to the risk value, it is detected whether the target object in the gripper has a risk of falling.

12. The method of claim 11, wherein, The change value of the relative pose between the target object and the gripper is determined according to target images containing the gripper acquired at different times, and the change value of the relative pose between the target object and the gripper includes: If the risk value is less than or equal to a preset risk threshold, it is determined that the target object in the gripper does not have a risk of falling; and If the risk value is greater than the preset risk threshold, it is determined that the target object in the gripper has a risk of falling.

13. The method of claim 11, further comprising: If the change value is less than or equal to the first change threshold, it is determined that the target object in the gripper does not have a risk of falling.

14. The method of claim 11, further comprising: If the change value is greater than or equal to the second change threshold, it is determined that the target object in the gripper falls.

15. The method of claim 8, further comprising: If the target object in the gripper has a risk of falling, the cleaning device is controlled to perform an object re-grab action; and If the target object in the gripper falls, the cleaning device is controlled to perform an object search action. The control of the cleaning device to perform the object re-grab action includes:

16. The method of claim 15, wherein, The cleaning device is controlled to stop moving and place the target object grabbed by the gripper on the ground; and The cleaning device is controlled to re-grab the target object. The cleaning device further includes a photoelectric sensor, or a force feedback sensor, or a current detection sensor, and the detection of whether the target object is grabbed in the gripper includes:

17. The method of claim 6, wherein, First sensing data collected by the photoelectric sensor, or the force feedback sensor, or the current detection sensor is acquired; and According to the first sensing data, it is detected whether the target object is grabbed in the gripper. ​ 18. The method of claim 17, further comprising: acquiring second sensing data collected by the photosensor, or the force feedback sensor, or the current detection sensor during the carrying of the target object by the cleaning device; detecting whether the target object in the gripper has fallen according to the second sensing data; and controlling the cleaning device to perform an object retrieving action if the target object in the gripper has fallen. The controlling the cleaning device to perform an object retrieving action comprises:

19. The method of claim 15 or 18, wherein, acquiring a first position of the cleaning device when the falling of the target object is detected; estimating a falling area of the target object according to the first position; controlling the cleaning device to search for the target object in the falling area; and controlling the cleaning device to re-grab the target object and carry the re-grabbed target object to a target position if the cleaning device searches for the target object in the falling area. The estimating a falling area of the target object according to the first position comprises:

20. The method of claim 19, wherein, acquiring a second position of the cleaning device when the target object is last detected not to have fallen; and estimating a falling area of the target object according to the first position and the second position. The controlling the cleaning device to search for the target object in the falling area comprises:

21. The method of claim 19, wherein, determining at least one target search point in the falling area; and controlling the cleaning device to move to each target search point and to observe a circle around each target search point to search for the target object. The determining at least one target search point in the falling area comprises:

22. The method of claim 21, wherein, determining a plurality of groups of search points in the falling area, wherein each group of search points comprises at least one search point; and calculating a search efficiency of each group of search points, and determining a search point in a group of search points with the highest search efficiency as the at least one target search point. The controlling the cleaning device to search for the target object in the falling area comprises:

23. The method of claim 19, wherein, acquiring second appearance attribute information of an object searched for by the cleaning device in the falling area; acquiring first appearance attribute information of the target object recorded by the cleaning device when the target object is first grabbed; calculating a similarity between the object searched for and the target object based on the first appearance attribute information and the second appearance attribute information; and determining an object with the highest similarity and exceeding a preset similarity threshold as the target object.

24. The method of claim 19, further comprising: controlling the cleaning device to perform other tasks and to cache the first appearance attribute information of the target object and current carrying task information if the cleaning device does not search for the target object in the falling area, wherein the first appearance attribute information and the task information are used for the cleaning device to search for the target object during the performance of the other tasks.

25. The method of claim 24, further comprising: ​ If the cleaning device searches for the target object during execution of other tasks, the control device controls the cleaning device to re-grab the target object and carry the re-grabbed target object to a target position. If the cleaning device does not search for the target object during execution of other tasks, the control device triggers an error prompt.

26. An object detection device applied to a cleaning device, the cleaning device comprising a mechanical arm and a gripper, the gripper being arranged at an end of the mechanical arm away from a body of the cleaning device, the mechanical arm being configured to cooperate with the gripper to grab an object, the device comprising: a first acquisition unit configured to acquire a target image containing the gripper; and a first detection unit configured to detect, based on the target image, whether a target object grabbed by the gripper exists.

27. A cleaning device control device, the cleaning device comprising a mechanical arm and a gripper, the gripper being arranged at an end of the mechanical arm away from a body of the cleaning device, the mechanical arm being configured to cooperate with the gripper to grab an object, the device comprising: a second acquisition unit configured to acquire an initial position of a target object and control the cleaning device to grab the target object based on the initial position; a second detection unit configured to detect whether a target object grabbed by the gripper exists; and a control unit configured to control the cleaning device to carry the target object to a target position if the target object grabbed by the gripper exists. The computer instructions are stored in a computer readable storage medium and are adapted to be read and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1 to 25. The at least one piece of program code is loaded and executed by the processor to implement the operations performed by the method of any one of claims 1 to 25. The at least one piece of program code is loaded and executed by the one or more processors to cause the one or more processors to implement the method of any one of claims 1 to 25.

28. A computer program product comprising computer instructions, wherein, ​ 29. A computer readable storage medium comprising at least one program code stored thereon, wherein, ​ 30. A cleaning device comprising one or more processors and one or more memories having at least one program code stored therein, wherein, ​

Images