Object detection method, cleaning equipment control method and device and cleaning equipment

Abstract

The invention discloses an object detection method, a cleaning equipment control method and device and cleaning equipment, the object detection method is applied to the cleaning equipment, the cleaning equipment comprises a mechanical arm and a clamping jaw, the clamping jaw is arranged at the end, away from a cleaning equipment body, of the mechanical arm, and the mechanical arm is used for being matched with the clamping jaw to grab an object. The object detection method comprises the following steps: acquiring a target image including the clamping jaw; and on the basis of the target image, whether a grabbed target object exists in the clamping jaw or not is detected. Through the technical scheme provided by the invention, the detection accuracy of the cleaning equipment on the grabbed object can be improved.

Description

Technical Field

[0001] The present application belongs to the field of cleaning equipment control technology, and in particular relates to an object detection method, a cleaning equipment control method, a device, a program product, a medium, and a cleaning equipment. Background Art

[0002] Cleaning equipment (such as sweepers and mops) has been widely used in family life today. It can complete housework on behalf of users, which brings great convenience to users. In addition to cleaning the floor, cleaning equipment can also organize debris. In the process of organizing debris, the gripper configured in the cleaning equipment is usually required to grasp objects. It is understandable that ensuring that the cleaning equipment can accurately detect the grasped object is the key to the cleaning equipment's ability to grasp the object. However, there is currently no mature object detection solution for cleaning equipment to accurately detect the grasped object. Based on this, how to improve the accuracy of cleaning equipment's detection of grasped objects is a technical problem that needs to be solved urgently. Summary of the Invention

[0003] The embodiments of the present application provide an object detection method, a cleaning equipment control method, an apparatus, a program product, a medium, and a cleaning equipment, which can improve the accuracy of the cleaning equipment in detecting the grasped object, at least to a certain extent.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0005] According to a first aspect of an embodiment of the present application, there is provided an object detection method, which is applied to a cleaning device, wherein the cleaning device comprises a robotic arm and a gripper, wherein the gripper is provided at an end of the robotic arm away from a body of the cleaning device, and the robotic arm is used to cooperate with the gripper to grasp an object, and the method comprises: acquiring a target image including the gripper; and based on the target image, detecting whether a grasped target object exists in the gripper.

[0006] In some embodiments of the present application, based on the aforementioned solution, the cleaning device further includes an image acquisition device, and acquiring the target image including the clamping claw includes: acquiring the target image including the clamping claw acquired by the image acquisition device.

[0007] In some embodiments of the present application, based on the aforementioned scheme, detecting whether there is a grasped target object in the gripper based on the target image includes: obtaining the robotic arm position parameters of the robotic arm, and determining the foreground data in the target image based on the robotic arm position parameters, wherein the foreground data is image data that moves with the movement of the gripper; and detecting whether there is a grasped target object in the gripper based on the foreground data.

[0008] In some embodiments of the present application, based on the aforementioned scheme, detecting whether there is a grasped target object in the gripper based on the foreground data includes: determining first posture data of the gripper in the foreground data; obtaining predetermined second posture data of the gripper in a closed state; and comparing the first posture data and the second posture data to detect whether there is a grasped target object in the gripper.

[0009] In some embodiments of the present application, based on the aforementioned scheme, detecting whether there is a grasped target object in the gripper based on the target image includes: segmenting the target objects in the target image through an image segmentation algorithm, where the target objects include at least the gripper; and detecting whether there is a grasped target object in the gripper based on the posture relationship between the gripper and the target objects other than the gripper.

[0010] Based on the technical solution proposed in the present application, since the acquired target image contains the gripper, by performing image analysis on the target image, it is possible to accurately detect whether there is a grasped target object in the gripper.

[0011] According to a second aspect of an embodiment of the present application, a method for controlling a cleaning device is provided, wherein the cleaning device includes a robotic arm and a gripper, wherein the gripper is provided at one end of the robotic arm away from a body of the cleaning device, and the robotic arm is used to cooperate with the gripper to grasp an object, and the method includes: obtaining an initial position of the target object, and based on the initial position, controlling the cleaning device to grasp the target object; detecting whether there is a grasped target object in the gripper; and if there is a grasped target object in the gripper, controlling the cleaning device to move the target object to the target position.

[0012] In some embodiments of the present application, based on the aforementioned scheme, detecting whether there is a grasped target object in the gripper includes: detecting whether there is a grasped target object in the gripper according to the method described in the first aspect of the embodiment of the present application.

[0013] In some embodiments of the present application, based on the aforementioned scheme, the method further includes: in the process of the cleaning equipment transporting the target object, determining the change value of the relative posture between the target object and the clamp according to the target images including the clamp acquired at different times, the change value is used to characterize the degree of sliding of the target object in the clamp; based on the change value, detecting whether the target object in the clamp has fallen, or whether there is a risk of falling.

[0014] In some embodiments of the present application, based on the aforementioned scheme, determining the change value of the relative posture between the target object and the gripper according to the target images including the gripper acquired at different times includes: determining foreground data in the target images acquired at different times to obtain multiple frames of foreground data; and determining the change value of the relative posture between the target object and the gripper based on the multiple frames of foreground data.

[0015] In some embodiments of the present application, based on the aforementioned scheme, determining the change value of the relative posture between the target object and the gripper according to the target images including the gripper acquired at different times includes: segmenting the target object and the gripper in each target image through an image segmentation algorithm; and determining the change value of the relative posture between the target object and the gripper according to the posture relationship between the target object and the gripper in each target image.

[0016] In some embodiments of the present application, based on the aforementioned scheme, detecting whether the target object in the clamp has fallen or whether there is a risk of falling according to the change value includes: if the change value is greater than a first change threshold and less than a second change threshold, determining the risk value of the target object falling from the clamp based on the change value, and / or the distance between the cleaning device and the target position, and / or the type of the target object; and detecting whether there is a risk of falling of the target object in the clamp based on the risk value.

[0017] In some embodiments of the present application, based on the aforementioned scheme, detecting whether the target object in the clamp is at risk of falling according to the risk value includes: if the risk value is less than or equal to a preset risk threshold, determining that the target object in the clamp is not at risk of falling; if the risk value is greater than the preset risk threshold, determining that the target object in the clamp is at risk of falling.

[0018] In some embodiments of the present application, based on the aforementioned solution, the method further includes: if the change value is less than or equal to the first change threshold, determining that there is no risk of the target object in the gripper falling.

[0019] In some embodiments of the present application, based on the aforementioned solution, the method further includes: if the change value is greater than or equal to the second change threshold, determining that the target object in the gripper has fallen.

[0020] In some embodiments of the present application, based on the aforementioned scheme, the method further includes: if there is a risk of the target object in the gripper falling, controlling the cleaning device to perform an object re-grabbing action; if the target object in the gripper falls, controlling the cleaning device to perform an object retrieval action.

[0021] In some embodiments of the present application, based on the aforementioned scheme, controlling the cleaning device to perform an object re-grabbing action includes: controlling the cleaning device to stop moving and placing the target object grasped by the clamp on the ground; controlling the cleaning device to re-grasp the target object.

[0022] In some embodiments of the present application, based on the aforementioned scheme, the cleaning device also includes a photoelectric sensor, or a force feedback sensor, or a current detection sensor, and the detecting whether there is a grasped target object in the clamp includes: obtaining first sensing data collected by the photoelectric sensor, or the force feedback sensor, or the current detection sensor; and detecting whether there is a grasped target object in the clamp based on the first sensing data.

[0023] In some embodiments of the present application, based on the aforementioned scheme, the method further includes: in the process of the cleaning device carrying the target object, obtaining second sensor data collected by the photoelectric sensor, or the force feedback sensor, or the current detection sensor; based on the second sensor data, detecting whether the target object in the clamp falls; if the target object in the clamp falls, controlling the cleaning device to perform an object retrieval action.

[0024] In some embodiments of the present application, based on the aforementioned scheme, the controlling of the cleaning device to perform an object retrieval action includes: obtaining a first position of the cleaning device when detecting that the target object has fallen; estimating the falling area of the target object based on the first position; controlling the cleaning device to search for the target object in the falling area; if the cleaning device finds the target object in the falling area, controlling the cleaning device to re-grab the target object and transporting the re-grabbed target object to the target position.

[0025] In some embodiments of the present application, based on the aforementioned scheme, estimating the drop area of the target object based on the first position includes: obtaining the second position of the cleaning device when the target object was last detected not to have fallen; estimating the drop area of the target object based on the first position and the second position.

[0026] In some embodiments of the present application, based on the aforementioned scheme, controlling the cleaning device to search for the target object in the drop area includes: determining at least one target search point in the drop area; controlling the cleaning device to move to each target search point, and observing each target search point for a circle to search for the target object.

[0027] In some embodiments of the present application, based on the aforementioned scheme, determining at least one target search point in the drop area includes: determining multiple groups of search points in the drop area, wherein each group of search points includes at least one search point; calculating the search efficiency of each group of search points, and determining the search point in the group of search points with the highest search efficiency as the at least one target search point.

[0028] In some embodiments of the present application, based on the aforementioned scheme, controlling the cleaning device to search for the target object in the drop area includes: if the cleaning device searches for an object in the drop area, obtaining second appearance attribute information of the searched object; obtaining first appearance attribute information of the target object recorded by the cleaning device when the target object is first grasped; based on the first appearance attribute information and the second appearance attribute information, calculating the similarity between the searched object and the target object; and determining the object with the highest similarity that exceeds a preset similarity threshold as the target object.

[0029] In some embodiments of the present application, based on the aforementioned scheme, the method further includes: if the cleaning device fails to search for the target object in the drop area, controlling the cleaning device to perform other tasks, and caching the first appearance attribute information of the target object and the current carrying task information, the first appearance attribute information and the task information being used by the cleaning device to search for the target object while performing other tasks.

[0030] In some embodiments of the present application, based on the aforementioned scheme, the method also includes: if the cleaning device searches for the target object while performing other tasks, controlling the cleaning device to re-grab the target object and transport the re-grabbed target object to the target position; if the cleaning device fails to search for the target object while performing other tasks, triggering an error prompt.

[0031] Based on the cleaning equipment control solution proposed in this application, before controlling the cleaning equipment to transport the target object to the target position, it is first detected whether there is a grasped target object in the clamping jaws. If there is a grasped target object in the clamping jaws, the cleaning equipment is controlled to transport the target object to the target position. It is understandable that if there is no grasped target object in the clamping jaws, the cleaning equipment will not be controlled to perform the walking action of moving from the initial position to the target position. It can be seen that the cleaning equipment control solution proposed in this application can ensure that the target object is transported from the initial position to the target position, thereby improving the reliability of the cleaning equipment in the process of performing the transport task.

[0032] According to a third aspect of an embodiment of the present application, an object detection device is provided, which is applied to a cleaning device. The cleaning device includes a robotic arm and a gripper, wherein the gripper is provided at an end of the robotic arm away from a body of the cleaning device, and the robotic arm is used to cooperate with the gripper to grasp an object. The device includes: a first acquisition unit, for acquiring a target image including the gripper; and a first detection unit, for detecting whether a grasped target object exists in the gripper based on the target image.

[0033] According to the fourth aspect of the embodiments of the present application, a cleaning equipment control device is provided, wherein the cleaning equipment includes a robotic arm and a gripper, wherein the gripper is provided at one end of the robotic arm away from the body of the cleaning equipment, and the robotic arm is used to cooperate with the gripper to grasp an object, and the device includes: a second acquisition unit, for acquiring an initial position of the target object, and controlling the cleaning equipment to grasp the target object based on the initial position; a second detection unit, for detecting whether there is a grasped target object in the gripper; and a control unit, for controlling the cleaning equipment to transport the target object to a target position if there is a grasped target object in the gripper.

[0034] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, which includes computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that a computer device having the processor executes to implement the operations performed by the method described in any one of the first and second aspect embodiments above.

[0035] According to the sixth aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which at least one computer program instruction is stored. The at least one computer program instruction is loaded and executed by a processor to implement the operations performed by the method described in any one of the first and second aspect embodiments above.

[0036] According to the seventh aspect of the embodiments of the present application, a cleaning device is provided, which includes one or more processors and one or more memories, wherein at least one computer program instruction is stored in the one or more memories, and the at least one computer program instruction is loaded and executed by the one or more processors to implement the operations performed by the method described in any one of the first and second aspect embodiments above.

[0037] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, explaining the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0039] Figure 1 The following is a diagram showing an application scenario of a cleaning device in an embodiment of the present application;

[0040] Figure 2 A flow chart of an object detection method in an embodiment of the present application is shown;

[0041] Figure 3 A flow chart showing a method for controlling a cleaning device in an embodiment of the present application is shown;

[0042] Figure 4 A diagram showing a cleaning device in an embodiment of the present application performing a transport task is shown;

[0043] Figure 5 A demonstration diagram showing a cleaning device searching for a target object in an embodiment of the present application is shown;

[0044] Figure 6 A block diagram of an object detection device in an embodiment of the present application is shown;

[0045] Figure 7 A block diagram of a cleaning equipment control device in an embodiment of the present application is shown;

[0046] Figure 8 A schematic structural diagram of the cleaning device in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0048] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0049] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices. It should also be noted that in the accompanying drawings, certain components that do not affect the explanation of the technical solutions of this application have been omitted for clarity.

[0050] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0051] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0052] In order to make those skilled in the art better understand this application, first combine Figure 1 A brief description of the application scenarios involved in this application is given.

[0053] See also Figure 1 , shows an application scenario diagram of the cleaning equipment in an embodiment of the present application.

[0054] In this application, if Figure 1The cleaning device 101 involved can be a sweeper, a mop, or a sweeper-mop all-in-one machine, and this application does not make any specific restrictions on this. Furthermore, the cleaning device 101 involved can include a robotic arm 101 and a gripper 102, wherein the gripper 102 is provided at one end of the robotic arm 101 away from the cleaning device 100 body, and the robotic arm 101 is used to cooperate with the gripper 102 to grab objects. It should be noted that, if Figure 1 The robotic arm 101 shown is a two-degree-of-freedom robotic arm, but in other embodiments, the robotic arm may have three degrees of freedom or even four degrees of freedom, which is not specifically limited in this application.

[0055] Furthermore, the cleaning device 101 involved may also include an image acquisition device 103, which may be used to acquire images including the gripper 102. In the present application, the image acquisition device 103 may be an RGB camera, a depth camera, or a 3D imaging sensor, etc. The image acquisition device 103 may be used to acquire two-dimensional images or three-dimensional point cloud images. Figure 1 As shown, the image acquisition device 103 can be installed in front of the chassis of the cleaning device 100. In other embodiments, the image acquisition device can also be installed on the top of the chassis of the cleaning device or directly installed at the end of the clamping claw. This application does not make specific limitations on this.

[0056] In one embodiment of the present application, the cleaning device 100 can be used to perform object handling tasks on the ground. For example, the cleaning device 100 can handle the object 104 from position A to position B. Specifically, the cleaning device 100 can control the robotic arm 101 and the gripper 102 to grab the object 104 at position A, and then move it to the vicinity of position B, and place the object 104 at position B, thereby completing the handling task for the object 104.

[0057] In this application, it is understood that when cleaning equipment performs object handling tasks on the ground, it needs to grasp objects with its grippers. In order for the grippers of the cleaning equipment to successfully grasp objects, it is necessary to ensure that the cleaning equipment accurately detects the grasped objects. In this case, this application first proposes an object detection method to improve the accuracy of the cleaning equipment's detection of grasped objects.

[0058] Reference Figure 2 , shows a flow chart of the object detection method in an embodiment of the present application, which can be applied to Figure 1 The cleaning device shown can be specifically executed by a device with a computing and processing function, referring to Figure 2 As shown, the cleaning equipment control method includes at least steps 210 to 220, which are described in detail as follows:

[0059] In step 210 , a target image including the gripper is acquired.

[0060] In this application, obtaining the target image including the gripper may be performed according to the following step 211:

[0061] Step 211: Acquire a target image including the gripper captured by the image capture device.

[0062] Specifically, in the present application, since the image acquisition device is installed directly in front of the chassis of the cleaning equipment, or installed on the top of the chassis of the cleaning equipment, or directly installed at the end of the clamp, after the clamp of the cleaning equipment grasps the object, the clamp will move to the field of view of the image acquisition device with the cooperation of the robotic arm, and thus the target image including the clamp can be captured by the image acquisition device (which can be a two-dimensional image or a three-dimensional point cloud image).

[0063] It should be noted that when the gripper is grasping the target object, there may be a blind spot in the field of view of the image acquisition device. Only when the gripper moves into the field of view of the image acquisition device with the cooperation of the robotic arm can the image acquisition device capture the target image including the gripper.

[0064] It should also be noted that in the present application, the image acquisition device can acquire images continuously, that is, the image acquisition device can acquire an image once every certain time interval, for example, once every 1 second. When the gripper moves into the field of view of the image acquisition device with the cooperation of the robotic arm, the image acquisition device can acquire multiple frames of target images including the gripper.

[0065] Continue to refer to Figure 2 In step 220, based on the target image, it is detected whether there is a grasped target object in the gripper.

[0066] In this application, based on the target image, detecting whether there is a grasped object in the gripper can be performed according to the following steps 221 to 222:

[0067] Step 221 : Acquire the manipulator position parameters of the manipulator, and determine foreground data in the target image based on the manipulator position parameters, wherein the foreground data is image data that moves with the movement of the gripper.

[0068] Step 222: Detect whether there is a grasped target object in the gripper based on the foreground data.

[0069] In the present application, while the image acquisition device is acquiring the target image including the gripper, the robotic arm position parameters of the robotic arm can be recorded, wherein the robotic arm position parameters may include the size of each link in the robotic arm, the angle between each adjacent link, and the relative position relationship between the robotic arm and the image acquisition device.

[0070] In the present application, the foreground data and background data in the target image can be distinguished based on the robot arm position parameters of the robot arm. It should be noted that the foreground data is image data that moves with the movement of the gripper, such as image data of the robot arm in the image, image data of the gripper, and image data of the object grasped by the gripper. The background data is image data that does not move with the movement of the gripper, such as image data of walls, furniture, home appliances, etc. After determining the foreground data in the target image, it can be detected whether there is a grasped target object in the gripper based on the foreground data.

[0071] In the present application, it can be understood that if the target image is captured by an RGB camera, then the foreground data may actually include the contour information of the gripper; if the target image is captured by a depth camera, then the foreground data may actually include the posture point cloud information of the gripper.

[0072] In the present application, since the background data in the target image has no reference function for detecting whether there is a grasped target object in the gripper, by determining the foreground data in the target image, the noise data including the background data in the target image is actually removed. In this way, the efficiency and accuracy of the image data analysis can be improved, thereby improving the efficiency and accuracy of the detection of the grasped object in the gripper.

[0073] Furthermore, in the present application, based on the foreground data, detecting whether there is a grasped target object in the gripper can be performed according to the following steps 2221 to 2223:

[0074] Step 2221: Determine the first posture data of the gripper in the foreground data.

[0075] Step 2222: Acquire predetermined second posture data of the gripper in a closed state.

[0076] Step 2223 : Compare the first posture data and the second posture data to detect whether there is a grasped target object in the gripper.

[0077] In the present application, the gripper's posture data may be the degree of opening of the gripper's components, i.e., the degree to which the gripper's components are open. It is understood that the degree of opening of the gripper's components when gripping an object is greater than the degree of opening of the gripper's components when closed (i.e., not gripping an object). Therefore, by comparing the first gripper's posture data determined in the foreground data with the second gripper's posture data when closed, it is possible to accurately detect whether a gripped target object is present in the gripper.

[0078] In this application, based on the target image, detecting whether there is a grasped object in the gripper can be performed according to the following steps 223 to 224:

[0079] Step 223 : Segment each target object in the target image using an image segmentation algorithm, where the target object at least includes the gripper.

[0080] Step 224 : Detect whether there is a target object to be grasped in the gripper based on the posture relationship between the gripper and the target object other than the gripper.

[0081] In the present application, the image segmentation algorithm can be used to segment out the various target objects in the target image, such as segmenting out the gripper in the cleaning equipment, segmenting out the robotic arm in the cleaning equipment, and segmenting out furniture, home appliances and other objects in the environment. It can be understood that if there is an object grasped in the gripper, the image segmentation algorithm can also be used to segment out the object grasped in the gripper. Furthermore, based on the positional relationship between the gripper and the target object other than the gripper, if there is a segmented object in the gripper, it can be determined that there is a grasped target object in the gripper. It can be seen that the image segmentation algorithm can accurately detect whether there is a grasped target object in the gripper.

[0082] Based on the object detection solution proposed in this application, since the acquired target image contains the gripper, by performing image analysis on the target image, it is possible to accurately detect whether there is a grasped target object in the gripper.

[0083] The cleaning equipment proposed in this application can be used to organize debris, that is, it can perform the task of transporting objects. Based on this, this application also proposes a cleaning equipment control method to improve the reliability of the cleaning equipment in the process of performing the transport task.

[0084] Reference Figure 3 , shows a flow chart of a cleaning device control method in an embodiment of the present application, which can be executed by a device with a computing and processing function, referring to Figure 3As shown, the cleaning equipment control method includes at least steps 310 to 330, which are described in detail as follows:

[0085] Step 310 : Acquire an initial position of the target object, and based on the initial position, control the cleaning device to grasp the target object.

[0086] Step 320: Detect whether there is a target object to be grasped in the gripper.

[0087] Step 330: If there is a target object to be grasped in the gripper, control the cleaning device to move the target object to a target position.

[0088] In the present application, during the process of the cleaning device tidying up the debris, the cleaning device can detect the area to be sorted. If a target object to be transported is detected in the area to be sorted, the initial position of the target object can be determined based on the sensor data (such as point cloud data) of the target object collected by the sensor on the cleaning device (such as a radar sensor, a laser sensor, etc.). After obtaining the initial position of the target object, the cleaning device can be controlled to move to the vicinity of the initial position, and then the robotic arm of the cleaning device can be controlled to drive the gripper to move to the initial position, and the gripper can be used to grab the target object to move the grabbed target object to the target position.

[0089] It should be noted that during the process of grasping the target object and transporting the target object to the target location, the cleaning device only controls the robotic arm and gripper to perform the grasping action, controls the movement of its body from the initial position to the target location, and controls the robotic arm and gripper to place the object at the target location. However, in practice, there is no guarantee that the cleaning device's gripper actually grasps the target object. If the cleaning device's gripper does not grasp the target object, then the purpose of transporting the target object from the initial position to the target location cannot be achieved.

[0090] According to the cleaning equipment control solution proposed in the present application, before controlling the cleaning equipment to transport the target object to the target position, it is first detected whether there is a grasped target object in the clamping jaws. If there is a grasped target object in the clamping jaws, the cleaning equipment is controlled to transport the target object to the target position. It is understandable that if there is no grasped target object in the clamping jaws, the cleaning equipment will not be controlled to perform the walking action of moving from the initial position to the target position. It can be seen that the cleaning equipment control solution proposed in the present application can ensure that the target object is transported from the initial position to the target position, thereby improving the reliability of the cleaning equipment in the process of performing the transport task.

[0091] In this application, it should also be noted that if the gripper detects the presence of a grasped target object, the gripper can be considered to have successfully grasped the object. At this time, first appearance attribute information of the grasped object (i.e., the target object) in the target image can be recorded. The appearance attribute information of the object may include information such as the object's type, size, outline, position relative to the gripper, color, texture, etc.

[0092] In such Figure 3 In step 320 , it is possible to detect whether there is a grasped target object in the gripper according to the object detection method described above.

[0093] Since the object detection method as described above can accurately detect whether there is a grasped target object in the clamp, thus detecting whether there is a grasped target object in the clamp by using the object detection method as described above can further ensure that the target object is transported from the initial position to the target position, thereby improving the reliability of the cleaning equipment in performing the transport task.

[0094] Next, based on the object detection method described above, this application will explain the control logic of the cleaning device during the process of transporting the target object:

[0095] In this application, the following steps 341 to 342 may be performed:

[0096] Step 341, during the process of the cleaning equipment carrying the target object, determine the change value of the relative posture between the target object and the clamp based on the target images including the clamp acquired at different times, and the change value is used to represent the degree of sliding of the target object in the clamp.

[0097] Step 342: Detect whether the target object in the gripper has fallen or is at risk of falling based on the change value.

[0098] In the present application, the image acquisition device can continuously acquire target images, for example, acquire a target image once every 1 second. Therefore, in the process of the cleaning equipment carrying the target object, multiple frames of target images including the clamp can be acquired, and then the relative posture between the target object and the clamp can be analyzed based on the target images acquired at different times. It can be understood that if the target object slides in the clamp, the relative posture between the target object and the clamp is different in different target images. Therefore, the present application can detect whether the target object in the clamp has fallen, or whether there is a risk of falling, based on the determined change value of the relative posture between the target object and the clamp.

[0099] It should be noted that, in this application, the change in relative posture may be a change in the relative distance between the target object and the gripper, a change in the relative angle between the target object and the gripper, or a change calculated based on the relative distance and relative angle. This application does not impose any specific limitations on this.

[0100] The present application will now illustrate the steps of determining the change in the relative posture between the target object and the gripper with reference to two specific embodiments.

[0101] In one embodiment of the present application, determining the change in the relative posture between the target object and the gripper based on the target images including the gripper acquired at different times may be performed according to the following steps 3411 to 3412:

[0102] Step 3411 , determining foreground data in the target image acquired at different times to obtain multiple frames of foreground data.

[0103] Step 3412: Determine the change in relative posture between the target object and the gripper based on the multiple frames of foreground data.

[0104] Based on the object detection scheme described above, foreground data is determined from the target images acquired at different times. The foreground data and background data in the target images can be distinguished based on the position parameters of the robotic arm at different times. Furthermore, multiple frames of foreground data that moves with the movement of the gripper can be obtained, and a relative position change value can be determined based on the change in the relative position between the target object and the gripper in these multiple frames of foreground data.

[0105] In another embodiment of the present application, determining the change in the relative posture between the target object and the gripper based on the target images including the gripper acquired at different times may also be performed according to the following steps 3413 to 3414:

[0106] Step 3413: Segment the target object and the gripper in each target image using an image segmentation algorithm.

[0107] Step 3414: Determine a change value of the relative posture between the target object and the gripper according to the posture relationship between the target object and the gripper in each target image.

[0108] Based on the object detection scheme described above, segmenting the target object and the gripper in target images acquired at different times can be achieved using an image segmentation algorithm. After segmenting the target object and the gripper in each target image, a relative position change value can be determined based on the change in the relative position of the target object and the gripper in each target image.

[0109] In the present application, the detection of whether the target object in the gripper has fallen or is at risk of falling based on the change value may be performed according to steps 3421 to 3423 as follows:

[0110] Step 3421: If the change value is greater than the first change threshold and less than the second change threshold, determine the risk value of the target object falling from the clamp based on the change value, and / or the distance between the cleaning device and the target position, and / or the type of the target object; and detect whether there is a risk of the target object in the clamp falling based on the risk value.

[0111] Step 3422: If the change value is less than or equal to the first change threshold, determine that there is no risk of the target object in the gripper falling.

[0112] Step 3423: If the change value is greater than or equal to the second change threshold, determine that the target object in the gripper has fallen.

[0113] In the present application, the step of detecting whether there is a risk of the target object in the gripper falling according to the risk value may be performed according to the following steps 34211 to 34212:

[0114] Step 34211: If the risk value is less than or equal to the preset risk threshold, determine that there is no risk of the target object in the gripper falling.

[0115] Step 34212: If the risk value is greater than the preset risk threshold, determine that the target object in the gripper is at risk of falling.

[0116] In the present application, if the change value is greater than the first change threshold and less than the second change threshold, it means that when the cleaning equipment is performing the handling task, there is a certain amount of sliding of the target object grasped in the clamp. At this time, the risk value of the target object falling from the clamp can be determined based on the change value (that is, considering the sliding degree of the target object in the clamp, if the sliding degree is greater, the risk of the target object falling from the clamp is higher), and / or the distance between the cleaning equipment and the target position (that is, considering whether the cleaning equipment is currently far away from the target position, if the distance is farther, the risk of the target object falling from the clamp is higher), and / or the type of the target object (that is, considering whether the type of target object (such as the shape of the object) is easy to fall, if the shape of the target object is a regular long strip, the risk of the target object falling from the clamp is higher). It can be understood that the higher the risk value, the greater the possibility of the target object falling from the clamp during the handling process.

[0117] Furthermore, when detecting whether the target object in the gripper is at risk of falling based on the risk value, the present application may specifically detect the risk by comparing the risk value with a preset risk threshold. That is, if the risk value is less than or equal to the preset risk threshold, it is determined that the target object in the gripper is not at risk of falling; if the risk value is greater than the preset risk threshold, it is determined that the target object in the gripper is at risk of falling.

[0118] In the present application, if the change value is less than or equal to the first change threshold, it indicates that the sliding degree of the target object grasped in the clamp is weak during the cleaning equipment's handling task. At this time, it can be determined that there is no risk of the target object in the clamp falling.

[0119] In the present application, if the change value is greater than or equal to the second change threshold, it indicates that the target object in the gripper has fallen. It should be noted that the second change threshold can be pre-set to a larger value, such as the length of the target object. If the target object in the gripper falls, it can be considered that the change value of the relative position between the target object and the gripper is a larger value (at least greater than the second change threshold).

[0120] Based on the cleaning equipment control scheme proposed in this application, the target images acquired at different times are used to determine the change in the relative position between the target object and the gripper. Furthermore, by comparing the change in value with the preset first and second change thresholds, it is determined whether the target object in the gripper of the cleaning equipment has fallen during the handling task, or whether there is a risk of falling. In this way, when the cleaning equipment is performing an object handling task, a more accurate reference can be provided for the subsequent control strategy to be implemented by the cleaning equipment, thereby ensuring that the target object is carried to the target position and improving the reliability of the cleaning equipment during the handling task.

[0121] Furthermore, in this application, the following steps 351 to 352 may also be performed:

[0122] Step 351: If there is a risk of the target object in the gripper falling, control the cleaning device to perform an object re-grasping action.

[0123] Step 352: If the target object in the gripper falls, control the cleaning device to perform an object retrieval operation.

[0124] In the present application, when there is a risk of the target object in the clamping jaws falling, in order to prevent the target object from falling from the clamping jaws during the handling process, the cleaning device can be controlled to perform an object re-grasping action.

[0125] Specifically, the control of the cleaning device to perform the object re-grasping action may be performed according to the following steps 3521 to 3522:

[0126] Step 3521: Control the cleaning device to stop moving and place the target object grasped by the clamp on the ground.

[0127] Step 3522: Control the cleaning device to re-grasp the target object.

[0128] In the present application, after controlling the cleaning device to re-grasp the target object, the cleaning device may continue to be controlled to perform the transporting task of transporting the target object to a target location.

[0129] In the present application, the cleaning device may further include a photoelectric sensor, a force feedback sensor, or a current detection sensor.

[0130] On the basis that the cleaning device includes a photoelectric sensor, a force feedback sensor, or a current detection sensor, Figure 3 In step 320, the detection of whether there is a grasped target object in the gripper can be performed according to the following steps 321 to 322:

[0131] Step 321: Acquire first sensing data collected by the photoelectric sensor, the force feedback sensor, or the current detection sensor.

[0132] Step 322: Detect whether there is a grasped target object in the gripper based on the first sensing data.

[0133] Specifically, in the present application, the photoelectric sensor can be installed on the gripper, and then it can be determined whether the gripper has grasped the target object based on whether the photoelectric sensor detects a changing photoelectric signal (i.e., the first sensing data) (if the gripper grasps the target object, the photoelectric signal emitted by the photoelectric sensor will be blocked by the target object).

[0134] In the present application, the force feedback sensor can be installed on the gripper or the robotic arm, and then whether the gripper has grasped the target object can be determined based on whether the force feedback sensor detects a changing force signal (i.e., the first sensing data) (if the gripper grasps the target object, the force signal detected by the force feedback sensor will become larger).

[0135] In the present application, the current detection sensor can be installed on the gripper or on the drive motor of the robotic arm, and then it can be determined whether the gripper has grasped the target object based on whether the current detection sensor detects a changing electrical signal (i.e., the first sensor data) (if the gripper grasps the target object, the current detection sensor detects that the electrical signal of the drive motor will become larger).

[0136] In this application, the following steps 351 to 353 may also be performed:

[0137] Step 351 : Acquire second sensing data collected by the photoelectric sensor, the force feedback sensor, or the current detection sensor during the process of the cleaning device carrying the target object.

[0138] Step 352: Detect whether the target object in the gripper falls according to the second sensing data.

[0139] Step 353: If the target object in the gripper falls, control the cleaning device to perform an object retrieval operation.

[0140] Specifically, in the present application, if a photoelectric sensor is installed on the gripper, when the target object on the gripper falls, the photoelectric signal (i.e., the second sensor data) emitted by the photoelectric sensor will be detected by the photoelectric sensor itself; if a force feedback sensor is installed on the gripper or the robotic arm, when the target object on the gripper falls, the force signal (i.e., the second sensor data) detected by the force feedback sensor will decrease; if a current detection sensor is installed on the gripper or the driving motor of the robotic arm, when the target object on the gripper falls, the electrical signal (i.e., the second sensor data) detected by the current detection sensor of the driving motor will decrease. In this way, whether the target object in the gripper has fallen can be detected based on the second sensor data collected by the photoelectric sensor, the force feedback sensor, or the current detection sensor.

[0141] Next, this application will further explain how to control the cleaning device to perform the object retrieval action:

[0142] In the present application, the control of the cleaning device to perform the object retrieval action may be performed according to the following steps 361 to 364:

[0143] Step 361: Acquire a first position of the cleaning device when detecting that the target object has fallen.

[0144] Step 362: Estimate the drop area of the target object based on the first position.

[0145] Step 363: Control the cleaning device to search for the target object in the drop area.

[0146] In step 364 , if the cleaning device finds the target object in the drop area, the cleaning device is controlled to re-grab the target object and move the re-grabbed target object to the target location.

[0147] In the present application, in order to control the cleaning device to perform an object retrieval action, it is first necessary to estimate the drop area of the target object so that the cleaning device can search for the target object in the drop area.

[0148] In the present application, estimating the falling area of the target object based on the first position may be performed according to the following steps 3621 to 3622:

[0149] Step 3621: Obtain the second position of the cleaning device when the last detection of the target object did not occur.

[0150] Step 3622: Estimate the drop area of the target object based on the first position and the second position.

[0151] In order to make those skilled in the art better understand this application, Figure 4 , described with a specific embodiment.

[0152] See also Figure 4 , shows a demonstration diagram of the cleaning equipment in an embodiment of the present application performing a transport task.

[0153] like Figure 4 As shown, the object transporting task performed by the cleaning device 100 is to transport the target object from the initial position A to the target position B. During this process, the cleaning device 100 is first controlled to grab the target object at the initial position A. After grabbing the target object, the cleaning device 100 is controlled to move in the object transporting direction. During the movement of the cleaning device 100, the target object actually falls from the clamping claw at position D2. Before position D2 (i.e., detection at position D1), the cleaning device can detect that the target object has not fallen from the clamping claw. When the cleaning device detects the target object at position D3, it can detect that the target object has fallen from the clamping claw. It is understandable that there is a certain time delay between the cleaning device detecting that the target object has fallen and the target object actually falling. Any historical position within this period of time may be the position where the target object fell from the clamping claw.

[0154] Based on this, in the present application, the second position D1 of the cleaning device when the target object was not dropped the last time it was detected can be obtained, and based on the first position D3 and the second position D1 when the cleaning device detected that the target object had dropped, the drop area C of the target object can be estimated. For example, the xy coordinates of the second position can be used as the minimum xy coordinates of the drop area, and the xy coordinates of the first position can be used as the maximum xy coordinates of the drop area, thereby determining the drop area C of the target object.

[0155] In other embodiments, considering that the target object may roll to other locations when it falls at a certain location, the initially determined drop area may be expanded according to a certain proportion to obtain the expanded drop area C. The advantage of doing so is that it increases the possibility of the cleaning equipment searching for the target object.

[0156] In this application, the drop area of the target location is estimated based on the first location. Alternatively, the drop area is determined on the ground based on certain rules using only the first location as a reference location. This application does not specifically limit this.

[0157] In the present application, after the drop area of the target object is determined, the cleaning device may be controlled to search for the target object in the drop area.

[0158] Specifically, controlling the cleaning device to search for the target object in the drop area may be performed according to the following steps 3631 to 3632:

[0159] Step 3631: Determine at least one target search point in the drop area.

[0160] Step 3632: Control the cleaning device to move to each target search point, and observe each target search point for a circle to search for the target object.

[0161] In order to make those skilled in the art better understand this application, Figure 5 A specific embodiment is used for description.

[0162] See also Figure 5 , shows a demonstration diagram of the cleaning equipment searching for a target object in an embodiment of the present application.

[0163] like Figure 5 As shown in sub-figure (a), after the target search points F, G, and H are determined in the drop area C, the cleaning device 100 can be controlled to move to the target search points F, G, and H in sequence, and rotate and observe the target search points F, G, and H for one circle respectively, so as to search for the target object within a certain area centered on the search point (for example, a certain area centered on the search point F is area f).

[0164] Also like Figure 5 As shown in sub-figure (b), after the target search points I, J, K, and L are determined in the drop area C, the cleaning device 100 can be controlled to move to the target search points I, J, K, and L in sequence, and rotate and observe the target search points I, J, K, and L for one circle respectively to search for the target object within a certain area centered on the search point.

[0165] In the present application, determining at least one target search point in the drop area may be performed according to the following steps 36311 to 36312:

[0166] Step 36311: Determine multiple groups of search points in the drop area, where each group of search points includes at least one search point.

[0167] Step 36312: Calculate the search efficiency of each group of search points, and determine the search point in the group of search points with the highest search efficiency as the at least one target search point.

[0168] In this application, reference is made to Figure 5 As shown, it is possible to determine in the drop area as follows Figure 5 A set of search points F, G, H shown in sub-graph (a) of Figure 5A set of search points I, J, K, and L as shown in sub-figure (b) of FIG 1 are then used to calculate the search efficiency of each of the two sets of search points. Factors such as the required search time and the search range covered may be considered when calculating the search efficiency. After calculating the search efficiency of each set of search points, the search point in the set with the highest search efficiency may be determined as the at least one target search point. In this application, the algorithm for calculating the search efficiency may be designed based on actual conditions and is not specifically limited in this application.

[0169] In the present application, by determining multiple groups of search points in the drop area and determining a group of search points with the highest search efficiency as the at least one target search point, the search efficiency of searching for the target object in the drop area can be improved.

[0170] In the present application, controlling the cleaning device to search for the target object in the drop area may also be performed according to the following steps 3633 to 3636:

[0171] Step 3633: If the cleaning device finds an object in the drop area, obtain second appearance attribute information of the found object.

[0172] Step 3634: Obtain first appearance attribute information of the target object recorded when the cleaning device first grasps the target object.

[0173] Step 3635: Calculate the similarity between the searched object and the target object based on the first appearance attribute information and the second appearance attribute information.

[0174] Step 3636: Determine the object with the highest similarity that exceeds a preset similarity threshold as the target object.

[0175] In the present application, since the cleaning device may search for multiple objects in the drop area, in this case, the first appearance attribute information of the searched object (such as the type of object, size, outline, position relative to the gripper, color, texture, etc.) can be obtained, and the first appearance attribute information of the target object recorded by the cleaning device when the target object is first grasped can be obtained. Then, based on the first appearance attribute information and the second appearance attribute information, the similarity between the searched object and the target object is calculated, and the object with the highest similarity and exceeding the preset similarity threshold is determined as the target object. It can be understood that by calculating the similarity between the searched object and the target object based on the appearance attribute information of the object, the target object can be accurately determined from the searched objects, so that the target object can be accurately retrieved, ensuring that the target object is transported from the initial position to the target position, thereby improving the reliability of the cleaning device in performing the transport task.

[0176] In this application, the following step 365 may also be performed:

[0177] Step 365: If the cleaning device fails to find the target object in the drop area, the cleaning device is controlled to perform other tasks, and the first appearance attribute information of the target object and the current transport task information are cached. The first appearance attribute information and the task information are used by the cleaning device to search for the target object while performing other tasks.

[0178] Furthermore, the following steps 366 to 367 may be performed:

[0179] Step 366: If the cleaning device searches for the target object while performing other tasks, control the cleaning device to re-grasp the target object and move the re-grabbed target object to the target location.

[0180] Step 367: If the cleaning device fails to find the target object while performing other tasks, an error message is triggered.

[0181] In actual applications, for cleaning equipment equipped with a robotic arm and a gripper, after grabbing an object, it will perform the object-carrying action. Due to the possibility of instability in the gripper itself when gripping an object, coupled with acceleration and deceleration, bumps, or collisions between the gripper and the environment during the handling process, the gripper may fall during the handling process. Based on the cleaning equipment control scheme proposed in this application, before controlling the cleaning equipment to carry the target object to the target position, it first detects whether the gripper is holding the target object. If the gripper is holding the target object, the cleaning equipment is controlled to carry the target object to the target position. It is understandable that if the gripper is not holding the target object, the cleaning equipment will not be controlled to perform the walking action of moving from the initial position to the target position. In the process of the cleaning equipment performing the object handling task, by detecting whether the target object in the gripper of the cleaning equipment has fallen, a more accurate reference basis can be provided for the cleaning equipment to perform what kind of control strategy in the future, thereby ensuring that the target object is carried to the target position and improving the reliability of the cleaning equipment in the process of performing the handling task.

[0182] The following describes an apparatus embodiment of the present application, which can be used to perform the object detection method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the object detection method of the present application.

[0183] See also Figure 6, shows a block diagram of an object detection device in an embodiment of the present application, wherein the device is applied to a cleaning device, wherein the cleaning device includes a robotic arm and a gripper, wherein the gripper is provided at one end of the robotic arm away from the cleaning device body, and the robotic arm is used to cooperate with the gripper to grasp objects.

[0184] like Figure 6 As shown, the object detection device 600 according to an embodiment of the present application includes: a first acquisition unit 601 and a first detection unit 602.

[0185] The first acquisition unit 601 is used to acquire a target image including the gripper; and the first detection unit 602 is used to detect whether there is a grasped target object in the gripper based on the target image.

[0186] In some embodiments of the present application, based on the aforementioned solution, the cleaning device further includes an image acquisition device, and the first acquisition unit 601 is configured to: acquire a target image including the clamping claw acquired by the image acquisition device.

[0187] In some embodiments of the present application, based on the aforementioned scheme, the first detection unit 602 is configured to: obtain the robotic arm position parameters of the robotic arm, and determine the foreground data in the target image based on the robotic arm position parameters, wherein the foreground data is image data that moves with the movement of the gripper; based on the foreground data, detect whether there is a grasped target object in the gripper.

[0188] In some embodiments of the present application, based on the aforementioned scheme, the first detection unit 602 is also configured to: determine the first posture data of the gripper in the foreground data; obtain predetermined second posture data of the gripper in a closed state; and compare the first posture data and the second posture data to detect whether there is a grasped target object in the gripper.

[0189] In some embodiments of the present application, based on the aforementioned scheme, the first detection unit 602 is configured to: segment the target objects in the target image through an image segmentation algorithm, where the target objects include at least the gripper; and detect whether there is a grasped target object in the gripper based on the posture relationship between the target objects other than the gripper and the gripper.

[0190] The following describes another device embodiment of the present application, which can be used to execute the cleaning equipment control method in the above embodiment of the present application. For details not disclosed in the device embodiment of the present application, please refer to the above embodiment of the cleaning equipment control method of the present application.

[0191] See also Figure 7, shows a block diagram of a cleaning equipment control device in an embodiment of the present application, wherein the cleaning equipment includes a robotic arm and a gripper, wherein the gripper is provided at one end of the robotic arm away from the cleaning equipment body, and the robotic arm is used to cooperate with the gripper to grasp objects.

[0192] like Figure 7 As shown, the cleaning equipment control device 700 according to an embodiment of the present application includes: a second acquisition unit 701, a second detection unit 702 and a control unit 703.

[0193] Among them, the second acquisition unit 701 is used to obtain the initial position of the target object, and based on the initial position, control the cleaning equipment to grasp the target object; the second detection unit 702 is used to detect whether there is a grasped target object in the gripper; the control unit 703 is used to control the cleaning equipment to move the target object to the target position if there is a grasped target object in the gripper.

[0194] In some embodiments of the present application, based on the aforementioned solution, the second detection unit 702 is configured to: detect whether there is a grasped target object in the gripper according to the object detection method described above.

[0195] In some embodiments of the present application, based on the aforementioned scheme, the device further includes: a detection unit, used to determine the change value of the relative posture between the target object and the clamp based on the target images including the clamp acquired at different times during the process of the cleaning equipment transporting the target object, the change value being used to characterize the degree of sliding of the target object in the clamp; and based on the change value, detecting whether the target object in the clamp has fallen or whether there is a risk of falling.

[0196] In some embodiments of the present application, based on the aforementioned scheme, the detection unit is configured to: determine foreground data in the target images acquired at different times to obtain multiple frames of foreground data; and determine the change value of the relative posture between the target object and the gripper based on the multiple frames of foreground data.

[0197] In some embodiments of the present application, based on the aforementioned scheme, the detection unit is configured to: segment the target object and the gripper in each target image through an image segmentation algorithm; and determine the change value of the relative posture between the target object and the gripper according to the posture relationship between the target object and the gripper in each target image.

[0198] In some embodiments of the present application, based on the aforementioned scheme, the detection unit is configured as follows: if the change value is greater than a first change threshold and less than a second change threshold, determine the risk value of the target object falling from the clamp based on the change value, and / or the distance between the cleaning device and the target position, and / or the type of the target object; and detect whether there is a risk of the target object in the clamp falling based on the risk value.

[0199] In some embodiments of the present application, based on the aforementioned scheme, the detection unit is configured as follows: if the risk value is less than or equal to a preset risk threshold, it is determined that there is no risk of the target object in the gripper falling; if the risk value is greater than the preset risk threshold, it is determined that there is a risk of the target object in the gripper falling.

[0200] In some embodiments of the present application, based on the aforementioned solution, the detection unit is configured to: if the change value is less than or equal to the first change threshold, determine that there is no risk of the target object in the gripper falling.

[0201] In some embodiments of the present application, based on the aforementioned solution, the detection unit is configured to: determine that the target object in the gripper has fallen if the change value is greater than or equal to the second change threshold.

[0202] In some embodiments of the present application, based on the aforementioned scheme, the control unit 703 is configured to: if there is a risk of the target object in the gripper falling, control the cleaning device to perform an object re-grasping action; if the target object in the gripper falls, control the cleaning device to perform an object retrieval action.

[0203] In some embodiments of the present application, based on the aforementioned solution, the control unit 703 is configured to: control the cleaning device to stop moving and place the target object grasped by the clamp on the ground; and control the cleaning device to grasp the target object again.

[0204] In some embodiments of the present application, based on the aforementioned scheme, the cleaning device also includes a photoelectric sensor, or a force feedback sensor, or a current detection sensor, and the second detection unit 702 is configured to: obtain first sensor data collected by the photoelectric sensor, or the force feedback sensor, or the current detection sensor; and detect whether there is a grasped target object in the gripper based on the first sensor data.

[0205] In some embodiments of the present application, based on the aforementioned scheme, the control unit 703 is configured to: obtain second sensing data collected by the photoelectric sensor, the force feedback sensor, or the current detection sensor during the process of the cleaning device carrying the target object; detect whether the target object in the gripper falls according to the second sensing data; if the target object in the gripper falls, control the cleaning device to perform an object retrieval action.

[0206] In some embodiments of the present application, based on the aforementioned scheme, the control unit 703 is configured to: obtain the first position of the cleaning device when it detects that the target object has fallen; estimate the falling area of the target object based on the first position; control the cleaning device to search for the target object in the falling area; if the cleaning device finds the target object in the falling area, control the cleaning device to re-grab the target object and move the re-grabbed target object to the target position.

[0207] In some embodiments of the present application, based on the aforementioned scheme, the control unit 703 is configured to: obtain the second position of the cleaning device when it last detected that the target object did not fall; and estimate the falling area of the target object based on the first position and the second position.

[0208] In some embodiments of the present application, based on the aforementioned scheme, the control unit 703 is configured to: determine at least one target search point in the drop area; control the cleaning device to move to each target search point, and observe each target search point for a circle to search for the target object.

[0209] In some embodiments of the present application, based on the aforementioned scheme, the control unit 703 is configured to: determine multiple groups of search points in the drop area, wherein each group of search points includes at least one search point; calculate the search efficiency of each group of search points, and determine the search point in the group of search points with the highest search efficiency as the at least one target search point.

[0210] In some embodiments of the present application, based on the aforementioned scheme, the control unit 703 is configured to: if the cleaning device searches for an object in the drop area, obtain the second appearance attribute information of the searched object; obtain the first appearance attribute information of the target object recorded by the cleaning device when the target object is first grasped; based on the first appearance attribute information and the second appearance attribute information, calculate the similarity between the searched object and the target object; and determine the object with the highest similarity that exceeds a preset similarity threshold as the target object.

[0211] In some embodiments of the present application, based on the aforementioned scheme, the control unit 703 is configured as follows: if the cleaning device fails to search for the target object in the drop area, control the cleaning device to perform other tasks, and cache the first appearance attribute information of the target object and the current handling task information, and the first appearance attribute information and the task information are used by the cleaning device to search for the target object while performing other tasks.

[0212] In some embodiments of the present application, based on the aforementioned scheme, the control unit 703 is configured as follows: if the cleaning device searches for the target object while performing other tasks, control the cleaning device to re-grab the target object and move the re-grabbed target object to the target position; if the cleaning device fails to search for the target object while performing other tasks, trigger an error prompt.

[0213] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that a computer device with the processor executes to implement the operations performed by the object detection method and the cleaning equipment control method as described above.

[0214] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores at least one computer program instruction, and the at least one computer program instruction is loaded and executed by a processor to implement the operations performed by the object detection method and the cleaning equipment control method as described above.

[0215] Based on the same inventive concept, the present application also provides a cleaning device, referring to Figure 8 , shows a structural schematic diagram of the cleaning equipment in an embodiment of the present application, wherein the cleaning equipment includes one or more memories 804, one or more processors 802, and at least one computer program (computer program instruction) stored in the memory 804 and executable on the processor 802. When the processor 802 executes the computer program, the object detection method and the cleaning equipment control method as described above are implemented.

[0216] Among them, Figure 8In the embodiment of the present invention, a bus architecture (represented by bus 800) is shown. Bus 800 may include any number of interconnected buses and bridges, and bus 800 links together various circuits including one or more processors represented by processor 802 and memory represented by memory 804. Bus 800 may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 805 provides an interface between bus 800 and receiver 801 and transmitter 803. Receiver 801 and transmitter 803 may be the same component, namely a transceiver, which provides a unit for communicating with various other devices over a transmission medium. Processor 802 is responsible for managing bus 800 and general processing, while memory 804 may be used to store data used by processor 802 when performing operations.

[0217] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored as one or more instructions or codes on or transmitted via a computer-readable medium. Other examples and implementations are within the scope and spirit of this application and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hardwiring, or a combination of any of these. Furthermore, the functional units may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0218] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0219] The units described as separate components may or may not be physically separate, and the components of the control device may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0220] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store computer program instructions.

[0221] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of the claims of the present application.

Claims

1. An object detection method, characterized in that: The method is applied to a cleaning device, the cleaning device comprising a robotic arm and a gripper, the gripper being provided at an end of the robotic arm away from a body of the cleaning device, the robotic arm being used to cooperate with the gripper to grasp an object, the method comprising: Acquiring a target image including the gripper; Based on the target image, it is detected whether there is a grasped target object in the gripper.

2. The method according to claim 1, characterized in that The cleaning device further includes an image acquisition device, and the acquisition of a target image including the clamping claw includes: A target image including the gripper captured by the image capture device is acquired.

3. The method according to claim 1, characterized in that The detecting whether there is a grasped target object in the gripper based on the target image includes: Obtaining a manipulator position parameter of the manipulator, and determining foreground data and background data in the target image based on the manipulator position parameter, wherein the foreground data is image data that moves with the movement of the gripper, and the background data is image data that does not move with the movement of the gripper; Based on the foreground data, it is detected whether there is a grasped target object in the gripper.

4. The method according to claim 3, characterized in that The detecting whether there is a grasped target object in the gripper based on the foreground data includes: determining first posture data of the gripper in the foreground data; Acquiring predetermined second posture data of the clamping jaw in a closed state; The first posture data and the second posture data are compared to detect whether there is a grasped target object in the gripper.

5. The method according to claim 1, characterized in that The detecting whether there is a grasped target object in the gripper based on the target image includes: Segmenting each target object in the target image using an image segmentation algorithm, wherein the target object at least includes the gripper; According to the positional relationship between the target object other than the gripper and the gripper, it is detected whether there is a grasped target object in the gripper.

6. A cleaning equipment control method, characterized in that: The cleaning device includes a robotic arm and a gripper, wherein the gripper is provided at an end of the robotic arm away from a body of the cleaning device, and the robotic arm is used to cooperate with the gripper to grasp an object. The method includes: Obtaining an initial position of a target object, and based on the initial position, controlling the cleaning device to grasp the target object; detecting whether there is a grasped target object in the gripper; If there is a target object to be grasped in the clamping claw, the cleaning device is controlled to move the target object to a target position.

7. The method according to claim 6, characterized in that The detecting whether there is a grasped target object in the gripper includes: According to the method according to any one of claims 1 to 5, it is detected whether there is a grasped target object in the gripper.

8. The method according to claim 7, characterized in that The method further comprises: During the process of the cleaning device carrying the target object, determining a change value of the relative posture between the target object and the clamping jaw based on target images including the clamping jaw acquired at different times, wherein the change value is used to represent the degree of sliding of the target object in the clamping jaw; According to the change value, it is detected whether the target object in the clamping jaw has fallen or whether there is a risk of falling.

9. The method according to claim 8, characterized in that The determining of a change in the relative posture between the target object and the gripper based on target images including the gripper acquired at different times includes: Determining foreground data in the target images acquired at different times to obtain multiple frames of foreground data; Determine a change in the relative posture between the target object and the gripper based on the multiple frames of foreground data.

10. The method according to claim 8, characterized in that The determining of a change in the relative posture between the target object and the gripper based on target images including the gripper acquired at different times includes: Segmenting the target object and the gripper in each target image using an image segmentation algorithm; According to the posture relationship between the target object and the gripper in each target image, a change value of the relative posture between the target object and the gripper is determined.

Images