Cleaning device control method and related device

Abstract

A cleaning device control method. A cleaning device (400) comprises a gripping arm (500), and the method comprises: acquiring sensing image information of a first target area; on the basis of the sensing image information, determining whether there is an object to be gripped; when there is an object to be gripped, acquiring image information to be analyzed of the object to be gripped; on the basis of the image information to be analyzed, determining gripping parameter information of the object to be gripped; and on the basis of the gripping parameter information, controlling the gripping arm (500) to perform a gripping operation on the object to be gripped.

Description

Cleaning device control method and related device

[0001] Cross-reference to Related Applications

[0002] This application claims priority to Chinese Patent Application No. 2024113225577, filed on September 20, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates to the technical field of cleaning device control, and more particularly to a cleaning device control method and related device. BACKGROUND

[0004] In recent years, with the rapid development of smart home and artificial intelligence technology, intelligent cleaning robots have become increasingly popular as an auxiliary tool in family life. Such devices can automatically complete the cleaning of the floor, helping users reduce the burden of daily cleaning.

[0005] However, the mainstream intelligent cleaning robots on the current market mainly focus on floor cleaning tasks. For scattered items on the floor such as shoes, clothes, paper balls, etc., existing intelligent cleaning robots cannot effectively identify, grasp, and organize them. This results in users still needing to manually organize these items, failing to achieve comprehensive automated cleaning and organization. In addition, some existing intelligent cleaning robots have simple obstacle avoidance functions, but lack the ability to classify and autonomously organize objects, failing to meet users' demand for higher home neatness. SUMMARY

[0006] The present disclosure provides a cleaning device control method and related device, aiming to at least solve one of the technical problems existing in the prior art or related art.

[0007] In a first aspect, the embodiments of the present disclosure provide a cleaning device control method, the cleaning device comprising a grasping manipulator, the method comprising:

[0008] obtaining perception image information of a first target area;

[0009] determining whether there is a to-be-grasped object based on the perception image information;

[0010] in a case where the to-be-grasped object exists, obtaining to-be-analyzed image information of the to-be-grasped object;

[0011] determining grasping parameter information of the to-be-grasped object based on the to-be-analyzed image information; and

[0012] controlling the grasping manipulator to perform a grasping operation on the to-be-grasped object based on the grasping parameter information.

[0013] In a second aspect, the present disclosure further provides a cleaning device control apparatus, comprising:

[0014] a first obtaining unit, configured to obtain perception image information of a first target region;

[0015] a first determining unit, configured to determine, based on the perception image information, whether there is an object to be grabbed;

[0016] a second obtaining unit, configured to, in a case where there is the object to be grabbed, obtain image information to be analyzed of the object to be grabbed;

[0017] a second determining unit, configured to determine, based on the image information to be analyzed, grabbing parameter information of the object to be grabbed; and

[0018] a control unit, configured to control a grabbing manipulator to perform a grabbing operation on the object to be grabbed based on the grabbing parameter information.

[0019] In a third aspect, the present disclosure further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor is configured to implement the steps of the cleaning device control method according to any one of the first aspect when executing the computer program stored in the memory.

[0020] In a fourth aspect, the present disclosure further provides a computer readable storage medium, comprising a computer program stored thereon, wherein the computer program, when executed by a processor, implements the cleaning device control method according to any one of the first aspect.

[0021] In a fifth aspect, the present disclosure further provides a cleaning device, comprising the electronic device according to the third aspect. BRIEF DESCRIPTION OF DRAWINGS

[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of preferred embodiments, and are not intended to limit the present disclosure. Moreover, like reference numerals in the various drawings indicate like elements. The drawings are as follows:

[0023] FIG. 1 is a flowchart of a cleaning device control method according to an embodiment of the present disclosure.

[0024] FIG. 2 is a structural schematic diagram of a cleaning device control apparatus according to an embodiment of the present disclosure.

[0025] FIG. 3 is a structural schematic diagram of an electronic device according to an embodiment of the present disclosure.

[0026] FIG. 4 is a structural schematic diagram of a cleaning device according to an embodiment of the present disclosure.

[0027] BRIEF DESCRIPTION OF DRAWINGS

[0028] cleaning device control device 200; first acquisition unit 201; first determination unit 202; second acquisition unit 203; second determination unit 204; control unit 205; electronic device 300; including memory 310; processor 320; computer program 311; electronic device 300; cleaning device 400; grabbing manipulator 500. Embodiments of the present application

[0029] The terms "first", "second", "third", "fourth" etc. (if any) in the description and claims of the present disclosure and above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices. The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all.

[0030] The cleaning device provided by the present disclosure is composed of a cleaning device body (i.e. a chassis) and a manipulator module installed on the chassis. The manipulator module can have multiple degrees of freedom, including telescopic function, rotation function around different joints, and the ability to rotate horizontally around the chassis, which can flexibly cope with complex operating environments. The end of the grabbing manipulator is equipped with a precise gripper, which can perform actions such as grabbing, carrying and placing according to different task requirements, and has precise rotation ability to ensure the completion of various complex operation tasks in limited space. In addition, the gripper can be self-adapted according to the shape, size and weight of the object to ensure stable grabbing, and can also protect fragile objects by adjusting the force.

[0031] The chassis of the cleaning device integrates a high-performance drive wheel system and cleaning components. The drive wheels support multiple motion modes, including straight-line motion, turning, acceleration, and deceleration, enabling the device to flexibly navigate different cleaning environments. The cleaning components on the chassis are designed to efficiently handle dust, stains, and other debris on the ground, and can adapt to different floor materials such as ceramic tiles, wooden floors, and carpets. The cleaning mode can be adjusted according to specific cleaning tasks. The chassis also has strong load-carrying capacity, which can stably carry the grabbing manipulator and other equipment, ensuring smooth movement during operation.

[0032] In addition to the grabbing and cleaning functions, the device has autonomous navigation and environmental perception capabilities. The chassis can be equipped with various advanced sensors: a depth camera to obtain three-dimensional spatial information around the device, helping it identify obstacles and navigation paths; a TOF (Time of Flight) sensor to provide accurate distance measurement data by measuring light reflection time; a laser radar to scan the surrounding environment and generate real-time maps, supporting autonomous navigation and obstacle avoidance; and an RGB camera to capture color images of the environment, assisting in identifying objects and paths around it. These sensors work together to ensure that the cleaning device can perceive and adapt to environmental changes in real time, avoid collisions and stalls, and successfully complete cleaning tasks.

[0033] The end gripper not only has the ability to perform operations, but also is equipped with high-precision RGB cameras to obtain image information of the objects to be grabbed. Through real-time image analysis, the grabbing manipulator can identify the shape, color, material, and other information of the object, further adjust the grabbing parameters, and improve the accuracy and success rate of grabbing. In addition, these cameras can also be used to monitor the grabbing process to ensure that the grabbing manipulator operates safely and efficiently. In some complex environments, the image acquisition module of the grabbing manipulator can assist in detailed object classification, placement, and other operations.

[0034] To ensure the coordinated work of the device, the chassis and the grabbing manipulator can move independently or work together. After grabbing an object, the grabbing manipulator can move to the designated location with the chassis and place the object. During this process, the two work smoothly, enabling the device to perform both cleaning and object handling tasks. In more complex task scenarios, the control system can dynamically adjust the motion mode of the grabbing manipulator and the chassis according to task requirements, enabling them to perform tasks independently or coordinate to complete more complex operations according to actual needs.

[0035] The cleaning device can also support remote control and automatic operation by the user. The user can monitor the device throughout the process, set cleaning paths, grabbing tasks, and placement areas, etc. through a mobile application. The intelligent algorithm built into the device can automatically adjust the action strategy based on the environment and tasks, so that it can still complete the task efficiently and intelligently in complex environments. At the same time, the device supports real-time state feedback. When encountering obstacles or being unable to complete the task, the device can feedback to the user and wait for further instructions to ensure the safety and reliability of the operation.

[0036] Referring to FIG. 1, a flowchart of a cleaning device control method according to an embodiment of the present disclosure is shown. The cleaning device control method can include:

[0037] S110, obtaining perception image information of a first target area.

[0038] By way of example, the first target area is an area of interest of the sensing system before the cleaning device performs a cleaning or object grabbing task. In this area, the device needs to analyze the environmental conditions and identify whether there are objects to be cleaned or grabbed. The cleaning device can automatically locate the first target area according to the user's set instructions or the autonomous navigation system. For example, the user can select a specific location in a room through a map interface, and the device will move to that location and scan it. The size and shape of the first target area can be adjusted according to the task requirements. For complex environments, the device may need to scan multiple times and obtain image information in stages to cover the entire target area. The camera or other visual sensors of the cleaning device will scan the specified target area and obtain the perception image information of the area. These image information will be used to analyze the environmental conditions of the target area, including the distribution and type of objects. The quality and accuracy of the perception image information are crucial for the subsequent steps, determining whether the objects can be successfully identified and grabbed.

[0039] The cleaning device control method further includes S120, determining whether there is an object to be grabbed based on the perception image information.

[0040] By way of example, the obtained perception image information is processed to analyze whether there is an object to be grabbed. By comparing the perception image information with a pre-set object feature library, it is determined whether there is an object that meets the grabbing conditions. If a to-be-grabbed object is detected, the system will proceed to the next step, otherwise the target area will be rescaned or adjusted.

[0041] The cleaning device control method further includes S130, obtaining to-be-analyzed image information of the to-be-grabbed object in the case that the to-be-grabbed object exists.

[0042] Exemplarily, once it is confirmed that there is an object to be grabbed in the target area, the system will further acquire image information of the object to be analyzed. This usually includes acquiring detailed images of the object from multiple angles to ensure that the system can fully understand the shape, size, material, and other characteristics of the object. These detailed image information will be used for further analysis and development of grabbing strategies.

[0043] The cleaning equipment control method further includes S140, determining grabbing parameter information of the object to be grabbed based on the image information to be analyzed.

[0044] Exemplarily, according to the acquired image information to be analyzed, the grabbing parameter information of the object is calculated. These parameters include the center of gravity of the object, the position of the grabbing point, the applicable grabbing force, and the weight information of the grabbed object. These parameters are crucial to ensure the success of the grabbing operation and must be accurately calculated according to the actual characteristics of the object.

[0045] The cleaning equipment control method further includes S150, controlling the grabbing mechanical arm to perform a grabbing operation on the object to be grabbed based on the grabbing parameter information.

[0046] Exemplarily, according to the grabbing parameter information calculated in the previous step, the grabbing mechanical arm controls the grabbing operation on the object. The grabbing mechanical arm will accurately position to the grabbing point of the object and grab according to the set force and angle. After grabbing, the grabbing mechanical arm can place the object to the user-specified location, completing the entire operation process.

[0047] In summary, the present disclosure can effectively identify various objects on the ground (such as shoes, clothes, paper balls, etc.) by obtaining the perception image information of the target area, using image processing technology and object recognition algorithms. After confirming the presence of the object, the system further obtains detailed image information of the object and calculates accurate grasping parameters based on this information, thereby controlling the multi-degree-of-freedom grasping manipulator to complete the grasping operation. This enables the cleaning device not only to clean the ground but also to arrange the ground objects, thereby improving the cleanliness of the home environment. The present disclosure supports users to operate the cleaning device through a remote control device (such as a mobile phone APP). Users can remotely control the grasping manipulator to perform the grasping task through the APP and instruct the device to place the objects at a specified location according to the needs. In addition, the system also has an automatic function, and users can instruct the device to automatically perform the grasping and placing operations of the objects, thereby realizing more convenient and intelligent home management. Compared with the existing simple obstacle avoidance function, the present disclosure enables the cleaning device to accurately identify and grasp objects with high precision through accurate image analysis and grasping parameter calculation, thereby avoiding the situation of grasping failure or damaging the objects. The system can develop appropriate grasping strategies according to the specific shape, size, and material properties of the objects to ensure the smooth progress of the grasping process. The present disclosure greatly improves the user experience in household cleaning and object arrangement. Users do not need to manually handle ground litter, and the device can automatically complete these operations, thereby reducing the user's labor burden and further improving the cleanliness and aesthetics of the home environment.

[0048] In some examples, the determining the grasping parameter information of the object to be grasped based on the image information to be analyzed comprises:

[0049] In the case where the image information to be analyzed includes a plurality of object images, the object attribute information corresponding to each object image is obtained; and

[0050] The grasping sequence of the plurality of objects to be grasped is determined according to the object attribute information.

[0051] For example, in some cases, the image information to be analyzed obtained by the device may contain images of multiple objects. That is, there are multiple possible grasping objects in the field of view of the cleaning device at the same time. At this time, the cleaning device needs to further process these images in order to select a suitable object for grasping.

[0052] For each object image in the image information to be analyzed, the device extracts attribute information related to the object. The object attribute information can include but is not limited to the following:

[0053] Object type: for example, object garbage or other litter;

[0054] Object size: judging the size of the object to determine whether it is suitable for the grasping manipulator of the device to grasp;

[0055] Object position: determining the distance and angle of the object relative to the grabbing robot to assess the difficulty of grabbing;

[0056] Object weight: for some cases, the device can estimate the weight of the object through image information or other sensors; and

[0057] Object surface features: analyzing the smoothness, roughness, etc. of the object surface to assess the stability of grabbing.

[0058] After obtaining the object attribute information of each object, the device will determine the most suitable object to grab according to these attribute information. This process can include but is not limited to the following judgment logic.

[0059] Prioritize objects that are easy to grab. For example, objects that are closer to the grabbing robot, have a surface that is easy to grab, and have a weight within the carrying range of the grabbing robot will be given priority.

[0060] When multiple objects meet the basic grabbing conditions, the device may determine the final object to be grabbed according to pre-set priority rules. Priority rules may be based on the importance of the object, task requirements, or other specific grabbing strategies.

[0061] Based on the method of some embodiments of the present disclosure, the cleaning device can intelligently analyze and select the object to be grabbed, ensuring the success rate and efficiency of the grabbing operation.

[0062] In some examples, the object attribute information includes object type information and object distance information,

[0063] The determination of the object to be grabbed according to the object attribute information includes:

[0064] determining a cleaning obstruction impact coefficient based on the object type information;

[0065] determining a grabbing difficulty coefficient according to the object distance information; and

[0066] determining the object to be grabbed based on the cleaning obstruction impact coefficient and the grabbing difficulty coefficient.

[0067] For example, after the scenario device identifies multiple grabbable objects, it will first extract the type information of each object (e.g. garbage or other debris) and the distance information of the object, i.e. the distance of the object from the device or the grabbing robot.

[0068] The device calculates an obstacle cleaning impact coefficient for each object based on the type information of the object. The coefficient reflects the degree of impact of the object on the cleaning effect. In general, the higher the impact coefficient, the greater the obstacle of the object to the cleaning work, and the higher the necessity of priority grabbing.

[0069] For example, the cleaning device identifies several different objects on the ground, such as paper, blocks, beverage bottles, and books. Based on the type information of the objects, the system calculates the obstacle cleaning impact coefficient of each object.

[0070] The paper has a low obstacle cleaning impact coefficient due to its thinness and ease of handling by the cleaning brush or suction function. Such objects have less impact on the cleaning work, and the necessity of priority grabbing is not high. The device may choose to clean directly instead of grabbing. The blocks are hard objects that can block the brush head or suction port of the cleaning device, so the obstacle cleaning impact coefficient is high. In order to avoid the device being blocked during operation, blocks will be given priority for grabbing to ensure smooth cleaning process. The beverage bottle is a medium-sized object and can affect the movement path of the cleaning device. The obstacle cleaning impact coefficient of such objects is high, and the cleaning device will give priority to removing it, especially when it is in a narrow space or passage.

[0071] The device then calculates a grabbing difficulty coefficient based on the distance information of the object. The coefficient reflects the difficulty of the device in actually operating to grab the object. It can be understood that objects closer to the grabbing manipulator will be given a lower grabbing difficulty coefficient because they are easier to grab. Objects farther away will be given a higher grabbing difficulty coefficient because grabbing them can require more complex operations of the grabbing manipulator.

[0072] Finally, the device considers the obstacle cleaning impact coefficient and the grabbing difficulty coefficient of each object to determine the optimal object to be grabbed. In general, if the obstacle cleaning impact coefficient of an object is high and the grabbing difficulty coefficient is relatively low, the device will give priority to grabbing the object. If there are multiple objects with similar comprehensive coefficients, the device can give priority to the object that is easier to grab.

[0073] Based on the method of some embodiments of the present disclosure, the cleaning device can intelligently select the optimal grabbing target among multiple objects, ensuring the efficiency and accuracy of the cleaning work. This selection mechanism ensures that the device not only effectively cleans the objects that have the greatest impact on the environment, but also considers the difficulty of operation of the grabbing manipulator in the grabbing operation, optimizing the cleaning process.

[0074] In some examples, the grabbing parameter information includes grabbing point information and grabbing weight information;

[0075] The determining of the grabbing parameter information of the object to be grabbed based on the image information to be analyzed comprises:

[0076] The determining of the type information and the volume information of the object to be grabbed based on the image information to be analyzed comprises:

[0077] The determining of the grabbing point information based on the type information comprises:

[0078] The determining of the grabbing weight information based on the type information and the volume information comprises:

[0079] For example, the cleaning device first extracts the type information and the volume information of the object to be grabbed by analyzing the image information of the object. The type information refers to the classification of the object, such as shape, material, or purpose, etc. The volume information is the three-dimensional size data of the object.

[0080] Then, the cleaning device determines the optimal grabbing point information according to the type information of the object. The grabbing point information refers to the position where the grabbing arm should grab the object to ensure the stability and success rate of grabbing. For example:

[0081] If the object to be grabbed is a circular object, the cleaning device may select the top of the object as the grabbing point; and

[0082] If the object to be grabbed is a square object, the cleaning device may select a corner or an edge as the grabbing point.

[0083] After determining the grabbing point, the cleaning device further estimates the grabbing weight information based on the type information and the volume information of the object. The grabbing weight information refers to the estimated weight of the object, which is crucial for the operation of the grabbing arm, because the grabbing arm needs to adjust its grabbing force according to the weight of the object. For example:

[0084] For large objects, the device may infer the approximate weight of the object according to its volume information combined with the type information (such as the density of the material); and

[0085] For small objects, the volume information may directly reflect its weight, and the type information is used to fine-tune the weight estimate.

[0086] Based on the method of some embodiments of the present disclosure, the cleaning device can accurately determine the grabbing parameter information, including the optimal grabbing point and the appropriate grabbing weight. Therefore, the cleaning device can be efficient and safe when performing the grabbing task, reducing the possibility of grabbing failure and ensuring the smooth progress of cleaning work.

[0087] In some examples, the above cleaning device control method further comprises:

[0088] In a case where the weight information of the object to be grabbed is greater than the grabbing weight limit information of the grabbing manipulator and / or the number of grabbing operation failures is greater than a preset number, grabbing failure feedback information is sent to a control device associated with the cleaning device.

[0089] Exemplarily, after the cleaning device analyzes the type information and volume information of the object to be grabbed, the weight information of the object is calculated or estimated. The cleaning device compares the weight information of the object to be grabbed with the grabbing weight limit information of the grabbing manipulator. The grabbing weight limit information refers to the maximum weight that can be grabbed by the manipulator within a safe operation range. If the weight information of the object to be grabbed is greater than this limit value, there is a risk of grabbing failure.

[0090] During the grabbing operation, the device also monitors the success or failure of grabbing in real time. If the grabbing operation fails, the device records the number of failures once. When the number of failures accumulates to a preset number, the device determines that the difficulty of the grabbing task exceeds the capability range of the grabbing manipulator.

[0091] When the device detects that the weight information of the object to be grabbed exceeds the grabbing weight limit information of the grabbing manipulator, or the number of grabbing operation failures is greater than a preset number, the device sends grabbing failure feedback information to a control device (such as a user's mobile phone APP) associated with the cleaning device. This feedback information usually includes the reason for failure (such as weight overrun or multiple failures) so that the user can timely understand the status of the grabbing operation and make corresponding adjustments or interventions.

[0092] In some examples, the perception image information is obtained based on a perception module of the cleaning device, and the image information to be analyzed is obtained based on an image acquisition unit at the end of the grabbing manipulator.

[0093] Exemplarily, the cleaning device is equipped with a perception module, which usually includes an RGB camera, a depth camera, or other types of sensor modules. This perception module is used to obtain image information of the environment around the cleaning device, which is referred to as perception image information. This information is used to preliminarily identify whether there is an object to be grabbed in the working area of the device, and the approximate position and distribution of the objects.

[0094] After the cleaning device confirms the presence of an object to be grabbed, the device further mobilizes the grabbing manipulator to perform more detailed operations. The end of the grabbing manipulator is provided with an image acquisition unit, which is usually a high-resolution camera or other types of visual sensors. When the grabbing manipulator approaches the object to be grabbed, the image acquisition unit obtains more detailed image information, which is referred to as image information to be analyzed.

[0095] The perception image information is acquired by the perception module of the device in a wide range, which is used for macroscopic judgment of the situation in the working area. These information helps the device determine whether it needs to perform a grabbing operation and the general direction of the grabbing operation.

[0096] The image information to be analyzed is acquired by the image acquisition unit at the end of the grabbing manipulator at the microscopic level, which is used to accurately identify the type, volume, grabbing point, and other detailed information of the object to be grabbed. These information directly affect the accuracy and success rate of the grabbing operation.

[0097] During operation, the cleaning device first uses the perception module to acquire perception image information to determine whether there is a need for grabbing. If grabbing is needed, the device will mobilize the grabbing manipulator and further analyze and process the image information to be analyzed acquired by the image acquisition unit at the end of the grabbing manipulator to determine the final grabbing parameters and perform the grabbing operation.

[0098] Through this hierarchical information acquisition and processing method, the cleaning device can effectively identify and grab objects in a wide range of environments, ensuring the accuracy and efficiency of the operation.

[0099] In some examples, before the step of acquiring the perception image information of the first target area, the above cleaning device control method further comprises:

[0100] In response to the movement remote control instruction sent by the control device associated with the cleaning device, the cleaning device is controlled to reach the first target area.

[0101] For example, the cleaning device is associated with a control device (such as a mobile phone APP). The user sends a movement remote control instruction through the control device to instruct the cleaning device to go to a specified first target area. This instruction can be sent in the form of manual input, voice instruction, or pre-set path selection.

[0102] After receiving the movement remote control instruction, the cleaning device will plan a path according to the instruction and move to the specified first target area. The navigation system (such as the system based on SLAM algorithm) inside the device will ensure that the device can accurately reach the target position. During the movement, the device will use its own sensors to avoid obstacles to ensure safe arrival at the destination.

[0103] Once the cleaning device reaches the first target area, it will start the perception module to acquire the perception image information of the area. These information is used to determine whether there are objects that can be grabbed in the target area and to provide basic data for subsequent grabbing operations.

[0104] This process ensures that the cleaning device can perform cleaning and grabbing operations at user-specified locations. Through user remote control instructions, the device can flexibly move between different locations, increasing the convenience and applicability of device operation.

[0105] It can be understood that in the above process of remotely instructing the robot to go to the specified target point and grab, the user can remotely view the above process in the APP through the camera carried by the main body or the camera carried by the grabbing mechanical arm.

[0106] In some examples, after the grabbing mechanical arm completes the grabbing operation on the object to be grabbed, the above cleaning device control method further comprises:

[0107] In response to the grabbing completion instruction or the drop remote control instruction sent by the control device associated with the cleaning device, the cleaning device is controlled to reach a second target area; and

[0108] In the case where the cleaning device reaches the drop waiting area corresponding to the second target area, the grabbing mechanical arm is controlled to perform a drop operation on the object to be dropped to place the object to be dropped in the second target drop area.

[0109] For example, the user can select the area where the grabbed object needs to be placed before selecting the grabbed object. After the mechanical arm of the cleaning device completes the grabbing operation, a grabbing completion instruction is automatically generated to control the cleaning device to reach the second target area.

[0110] The cleaning device can also wait for further instructions after the grabbing operation is completed. If the user sends a drop remote control instruction through the control device (such as a mobile phone APP), the device will receive this instruction and prepare for subsequent drop operation.

[0111] After receiving the drop remote control instruction, the cleaning device will move to the second target area according to the instruction. The second target area is the object drop position specified by the user. The navigation system inside the device will help the device safely and accurately reach the area.

[0112] When the cleaning device approaches the second target area, it will first confirm whether it has entered the drop waiting area. The drop waiting area is a specified area near the target drop area, where the device needs to wait and prepare to perform the drop operation.

[0113] Once the cleaning device confirms that it has reached the drop waiting area, the device will control the grabbing mechanical arm to perform a drop operation on the object to be dropped. The drop operation includes releasing the grabbed object from the grabbing mechanical arm and accurately placing it in the target drop area.

[0114] According to the method provided by the embodiment of the present disclosure, it is ensured that the cleaning device can not only grasp the object, but also move and drop the object to the specified position under the instruction of the user, and complete the entire operation process.

[0115] In some examples, the cleaning device control method further includes:

[0116] In the case that the second target area is the area that the cleaning device can reach, the drop waiting area corresponding to the second target area is within the area range of the second target area; or

[0117] In the case that the second target area is the area that the cleaning device cannot reach, the drop waiting area corresponding to the second target area is within the area range adjacent to the second target area.

[0118] For example, the user can directly control the cleaning device through a remote control device (such as a mobile phone APP), or can select a specific target point in the map interface provided by the device. The target point is the position where the user wants the device to drop the grasped object.

[0119] If the target point is an area that the device can reach, the drop waiting area will be set within the area range of the target point, that is, the device can smoothly enter the area and perform the drop operation.

[0120] If the target point is an area that the device cannot reach, the drop waiting area will be set within the area range adjacent to the target point. The device will stay in this drop waiting area and perform the drop operation to place the object as close to the target point as possible.

[0121] After determining the drop waiting area, the device will control the grasping manipulator to perform the drop operation to safely and accurately place the object in the specified area.

[0122] This operation process ensures that the user can flexibly control the device to move and drop the grasped object to the specified position, even if the device cannot completely reach the target point, the object can be placed in a suitable position.

[0123] In some examples, the target drop area is determined based on one or more of a cleaning device movement limitation condition, a grasping manipulator deployment limitation condition, a special area limitation condition, a user setting limitation condition, an area material limitation condition, and an object stacking limitation condition.

[0124] For example, the cleaning device movement limitation condition ensures that the cleaning device is not limited during movement and operation. For example, the target drop area should not be located in a low area or an area with dense obstacles that the device cannot pass through, so as to avoid collision or jamming of the device during movement.

[0125] The deployment restriction condition of the grabbing manipulator ensures that the grabbing manipulator can be fully deployed when performing the placing operation. The target drop area should not be located in an area where the grabbing manipulator cannot be normally deployed, such as near a wall or other obstacles, so that the grabbing manipulator can be avoided in the placing operation or cannot operate normally.

[0126] The special area restriction condition ensures that the target drop area does not have a negative impact on the safety or operating efficiency of the device. For example, the target drop area should not be a dangerous area such as a cliff to avoid objects falling or accidents occurring to the device; nor should it be a threshold or an extended area before and after the threshold, which may affect the passage of the device.

[0127] The user-set restriction condition can set certain forbidden areas through the APP, and the device will avoid these areas when selecting the target drop area. For example, the user may set some areas as cleaning forbidden areas, and the device will automatically avoid these areas when placing objects.

[0128] The area material restriction condition selects the target drop area based on the material properties of the area. The target drop area should not be a material area such as a carpet, which may affect the stable placement of objects or cause objects to fall after being placed.

[0129] The object stacking restriction condition is affected by the type of object being grabbed, and the selection of the target drop area is also affected by the object stacking condition. For example:

[0130] For shoe-type objects: the target drop area should be selected to not allow shoes to be stacked, to avoid deformation or damage; and

[0131] For fabric or bulk-type objects: they can be stacked on similar objects, so the target drop area can be selected to meet this stacking condition.

[0132] By considering one or more of the above conditions, the cleaning device can intelligently select the optimal target drop area to ensure that objects are placed in a safe and appropriate location. This design not only improves the operating efficiency of the device, but also increases the adaptability of the device in complex environments, and can meet the diverse needs of users.

[0133] In some examples, real-time image information is obtained based on the camera unit corresponding to the cleaning device.

[0134] The above real-time image information is sent to the control terminal of the above cleaning device, so that the above target user remotely monitors the above cleaning device through the above control terminal.

[0135] Exemplarily, a camera unit is provided on the device body of the cleaning device and / or on the front end of the grabbing mechanical arm, responsible for collecting real-time image information of the surrounding environment during the operation of the device, so as to capture the cleaning state and environmental conditions in the working area of the device. The camera unit can provide visual feedback of the current working environment of the device for the user. Such image information is updated in real time, ensuring that the operating state of the cleaning device in different environments can be monitored.

[0136] The real-time image information collected by the camera unit of the cleaning device is transmitted to the control terminal of the cleaning device through a transmission module inside the device. The control terminal can be a remote device such as a smartphone, tablet computer or computer, which receives these image data through a network connection. Wireless networks (such as Wi-Fi, 5G, etc.) or wired networks can be used for image data transmission.

[0137] The user can always know the working state of the cleaning device by receiving the real-time image information transmitted by the cleaning device through the control terminal. The user can not only view the running area of the device, but also monitor the cleaning effect of the device to determine whether the device is working correctly or whether intervention is needed. In addition to simple monitoring, the user can also send operation instructions to the cleaning device through the control terminal. For example, when the device encounters an obstacle or the cleaning effect is not ideal, the user can remotely control the device to change the running path or adjust the cleaning mode.

[0138] Please refer to FIG. 2, which is a structural schematic diagram of a cleaning device control apparatus 200 according to an embodiment of the present disclosure, which can include:

[0139] The first acquisition unit 201 is configured to acquire perception image information of a first target area.

[0140] The first determination unit 202 is configured to determine whether there is a to-be-grabbed object based on the perception image information.

[0141] The second acquisition unit 203 is configured to acquire to-be-analyzed image information of the to-be-grabbed object in the case where there is the to-be-grabbed object.

[0142] The second determination unit 204 is configured to determine grabbing parameter information of the to-be-grabbed object based on the to-be-analyzed image information.

[0143] The control unit 205 is configured to control the grabbing mechanical arm to perform a grabbing operation on the to-be-grabbed object based on the grabbing parameter information.

[0144] As shown in FIG. 3, the electronic device 300 according to the embodiments of the present disclosure includes a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. The processor 320 implements the steps of any of the cleaning device control methods described above when executing the computer program 311.

[0145] Since the electronic device described in the embodiments is a device used to implement the cleaning device according to the embodiments of the present disclosure, based on the methods described in the embodiments of the present disclosure, those skilled in the art can understand the specific implementation of the electronic device and its various forms of variation. Therefore, how the electronic device implements the methods according to the embodiments of the present disclosure will not be described in detail here, as long as the device used by those skilled in the art to implement the methods according to the embodiments of the present disclosure falls within the scope of the present disclosure.

[0146] In the implementation process, the computer program 311 can implement any of the embodiments of the first aspect when executed by the processor.

[0147] As shown in FIG. 4, the cleaning device 400 according to the embodiments of the present disclosure includes the electronic device 300 shown in FIG. 3.

[0148] It should be noted that in the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0149] In the first aspect, the embodiments of the present disclosure provide a cleaning device control method, the cleaning device includes a grabbing mechanical arm, and the cleaning device control method includes:

[0150] obtaining perception image information of a first target area;

[0151] determining whether there is a to-be-grabbed object based on the perception image information;

[0152] in a case where the to-be-grabbed object exists, obtaining to-be-analyzed image information of the to-be-grabbed object;

[0153] determining grabbing parameter information of the to-be-grabbed object based on the to-be-analyzed image information; and

[0154] controlling the grabbing mechanical arm to perform a grabbing operation on the to-be-grabbed object based on the grabbing parameter information.

[0155] In some embodiments, the cleaning device control method further includes:

[0156] in a case where the to-be-analyzed image information includes a plurality of object images, obtaining object attribute information corresponding to each object image; and

[0157] determine a grabbing sequence of the plurality of objects to be grabbed according to the object attribute information.

[0158] In some embodiments, the object attribute information comprises object type information and object distance information,

[0159] The determining of the object to be grabbed according to the object attribute information comprises:

[0160] determining a cleaning-obstructing influence coefficient based on the object type information;

[0161] determining a grabbing-difficulty coefficient according to the object distance information; and

[0162] determining the object to be grabbed based on the cleaning-obstructing influence coefficient and the grabbing-difficulty coefficient.

[0163] In some embodiments, the grabbing parameter information comprises grabbing point information and grabbing weight information;

[0164] The determining of the grabbing parameter information of the object to be grabbed based on the image information to be analyzed comprises:

[0165] determining type information and volume information of the object to be grabbed based on the image information to be analyzed;

[0166] determining the grabbing point information based on the type information; and

[0167] determining the grabbing weight information based on the type information and the volume information.

[0168] In some embodiments, the above cleaning device control method further comprises:

[0169] In a case where the weight information of the object to be grabbed is greater than grabbing weight limit information of the grabbing manipulator and / or a number of grabbing operation failures is greater than a preset number, sending grabbing failure feedback information to a control device associated with the cleaning device.

[0170] In some embodiments, the perception image information is acquired based on a perception module of the cleaning device, and the image information to be analyzed is acquired based on an image acquisition unit at an end of the grabbing manipulator.

[0171] In some embodiments, before the step of acquiring the perception image information of the first target area, the above cleaning device control method further comprises:

[0172] In response to a moving remote control instruction sent by a control device associated with the cleaning device, controlling the cleaning device to reach the first target area.

[0173] In some embodiments, after the grabbing manipulator completes the grabbing operation on the object to be grabbed, the above-mentioned cleaning equipment control method further comprises:

[0174] In response to the grabbing completion instruction or the drop remote control instruction sent by the control device associated with the cleaning equipment, the cleaning equipment is controlled to reach a second target area; and

[0175] In the case that the cleaning equipment reaches a drop waiting area corresponding to the second target area, the grabbing manipulator is controlled to perform a drop operation on the object to be dropped to place the object to be dropped in the second target drop area.

[0176] In some embodiments, the above-mentioned cleaning equipment control method further comprises:

[0177] In the case that the second target area is an area that the cleaning equipment can reach, the drop waiting area corresponding to the second target area is within the area range of the second target area; or

[0178] In the case that the second target area is an area that the cleaning equipment cannot reach, the drop waiting area corresponding to the second target area is within the area range adjacent to the second target area.

[0179] In some embodiments, the target drop area is determined based on one or more of a cleaning equipment movement restriction condition, a grabbing manipulator deployment restriction condition, a special area restriction condition, a user-set restriction condition, an area material restriction condition, and an object stacking restriction condition.

[0180] In some embodiments, real-time image information is obtained by a camera unit corresponding to the cleaning equipment.

[0181] The above-mentioned real-time image information is sent to a control terminal of the cleaning equipment, so that the target user remotely monitors the cleaning equipment through the control terminal.

[0182] In a second aspect, the embodiments of the present disclosure further provide a cleaning equipment control device, comprising:

[0183] A first acquisition unit is configured to acquire perception image information of a first target area.

[0184] A first determination unit is configured to determine whether there is an object to be grabbed based on the perception image information.

[0185] A second acquisition unit is configured to acquire analysis image information of the object to be grabbed in the case that there is the object to be grabbed.

[0186] A second determination unit is configured to determine grabbing parameter information of the object to be grabbed based on the analysis image information.

[0187] a control unit configured to control the picking manipulator to perform a picking operation on the object to be picked based on the picking parameter information.

[0188] In a third aspect, the embodiments of the present disclosure also provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor is configured to implement the steps of the cleaning device control method of any one of the first aspect when executing the computer program stored in the memory.

[0189] In a fourth aspect, the embodiments of the present disclosure also provide a computer readable storage medium, comprising a computer program stored thereon, wherein the computer program, when executed by a processor, implements the cleaning device control method of any one of the first aspect.

[0190] In a fifth aspect, the embodiments of the present disclosure also provide a cleaning device, comprising the electronic device of the third aspect.

[0191] In summary, the cleaning device control method of the embodiments of the present disclosure includes: obtaining perception image information of a first target area; determining whether there is a to-be-grabbed object based on the perception image information; in the case that there is the to-be-grabbed object, obtaining to-be-analyzed image information of the to-be-grabbed object; determining grabbing parameter information of the to-be-grabbed object based on the to-be-analyzed image information; and controlling a grabbing manipulator to perform a grabbing operation on the to-be-grabbed object based on the grabbing parameter information. The present disclosure can effectively identify various objects (such as shoes, clothes, paper balls, etc.) on the ground by obtaining the perception image information of the target area, using image processing technology and object recognition algorithms. After confirming the existence of the object, the system further obtains detailed image information of the object, and calculates accurate grabbing parameters based on the information, so as to control the multi-degree-of-freedom grabbing manipulator to complete the grabbing operation. This makes the cleaning device not only capable of sweeping the ground, but also capable of arranging the ground objects, thereby improving the cleanliness of the home environment. The present disclosure supports the user to operate the cleaning device through a remote control device (such as a mobile phone APP), and the user can remotely control the grabbing manipulator to perform a grabbing task through the APP, and instruct the device to place the object at a specified position according to the demand. In addition, the system also has an automatic function, and the user can instruct the device to automatically perform the grabbing and placing operations of the object, thereby realizing more convenient and intelligent home management. Compared with the existing simple obstacle avoidance function, the present disclosure can complete the identification and grabbing of the object with high precision through accurate image analysis and grabbing parameter calculation, so as to avoid the situation of grabbing failure or damaging the object. The system can formulate a corresponding grabbing strategy according to the specific shape, size and material of the object, and ensure the smooth progress of the grabbing process. The present disclosure greatly improves the user experience in home cleaning and object arrangement. The user does not need to manually process the ground debris, and the device can automatically complete these operations, thereby reducing the labor burden of the user and further improving the cleanliness and beauty of the home environment.

[0192] The cleaning device control method provided by the present disclosure, other advantages, objects and features of the present disclosure will be embodied by the above description, and will be understood by those skilled in the art through research and practice of the present disclosure.

[0193] Those skilled in the art will appreciate that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-readable program code.

[0194] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0195] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks.

[0196] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0197] The embodiments of the present disclosure further provide a computer program product, which comprises computer software instructions, and when the computer software instructions run on a processing device, the processing device executes the flow of the intelligent drinking water service method.

[0198] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present disclosure are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that the computer can store or be integrated into a data storage device such as a server, data center, etc. containing one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0199] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0200] In several embodiments provided by the present disclosure, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic. For example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0201] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0202] In addition, each functional unit in the various embodiments of the present disclosure can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0203] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present disclosure, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of the present disclosure. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

[0204] In the present disclosure, the terms "first", "second", "third" are only used for descriptive purposes, and cannot be understood or implied to indicate or imply relative importance. The term "multiple" refers to two or more, unless otherwise explicitly limited. The terms "mount", "connect", "connect", "fix" and the like should be broadly understood. For example, "connection" can be fixed connection, or detachable connection, or integral connection. "Connection" can be direct connection, or indirect connection through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in the present disclosure can be understood according to the specific circumstances.

[0205] In the description of the present disclosure, the description of the terms "one embodiment", "some embodiments", "a specific embodiment" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the present disclosure, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0206] The above is only the preferred embodiment of the present disclosure, and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A cleaning device control method, the cleaning device comprising a grabbing manipulator, the method comprising: obtaining perception image information of a first target area; determining whether there is an object to be grabbed based on the perception image information; in a case where there is the object to be grabbed, obtaining image information to be analyzed of the object to be grabbed; determining grabbing parameter information of the object to be grabbed based on the image information to be analyzed; and controlling the grabbing manipulator to perform a grabbing operation on the object to be grabbed based on the grabbing parameter information.

2. The cleaning device control method of claim 1, further comprising: in a case where the image information to be analyzed comprises a plurality of object images, obtaining object attribute information corresponding to each object image; and determining a grabbing sequence of the plurality of objects to be grabbed according to the object attribute information. The object attribute information comprises object type information and object distance information, 3. The cleaning device control method of claim 2, wherein, The determination of the object to be grabbed according to the object attribute information comprises: determining an obstacle cleaning influence coefficient based on the object type information; determining a grabbing difficulty coefficient according to the object distance information; and determining the object to be grabbed based on the obstacle cleaning influence coefficient and the grabbing difficulty coefficient. The grabbing parameter information comprises grabbing point information and grabbing weight information; 4. The cleaning device control method of claim 1, wherein, The determination of the grabbing parameter information of the object to be grabbed based on the image information to be analyzed comprises: determining type information and volume information of the object to be grabbed based on the image information to be analyzed; determining the grabbing point information based on the type information; and determining the grabbing weight information based on the type information and the volume information.

5. The cleaning device control method of any one of claims 1 to 4, further comprising: in a case where weight information of the object to be grabbed is greater than grabbing weight limit information of the grabbing manipulator and / or a number of grabbing operation failures is greater than a preset number, sending grabbing failure feedback information to a control device associated with the cleaning device. The perception image information is obtained based on a perception module of the cleaning device, and the image information to be analyzed is obtained based on an image acquisition unit at an end of the grabbing manipulator.

6. The cleaning device control method of claim 1, wherein, Before the step of obtaining the perception image information of the first target area, the method further comprises:

7. The cleaning device control method of claim 1, wherein, in response to a movement remote control instruction sent by a control device associated with the cleaning device, controlling the cleaning device to reach the first target area. After the grabbing manipulator completes the grabbing operation on the object to be grabbed, the method further comprises:

8. The cleaning device control method of claim 1, wherein, in response to a grabbing completion instruction or a drop remote control instruction sent by the control device associated with the cleaning device, controlling the cleaning device to reach a second target area; and in a case where the cleaning device reaches a drop waiting area corresponding to the second target area, controlling the grabbing manipulator to perform a drop operation on an object to be dropped to place the object to be dropped in the second target drop area.

9. The cleaning device control method of claim 8, further comprising: ​ In a case where the second target area is a reachable area of the cleaning device, the drop waiting area corresponding to the second target area is within the area range of the second target area. Or In a case where the second target area is an unreachable area of the cleaning device, the drop waiting area corresponding to the second target area is within the area range adjacent to the second target area.

10. The cleaning device control method of claim 8, wherein, The target drop area is determined based on one or more of a cleaning device movement restriction condition, a grabbing mechanical arm deployment restriction condition, a special area restriction condition, a user setting restriction condition, an area material restriction condition, and an object stacking restriction condition.

11. The cleaning device control method of any one of claims 1-10, further comprising: acquiring real-time image information based on a camera unit corresponding to the cleaning device; and sending the real-time image information to a control terminal of the cleaning device, so that the target user remotely monitors the cleaning device through the control terminal.

12. A cleaning device control apparatus, comprising: a first acquisition unit configured to acquire perception image information of a first target area; a first determination unit configured to determine whether there is a to-be-grabbed object based on the perception image information; a second acquisition unit configured to acquire to-be-analyzed image information of the to-be-grabbed object in a case where the to-be-grabbed object exists; a second determination unit configured to determine grabbing parameter information of the to-be-grabbed object based on the to-be-analyzed image information; and a control unit configured to control a grabbing mechanical arm to perform a grabbing operation on the to-be-grabbed object based on the grabbing parameter information. a memory and a processor, the processor being configured to implement the steps of the cleaning device control method of any one of claims 1-11 when executing a computer program stored in the memory.

13. An electronic device comprising: a computer program stored thereon, the computer program being configured to implement the steps of the cleaning device control method of any one of claims 1-11 when executed by a processor.

14. A computer-readable storage medium comprising:

15. A cleaning device, comprising the electronic device of claim 13. ​

Images