Cleaning equipment control method and related equipment

Abstract

The invention discloses a cleaning equipment control method and related equipment, and relates to the field of cleaning equipment control, the cleaning equipment comprises a grabbing mechanical arm, and the method comprises the steps that perception image information of a first target area is acquired; determining whether a to-be-grabbed object exists or not based on the perceived image information; under the condition that the to-be-grabbed object exists, obtaining to-be-analyzed image information of the to-be-grabbed object; determining grabbing parameter information of the to-be-grabbed object based on the to-be-analyzed image information; and based on the grabbing parameter information, a grabbing mechanical arm is controlled to execute grabbing operation on the to-be-grabbed object.

Description

Technical Field

[0001] This specification relates to the technical field of cleaning equipment control. More specifically, the present disclosure relates to a cleaning equipment control method and related devices. Background Art

[0002] In recent years, with the rapid development of smart home and artificial intelligence technologies, intelligent cleaning robots have become increasingly popular auxiliary tools in household life. Such devices can automatically complete the floor cleaning work, helping users reduce the burden of daily cleaning.

[0003] However, the mainstream cleaning robots on the current market mainly focus on floor cleaning tasks. For scattered items on the floor such as shoes, clothes, paper balls, etc., existing devices cannot effectively identify, grab, and organize them. This results in users still having to manually organize these items, failing to achieve comprehensive automated cleaning and organization. In addition, some existing cleaning devices have a simple obstacle avoidance function, but lack the ability to classify and autonomously organize objects, unable to meet users' demands for higher household tidiness. Summary of the Invention

[0004] The present disclosure aims to at least solve one of the technical problems existing in the prior art or related technologies.

[0005] In view of this, in a first aspect, embodiments of the present disclosure propose a cleaning equipment control method. The above cleaning equipment includes a grabbing robotic arm, and the method includes:

[0006] Obtain the perceptual image information of the first target area;

[0007] Determine whether there is an object to be grabbed based on the perceptual image information;

[0008] In the case where there is the object to be grabbed, obtain the image information to be analyzed of the object to be grabbed;

[0009] Determine the grabbing parameter information of the object to be grabbed based on the image information to be analyzed;

[0010] Control the grabbing robotic arm to perform a grabbing operation on the object to be grabbed based on the grabbing parameter information.

[0011] In a feasible implementation, the method further includes:

[0012] In the case where the image information to be analyzed includes multiple object images, obtain the object attribute information corresponding to each object image;

[0013] Determine the grabbing order of the multiple objects to be grabbed according to the object attribute information.

[0014] In a feasible implementation, the object attribute information includes object type information and object distance information.

[0015] Determining the object to be grasped according to the object attribute information includes:

[0016] Determining the cleaning obstruction influence coefficient based on the object type information;

[0017] Determining the grasping difficulty coefficient according to the object distance information;

[0018] Determining the object to be grasped based on the cleaning obstruction influence coefficient and the grasping difficulty coefficient.

[0019] In a feasible implementation, the grasping parameter information includes grasping point position information and grasping weight information;

[0020] Determining the grasping parameter information of the object to be grasped based on the image information to be analyzed includes:

[0021] Determining the type information and volume information of the object to be grasped based on the image information to be analyzed;

[0022] Determining the grasping point position information based on the type information;

[0023] Determining the grasping weight information based on the type information and the volume information.

[0024] In a feasible implementation, it further includes:

[0025] When the weight information of the object to be grasped is greater than the grasping weight limit information of the grasping robotic arm and / or the number of failed grasping operations is greater than the preset number, sending a grasping failure feedback information to the control device associated with the cleaning device.

[0026] In a feasible implementation, the sensed image information is obtained based on the sensing module of the cleaning device, and the image information to be analyzed is obtained based on the image acquisition unit at the end of the grasping robotic arm.

[0027] In a feasible implementation, before the step of obtaining the sensed image information of the first target area, the method further includes:

[0028] Responding to the mobile remote control instruction sent by the control device associated with the cleaning device, and controlling the cleaning device to reach the first target area.

[0029] In a feasible implementation, after the grasping robotic arm completes the grasping operation on the object to be grasped, the method further includes:

[0030] In response to the completion of the grasping instruction or the delivery remote control instruction sent by the control device associated with the above-mentioned cleaning device, control the above-mentioned cleaning device to reach the second target area;

[0031] When the cleaning device reaches the delivery waiting area corresponding to the second target area, control the grasping robotic arm to perform a delivery operation on the object to be delivered, so as to place the object to be delivered in the above-mentioned second target delivery area.

[0032] In a feasible implementation manner, it further includes:

[0033] When the second target area is an area reachable by the cleaning device, the delivery waiting area corresponding to the second target area is within the area range of the second target area; or,

[0034] When the second target area is an area unreachable by the cleaning device, the delivery waiting area corresponding to the second target area is within the adjacent area range outside the second target area.

[0035] In a feasible implementation manner, the target delivery area is determined based on one or more of the cleaning device movement restriction conditions, the grasping robotic arm deployment restriction conditions, the special area restriction conditions, the user setting restriction conditions, the area material restriction conditions, and the object stacking restriction conditions.

[0036] In a feasible implementation manner, obtain real-time image information based on the camera unit corresponding to the above-mentioned cleaning device;

[0037] Send the above-mentioned real-time image information to the control terminal of the above-mentioned cleaning device, so that the target user can remotely monitor the above-mentioned cleaning device through the above-mentioned control terminal.

[0038] In a second aspect, an embodiment of the present disclosure further proposes a cleaning device control device, including:

[0039] A first acquisition unit, configured to acquire the sensed image information of the first target area;

[0040] A first determination unit, configured to determine whether there is an object to be grasped based on the sensed image information;

[0041] A second acquisition unit, configured to acquire the image information to be analyzed of the object to be grasped when there is the object to be grasped;

[0042] A second determination unit, configured to determine the grasping parameter information of the object to be grasped based on the image information to be analyzed;

[0043] A control unit, configured to control the grasping robotic arm to perform a grasping operation on the object to be grasped based on the grasping parameter information.

[0044] In a third aspect, an embodiment of the present disclosure further provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor is configured to implement the steps of the cleaning device control method according to any one of the first aspects when executing the computer program stored in the memory.

[0045] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and the computer program implements the cleaning device control method according to any one of the first aspects when executed by a processor.

[0046] In a fifth aspect, an embodiment of the present disclosure further provides a cleaning device, including the electronic device according to the third aspect of the claims.

[0047] In summary, the cleaning device control method of the embodiments of the present disclosure includes: obtaining the sensed image information of the first target area; determining whether there is an object to be grasped based on the sensed image information; obtaining the image information to be analyzed of the object to be grasped when there is the object to be grasped; determining the grasping parameter information of the object to be grasped based on the image information to be analyzed; and controlling the grasping robotic arm to perform a grasping operation on the object to be grasped based on the grasping parameter information. The present disclosure can effectively identify various objects (such as shoes, clothes, paper balls, etc.) on the ground by obtaining the sensed image information of the target area and using image processing technology and object recognition algorithms. After confirming the existence of the object, the system further obtains the detailed image information of the object and calculates accurate grasping parameters based on this information, so as to control the multi-degree-of-freedom grasping robotic arm to complete the grasping operation. This enables the cleaning device not only to clean the ground but also to tidy up the items on the ground, improving the cleanliness of the home environment. The present disclosure supports users to operate the cleaning device through a remote control device (such as a mobile phone APP). Users can remotely control the grasping robotic arm to perform a grasping task through the APP and instruct the device to place the item at a designated position according to their needs. In addition, the system also has an automatic function. Users can instruct the device to automatically perform the grasping and placing operations of the item, thus realizing more convenient and intelligent home management. Compared with the existing simple obstacle avoidance function, the present disclosure enables the cleaning device to complete the recognition and grasping of objects with high precision through accurate image analysis and grasping parameter calculation, avoiding the situation of grasping failure or damaging items. The system can formulate corresponding grasping strategies according to the specific shape, size, and material characteristics of the object to ensure the smooth progress of the grasping process. The present disclosure greatly improves the user experience in home cleaning and item tidying. Users do not need to manually handle the ground debris, and the device can automatically complete these operations, reducing the user's labor burden and further improving the cleanliness and beauty of the home environment.

[0048] The cleaning device control method proposed by the present disclosure. Other advantages, objectives, and features of the present disclosure will be partially reflected by the following description, and will also be understood by those skilled in the art through research and practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to limit this specification. Also, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0050] Figure 1 It is a schematic flowchart of a cleaning device control method provided by an embodiment of the present disclosure;

[0051] Figure 2 It is a schematic structural diagram of a cleaning device control device proposed by an embodiment of the present disclosure;

[0052] Figure 3 It is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure;

[0053] Figure 4 It is a schematic structural diagram of a cleaning device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present disclosure and the above drawings are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order different from that shown or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices. The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments.

[0055] The cleaning device proposed in this application consists of a cleaning device main body (i.e., the chassis) and a robotic arm module mounted on the chassis. The robotic arm module can have multiple degrees of freedom, including telescopic functions, rotational functions around different joints, and the ability to rotate horizontally around the chassis, enabling it to flexibly handle complex operating environments. The end of the grasping robotic arm is equipped with precise grippers, which can perform actions such as grasping, handling, and placing according to different task requirements, and have precise rotational capabilities to ensure the completion of various complex operating tasks within limited spaces. In addition, the grippers can adaptively adjust according to the shape, size, and weight of the object to ensure stable grasping, and can also protect fragile items by adjusting the force.

[0056] The chassis of the cleaning device integrates a high-performance drive wheel system and cleaning components. The drive wheels support multiple motion modes, including linear motion, turning, acceleration, and deceleration, enabling the device to move flexibly through different cleaning environments. The cleaning components on the chassis are used to efficiently handle ground dust, stains, etc., can adapt to different floor materials, such as tiles, wooden floors, carpets, etc., and can adjust the cleaning mode according to specific cleaning tasks. The chassis also has a strong load-bearing capacity, capable of stably carrying the grasping robotic arm and other devices, ensuring a stable movement during operation.

[0057] In addition to the grasping and cleaning functions, the device has autonomous navigation and environmental perception capabilities. The chassis can be equipped with a variety of advanced sensors: a depth camera is used to obtain three-dimensional space information around the device to help the device identify obstacles and navigation paths; a TOF (Time of Flight) sensor provides precise distance measurement data by measuring the light reflection time; a lidar scans the surrounding environment of the device to generate a real-time map to support the device for autonomous navigation and obstacle avoidance; an RGB camera captures the color images of the environment to assist in identifying surrounding objects and paths. These sensors work together to ensure that the cleaning device can perceive and adapt to environmental changes in real time, avoid collisions and jams, and successfully complete the cleaning task.

[0058] The grippers at the end not only have the ability to perform operations but are also equipped with high-precision RGB cameras for obtaining image information of the object to be grasped. Through real-time image analysis, the grasping robotic arm can identify information such as the shape, color, and material of the object, further adjust the grasping parameters, and improve the accuracy and success rate of grasping. In addition, these cameras can also be used to monitor the grasping process to ensure the safe and efficient operation of the grasping robotic arm. In some complex environments, the image acquisition module of the grasping robotic arm can assist in detailed item classification, placement, etc.

[0059] To ensure the coordinated operation of the device, the chassis and the grasping robotic arm can move independently or work together. After grasping an item, the grasping robotic arm can move with the chassis to a designated location and place the item. In this process, the two cooperate smoothly, enabling the device to perform cleaning and item handling tasks simultaneously. In more complex task scenarios, the control system can dynamically adjust the movement modes of the grasping robotic arm and the chassis according to the task requirements, enabling them to perform tasks separately or coordinate and cooperate according to actual needs to complete more complex operations.

[0060] This cleaning device also supports remote control and automated operation by users. Users can monitor the device throughout the process via a mobile application, setting cleaning paths, grasping tasks, placement areas, etc. The intelligent algorithm built into the device can automatically adjust the action strategy based on the environment and tasks, enabling it to complete tasks efficiently and intelligently in complex environments. At the same time, the device supports real-time status feedback. When encountering obstacles or being unable to complete a task, the device can feedback to the user and wait for further instructions to ensure the safety and reliability of the operation.

[0061] Please refer to Figure 1 , which is a schematic flowchart of a control method for a cleaning device provided by an embodiment of the present disclosure, and specifically may include:

[0062] S110. Obtain the sensed image information of the first target area;

[0063] Exemplarily, the first target area is the area that the sensing system focuses on before the cleaning device executes the cleaning or item grasping task. Within this area, the device needs to analyze the environmental conditions and identify whether there are objects to be cleaned or grasped. The cleaning device can automatically locate the first target area according to the instructions set by the user or the autonomous navigation system. For example, the user can select a specific location in a certain room through the map interface, and the device will move to that location and scan it. The size and shape of the first target area can be adjusted according to the task requirements. For complex environments, the device may need to perform multiple scans and obtain image information in stages to cover the entire target area. The camera or other visual sensors of the cleaning device will scan the designated target area to obtain the sensed image information of this area. These image information will be used to analyze the environmental conditions of the target area, including the distribution and type of objects. The quality and accuracy of the sensed image information are crucial for the subsequent steps and determine whether objects can be successfully identified and grasped.

[0064] S120. Determine whether there are objects to be grasped based on the sensed image information;

[0065] Exemplarily, the acquired perceptual image information is processed to analyze whether there is an object to be grasped. By comparing the perceptual image information with a preset object feature library, it is determined whether there is an object that meets the grasping conditions. If an object to be grasped is detected, the system will enter the next step; otherwise, the target area will be scanned again or adjusted.

[0066] S130. In the case where there is the object to be grasped, acquire the image information to be analyzed of the object to be grasped;

[0067] Exemplarily, once it is confirmed that there is an object to be grasped in the target area, the system will further acquire the image information to be analyzed of the object. This usually includes acquiring detailed images of the object from multiple angles to ensure that the system can comprehensively understand the characteristics of the object such as its shape, size, and material. These detailed image information will be used for further analysis and formulating the grasping strategy.

[0068] S140. Based on the image information to be analyzed, determine the grasping parameter information of the object to be grasped;

[0069] Exemplarily, according to the acquired image information to be analyzed, calculate the grasping parameter information of the object. These parameters include the center of gravity of the object, the position of the grasping point, the applicable grasping force, and the weight information of the grasped object, etc. These parameters are crucial for ensuring the success of the grasping operation and must be accurately calculated according to the actual characteristics of the object.

[0070] S150. Based on the grasping parameter information, control the grasping robotic arm to perform a grasping operation on the object to be grasped.

[0071] Exemplarily, according to the grasping parameter information calculated in the previous step, control the grasping robotic arm to perform a grasping operation on the object. The grasping robotic arm will accurately locate to the grasping point of the object and perform the grasping according to the set force and angle. After grasping, the grasping robotic arm can place the object at the position specified by the user to complete the entire operation process.

[0072] In summary, the present disclosure can effectively identify various objects on the ground (such as shoes, clothes, paper balls, etc.) by obtaining the perceptual image information of the target area and using image processing techniques and object recognition algorithms. After confirming the existence of the object, the system further obtains the detailed image information of the object and calculates accurate grasping parameters based on this information, so as to control the multi-degree-of-freedom grasping manipulator to complete the grasping operation. This enables the cleaning device not only to clean the ground, but also to tidy up the items on the ground, improving the cleanliness of the home environment. The present disclosure supports users to operate the cleaning device through a remote control device (such as a mobile phone APP). Users can use the APP to remotely control the grasping manipulator to perform the grasping task and instruct the device to place the item at a designated position according to their needs. In addition, the system also has an automatic function. Users can instruct the device to automatically perform the grasping and placing operations of the item, thus realizing a more convenient and intelligent home management. Compared with the existing simple obstacle avoidance function, the present disclosure enables the cleaning device to complete the recognition and grasping of objects with high precision through accurate image analysis and grasping parameter calculation, avoiding the situation of grasping failure or damaging the item. The system can formulate corresponding grasping strategies according to the specific shape, size, material and other characteristics of the object to ensure the smooth progress of the grasping process. The present disclosure greatly improves the user experience in home cleaning and item tidying. Users do not need to manually handle the ground debris, and the device can automatically complete these operations, reducing the labor burden of users and further improving the cleanliness and beauty of the home environment.

[0073] In some examples, determining the grasping parameter information of the object to be grasped based on the image information to be analyzed includes:

[0074] When the image information to be analyzed includes multiple object images, obtaining the object attribute information corresponding to each object image;

[0075] Determining the grasping order of the multiple objects to be grasped according to the object attribute information.

[0076] Exemplarily, in some cases, the image information to be analyzed obtained by the device may include images of multiple objects. That is, there are multiple possible grasping objects within the field of view of the cleaning device at the same time. At this time, the cleaning device needs to further process these images in order to select a suitable object for grasping.

[0077] For each object image in the image information to be analyzed, the device extracts the attribute information related to the object. The object attribute information may include but is not limited to the following:

[0078] Object type: For example, whether the object is garbage or other sundries.

[0079] Object size: Judging the size of the object to determine whether it is suitable for the grasping manipulator of the device to grasp.

[0080] Object position: Determine the distance and angle of the object relative to the grasping manipulator to evaluate the difficulty of grasping.

[0081] Object weight: In some cases, the device may estimate the weight of the object through image information or other sensors.

[0082] Object surface characteristics: Analyze the smoothness, roughness, etc. of the object surface to evaluate the stability of grasping.

[0083] After obtaining the object attribute information of each object, the device will determine the most suitable object to be grasped based on this attribute information. This process may include but is not limited to the following judgment logics:

[0084] Preferentially select objects that are easy to grasp. For example, objects that are closer to the grasping manipulator, have a surface that is easy to grasp, and whose weight is within the carrying range of the grasping manipulator will be given priority.

[0085] When multiple objects all meet the basic grasping conditions, the device may determine the final object to be grasped according to the preset priority rules. The priority rules may be based on the importance of the object, task requirements, or other specific grasping strategies.

[0086] Based on the method provided in this embodiment, the cleaning device can intelligently analyze and select the object to be grasped, ensuring the success rate and efficiency of the grasping operation.

[0087] In some examples, the object attribute information includes object type information and object distance information.

[0088] Determining the object to be grasped according to the object attribute information includes:

[0089] Determine the coefficient of influence on cleaning obstruction based on the object type information;

[0090] Determine the grasping difficulty coefficient according to the object distance information;

[0091] Determine the object to be grasped based on the coefficient of influence on cleaning obstruction and the grasping difficulty coefficient.

[0092] Exemplarily, after the cleaning device identifies multiple graspable objects, it will first extract the type information of each object (for example, whether the object is garbage or other sundries) and the object distance information, that is, the distance between the object and the device or the grasping manipulator.

[0093] The device calculates a coefficient of influence on cleaning obstruction according to the type information of each object. This coefficient reflects the degree of influence of the object on the cleaning effect. Generally, the higher the influence coefficient, the greater the obstruction of the object to the cleaning work, and the higher the necessity of preferential grasping.

[0094] For example, the cleaning device identifies several different objects on the ground: such as pieces of paper, building blocks, beverage bottles, and books. Based on the type information of the objects, the system calculates the cleaning obstruction influence coefficient for each object:

[0095] Due to its thin and light nature and easy handling by the cleaning brush or suction function, the piece of paper has a low cleaning obstruction influence coefficient. Such objects have a small impact on the cleaning work, and the necessity of priority grasping is not high. The device may choose to directly sweep rather than grasp. Building blocks are hard objects that may block the brush head or suction inlet of the cleaning device. Therefore, their cleaning obstruction influence coefficient is high. To avoid obstruction during device operation, building block objects will be preferentially grasped to ensure a smooth cleaning process. Beverage bottles are medium-sized objects and may affect the movement path of the cleaning device. The cleaning obstruction influence coefficient of such objects is high, and the cleaning device will give priority to removing them, especially when they are located in a narrow space or passage.

[0096] The device then calculates a grasping difficulty coefficient based on the distance information of the objects. This coefficient reflects the difficulty of the device in grasping the object during actual operation. It can be understood that objects closer to the grasping robotic arm will be given a lower grasping difficulty coefficient because they are easier to grasp, while objects farther away will be given a higher grasping difficulty coefficient because grasping them may require more complex operations by the grasping robotic arm.

[0097] Finally, the device comprehensively considers the cleaning obstruction influence coefficient and the grasping difficulty coefficient of each object to determine the optimal object to be grasped. Usually: If the cleaning obstruction influence coefficient of a certain object is very high and the grasping difficulty coefficient is relatively low, the device will preferentially select this object for grasping. If the comprehensive coefficients of multiple objects are similar, the device may preferentially select the object that is easier to grasp.

[0098] Through the method provided in this embodiment, the cleaning device can intelligently select the optimal grasping target among multiple objects, ensuring the efficiency and accuracy of the cleaning work. This selection mechanism ensures that the device can not only effectively clean the objects that have the greatest impact on the environment but also consider the operation difficulty of the grasping robotic arm during the grasping operation to optimize the cleaning process.

[0099] In some examples, the grasping parameter information includes grasping point position information and grasping weight information;

[0100] Determining the grasping parameter information of the object to be grasped based on the image information to be analyzed includes:

[0101] Determining the type information and volume information of the object to be grasped based on the image information to be analyzed;

[0102] Determine the grasping point information based on the type information;

[0103] Determine the grasping weight information based on the type information and the volume information.

[0104] Exemplarily, the cleaning device first analyzes the image information of the object to be grasped, and extracts the type information and volume information of the object. The type information refers to the classification of the object, such as shape, material, or use, etc.; the volume information is the three-dimensional size data of the object.

[0105] Next, the cleaning device determines the optimal grasping point information according to the type information of the object. The grasping point information refers to the position where the grasping robotic arm should grasp the object to ensure the stability and success rate of the grasping. For example:

[0106] If the object to be grasped is a circular object, the cleaning device may choose the top of the object as the grasping point.

[0107] If the object to be grasped is a square object, the cleaning device may choose the corner or edge as the grasping point.

[0108] After determining the grasping point, the cleaning device further estimates the grasping weight information based on the type information and volume information of the object. The grasping weight information refers to the estimated weight value of the object, which is crucial for the operation of the grasping robotic arm because the grasping robotic arm needs to adjust its grasping force according to the weight of the object. For example:

[0109] For large objects, the device may infer the approximate weight of the object based on its volume information combined with type information (such as the density of the material).

[0110] For small objects, the volume information may directly reflect its weight, and the type information is used to fine-tune the weight estimate.

[0111] Based on the method provided in this embodiment, the cleaning device can accurately determine the grasping parameter information, including the optimal grasping point and the appropriate grasping weight. Thus, the cleaning device can be both efficient and safe when performing the grasping task, reducing the possibility of grasping failure and ensuring the smooth progress of the cleaning work.

[0112] In some examples, it further includes:

[0113] When the weight information of the object to be grasped is greater than the grasping weight limit information of the grasping robotic arm and / or the number of grasping operation failures is greater than the preset number, send a grasping failure feedback message to the control device associated with the cleaning device.

[0114] Exemplarily, after the cleaning device analyzes the type information and volume information of the object to be grasped, it will calculate or estimate the weight information of the object. The cleaning device will compare the weight information of the object to be grasped with the grasping weight limit information of the grasping robotic arm. The grasping weight limit information refers to the maximum weight that the robotic arm can grasp within the safe operating range. If the weight information of the object to be grasped is greater than this limit value, there is a risk of grasping failure.

[0115] During the grasping operation, the device will also monitor the success or failure of the grasping in real time. If the grasping operation fails, the device will record the number of failures once. When the number of failures accumulates to the preset number, the device will determine that the difficulty of the grasping task exceeds the capacity range of the grasping robotic arm.

[0116] When the device detects that the weight information of the object to be grasped exceeds the grasping weight limit information of the grasping robotic arm, or the number of failed grasping operations is greater than the preset number, the device will send a grasping failure feedback message to the control device associated with the cleaning device (such as the user's mobile phone APP). This feedback message usually includes the reason for the failure (such as weight exceeding the limit or multiple failures) so that the user can timely understand the status of the grasping operation and make corresponding adjustments or interventions.

[0117] In some examples, the sensed image information is obtained based on the sensing module of the cleaning device, and the image information to be analyzed is obtained based on the image acquisition unit at the end of the grasping robotic arm.

[0118] Exemplarily, the cleaning device is equipped with a sensing module, which usually includes an RGB camera, a depth camera or other types of sensor modules. This sensing module is used to obtain the image information of the surrounding environment of the cleaning device, which is called the sensed image information. This information is used to preliminarily identify whether there are objects that can be grasped in the working area of the device, as well as the approximate positions and distributions of these objects.

[0119] After the cleaning device confirms the existence of an object that can be grasped, the device will further mobilize the grasping robotic arm to perform more refined operations. An image acquisition unit is equipped at the end of the grasping robotic arm, which is usually a high-resolution camera or other types of visual sensors. When the grasping robotic arm approaches the object to be grasped, this image acquisition unit will obtain more detailed image information, which is called the image information to be analyzed.

[0120] The sensed image information is obtained through the sensing module of the device over a wide range and is used for a macroscopic judgment of the situation in the working area. This information helps the device determine whether a grasping operation is needed and the approximate direction of the grasping operation.

[0121] The image information to be analyzed is obtained at the microscopic level by the image acquisition unit at the end of the grasping robotic arm, and is used to accurately identify details such as the type, volume, and grasping points of the object to be grasped. These information directly affect the accuracy and success rate of the grasping operation.

[0122] During the operation, the cleaning device first uses the sensing module to obtain the sensed image information to determine whether there is a grasping requirement. If grasping is required, the device will activate the grasping robotic arm and perform further analysis and processing based on the image information to be analyzed obtained by the image acquisition unit at the end of the grasping robotic arm to determine the final grasping parameters and execute the grasping operation.

[0123] Through this hierarchical information acquisition and processing method, the cleaning device can effectively identify and grasp objects in a wide range of environments, ensuring the accuracy and efficiency of the operation.

[0124] In some examples, before the step of obtaining the sensed image information of the first target area, the method further includes:

[0125] In response to the mobile remote control instruction sent by the control device associated with the cleaning device, control the cleaning device to reach the first target area.

[0126] Exemplarily, the cleaning device is associated with a control device (such as a mobile phone APP). The user sends a mobile remote control instruction through the control device to instruct the cleaning device to go to the specified first target area. This instruction can be sent in the form of manual input, voice instruction, or preset path selection.

[0127] After receiving the mobile remote control instruction, the cleaning device will plan a path according to the instruction and move to the specified first target area. The navigation system inside the device (such as a system based on the SLAM algorithm) will ensure that the device can accurately reach the target position. During the movement, the device will use its own sensors to avoid obstacles and ensure safe arrival at the destination.

[0128] Once the cleaning device reaches the first target area, it will activate the sensing module to obtain the sensed image information of this area. These information are used to determine whether there are graspable objects in the target area and provide basic data for subsequent grasping operations.

[0129] This process ensures that the cleaning device can perform cleaning and grasping operations at the position specified by the user. Through the user's remote control instruction, the device can move flexibly between different positions, increasing the convenience and applicability of the device operation.

[0130] It can be understood that during the above process of remotely controlling and instructing the robot to move to the designated target point and perform grasping, the user can remotely view the above process in the APP through the camera carried by the main body or the camera carried by the grasping robotic arm.

[0131] In some examples, after the grasping robotic arm completes the grasping operation on the object to be grasped, the method further includes:

[0132] In response to the grasping completion instruction or the placement remote control instruction sent by the control device associated with the above cleaning device, controlling the above cleaning device to reach the second target area;

[0133] When the cleaning device reaches the placement waiting area corresponding to the second target area, controlling the grasping robotic arm to perform a placement operation on the object to be placed, so as to place the object to be placed in the above second target placement area.

[0134] Exemplarily, before selecting to grasp an object, the user can select the area where the object needs to be placed. After the robotic arm of the cleaning device completes the grasping operation, a grasping completion instruction is automatically generated to control the cleaning device to reach the second target area.

[0135] The cleaning device can also wait for further instructions after the grasping operation is completed. If the user sends a placement remote control instruction through a control device (such as a mobile APP), the device will receive this instruction and prepare for the subsequent placement operation.

[0136] After receiving the placement remote control instruction, the cleaning device will move to the second target area according to the instruction. The second target area is the object placement position designated by the user. The navigation system inside the device will help the device reach this area safely and accurately.

[0137] When the cleaning device approaches the second target area, it will first confirm whether it has entered the placement waiting area. The placement waiting area is a designated area close to the target placement area, where the device needs to wait and prepare to perform the placement operation.

[0138] Once the cleaning device confirms that it has reached the placement waiting area, the device will control the grasping robotic arm to perform a placement operation on the object to be placed. The placement operation includes releasing the grasped object from the grasping robotic arm and accurately placing it within the target placement area.

[0139] The method provided by the embodiments of the present application ensures that the cleaning device can not only grasp an object, but also move and place the object to the designated position under the instruction of the user to complete the entire operation process.

[0140] In some examples, it further includes:

[0141] When the second target area is an area accessible by the cleaning device, the corresponding placement waiting area of the second target area is within the area range of the second target area; or,

[0142] When the second target area is an area inaccessible by the cleaning device, the corresponding placement waiting area of the second target area is within the adjacent area range outside the second target area.

[0143] Exemplarily, the user can directly control the cleaning device through a remote control device (such as a mobile phone APP), or select a specific target point in the map interface provided by the device. This target point is the position where the user hopes the device will place the grasped object.

[0144] If the target point is an area accessible by the device, the placement waiting area will be set within the area range of this target point, that is, the device can smoothly enter this area and perform the placement operation.

[0145] If the target point is an area inaccessible by the device, the placement waiting area will be set within the adjacent area range outside the target point. The device will stay in this placement waiting area and perform the placement operation here, placing the object at a position as close as possible to the target point.

[0146] After determining the placement waiting area, the device will control the grasping manipulator to perform the placement operation and place the object safely and accurately within the specified area.

[0147] This operation process ensures that the user can flexibly control the device to move and place the grasped object to the specified position. Even when the device cannot fully reach the target point, it can ensure that the object is placed in a suitable position.

[0148] In some examples, the target placement area is determined based on one or more of the cleaning device movement restriction conditions, grasping manipulator deployment restriction conditions, special area restriction conditions, user setting restriction conditions, area material restriction conditions, and object stacking restriction conditions.

[0149] Exemplarily, the cleaning device movement restriction conditions ensure that the cleaning device is not restricted during movement and operation. For example, the target placement area should not be located in a low area or an area with dense obstacles that are difficult for the device to pass through, so as to avoid the device colliding or getting stuck during movement.

[0150] The grasping manipulator deployment restriction conditions ensure that the grasping manipulator can be fully deployed when performing the placement operation. The target placement area should not be located in an area where the grasping manipulator cannot be normally deployed, such as near a wall or other obstacles, which can avoid the grasping manipulator being restricted or unable to operate normally during the placement operation.

[0151] Special area restriction conditions ensure that the target placement area will not have a negative impact on the safety or operating efficiency of the device. For example, the target placement area should not be a dangerous area such as a cliff to avoid objects falling or the device having an accident; nor should it be a threshold or the extended area before and after the threshold, as these areas may affect the passage of the device.

[0152] User-set restriction conditions can set certain restricted areas through the APP, and the device will avoid these areas when selecting the target placement area. For example, the user may set some areas as cleaning restricted areas, and the device will automatically avoid these areas when placing objects.

[0153] Area material restriction conditions select the target placement area based on the material characteristics of the area. The target placement area should not be a material area such as a carpet, as these areas may affect the stable placement of objects or cause the objects to fall after being placed.

[0154] Object stacking restriction conditions Depending on the type of the grasped object, the selection of the target placement area will also be affected by the object stacking conditions. For example:

[0155] Shoe-like objects: The target placement area should be selected at a position where shoes are not allowed to be stacked to avoid deformation or damage.

[0156] Fabric-like or lump-like objects: They can be stacked on top of similar objects, so the target placement area can be selected at a position that meets this stacking condition.

[0157] By considering one or more of the above conditions, the cleaning device can intelligently select the optimal target placement area to ensure that the object is placed in a safe and appropriate position. This design not only improves the operating efficiency of the device but also increases the adaptability of the device in complex environments and can meet the diverse needs of users.

[0158] In some examples, real-time image information is obtained based on the camera unit corresponding to the above cleaning device;

[0159] The above real-time image information is sent to the control terminal of the above cleaning device so that the above target user can remotely monitor the above cleaning device through the above control terminal.

[0160] Exemplarily, a camera unit is equipped on the device body of the cleaning device and / or at the front end of the grasping robotic arm, which is responsible for collecting real-time image information of the surrounding environment during the operation of the device to capture the cleaning status and environmental conditions within the working area of the device. The camera unit can provide visual feedback on the current working environment of the device for the user. This image information is updated in real time to ensure that the operating status of the cleaning device in different environments can be monitored.

[0161] The real-time image information collected by the camera unit of the cleaning device will be sent to the control terminal of the cleaning device through the transmission module inside the device. The control terminal can be a remote device, such as a smartphone, a tablet computer, or a computer, which receives these image data through a network connection. Wireless networks (such as Wi-Fi, 5G, etc.) or wired networks can be used for the transmission of image data.

[0162] By receiving the real-time image information transmitted by the cleaning device through the control terminal, the user can always understand the working status of the cleaning device. The user can not only view the operating area of the device, but also monitor the cleaning effect of the device to judge whether the device is working properly or whether intervention is needed. In addition to simple monitoring, the user can also send operation instructions to the cleaning device through the control terminal. For example, when the device encounters an obstacle or the cleaning effect is not ideal, the user can remotely control the device to change the running path or adjust the cleaning mode.

[0163] Please refer to Figure 2 , a structural schematic diagram of a cleaning device control device 200 provided by an embodiment of the present disclosure may include:

[0164] A first acquisition unit 201, configured to acquire the sensed image information of a first target area;

[0165] A first determination unit 202, configured to determine whether there is an object to be grasped based on the sensed image information;

[0166] A second acquisition unit 203, configured to acquire the image information to be analyzed of the object to be grasped when there is the object to be grasped;

[0167] A second determination unit 204, configured to determine the grasping parameter information of the object to be grasped based on the image information to be analyzed;

[0168] A control unit 205, configured to control a grasping robotic arm to perform a grasping operation on the object to be grasped based on the grasping parameter information. [[ID=***]]

[0169] As Figure 3 shown, an embodiment of the present disclosure further provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored on the memory 310 and executable on the processor. When the processor 320 executes the computer program 311, it implements the steps of any of the methods for controlling the cleaning device described above.

[0170] Since the electronic device introduced in this embodiment is the device used in a cleaning device in an embodiment of the present application, based on the method introduced in the embodiment of the present application, those skilled in the art can understand the specific implementation manners of the electronic device in this embodiment and its various variations. Therefore, the specific implementation of how this electronic device implements the method in the embodiment of the present application will not be described in detail here. As long as the device used by those skilled in the art to implement the method in the embodiment of the present application belongs to the scope protected by the present application.

[0171] In the specific implementation process, when the computer program 311 is executed by the processor, it can implement any implementation manner in the corresponding embodiment of the first aspect.

[0172] As Figure 4 shown, an embodiment of the present application further provides a cleaning device 400, including the electronic device 300 as Figure 3 shown.

[0173] It should be noted that in the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0174] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-readable program code.

[0175] The present disclosure is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowcharts and / or block diagrams, and the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0176] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one or more of the processes Figure 1 one or more of the processes and / or blocks Figure 1 one or more of the blocks.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the processes Figure 1 one or more of the processes and / or blocks Figure 1 one or more of the blocks.

[0178] Embodiments of the present disclosure also provide a computer program product, which includes computer software instructions. When the computer software instructions run on a processing device, the processing device is caused to execute the processes of the intelligent drinking water service method.

[0179] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present disclosure are produced in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store, or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0180] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0181] In several embodiments provided by the present disclosure, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms.

[0182] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional unit in various embodiments of the present disclosure can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0184] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of the present disclosure. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0185] In the present disclosure, the terms "first", "second", "third" are for descriptive purposes only and should not be construed as indicating or implying relative importance; the term "plurality" means two or more, unless otherwise clearly defined. Terms such as "installed", "connected", "coupled", "fixed" and the like should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; "coupled" can be a direct connection or an indirect connection through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present disclosure can be understood according to specific circumstances.

[0186] In the description of this specification, the descriptions of terms such as "one embodiment", "some embodiments", "specific embodiments" etc. mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or instance. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0187] The above are only the preferred embodiments of the present disclosure and are not intended to limit the present disclosure. For those skilled in the art, various changes and modifications can be made to the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A cleaning equipment control method, characterized in that: The cleaning device includes a gripping robot arm, and the method includes: Acquiring perceived image information of a first target area; Determining whether there is an object to be grasped based on the perceived image information; When the object to be grasped exists, acquiring image information to be analyzed of the object to be grasped; determining grasping parameter information of the object to be grasped based on the image information to be analyzed; The grasping robot arm is controlled to perform a grasping operation on the object to be grasped based on the grasping parameter information.

2. The cleaning equipment control method according to claim 1, characterized in that: The method further comprises: In a case where the image information to be analyzed includes multiple object images, obtaining object attribute information corresponding to each object image; The order of grasping the multiple objects to be grasped is determined according to the object attribute information.

3. The cleaning equipment control method according to claim 2, characterized in that: The object attribute information includes object type information and object distance information, The determining of the object to be grasped according to the object attribute information includes: determining a cleaning obstruction impact coefficient based on the object type information; determining a grasping difficulty coefficient according to the object distance information; The object to be grasped is determined based on the cleaning obstruction influence coefficient and the grasping difficulty coefficient.

4. The cleaning equipment control method according to claim 1, characterized in that: The grabbing parameter information includes grabbing point information and grabbing weight information; The determining the grasping parameter information of the object to be grasped based on the image information to be analyzed includes: determining type information and volume information of the object to be grasped based on the image information to be analyzed; Determining the grasping point information based on the type information; The grasping weight information is determined based on the type information and the volume information.

5. The cleaning equipment control method according to any one of claims 1 to 4, characterized in that: Also includes: When the weight information of the object to be grasped is greater than the grasping weight limit information of the grasping robot arm and / or the number of grasping operation failures is greater than a preset number, grasping failure feedback information is sent to the control device associated with the cleaning device.

6. The cleaning equipment control method according to claim 1, characterized in that: The perceived image information is obtained based on a perception module of the cleaning device, and the image information to be analyzed is obtained based on an image acquisition unit at the end of the grabbing robot arm.

7. A cleaning equipment control device, characterized in that: include: a first acquiring unit, configured to acquire perceived image information of a first target area; a first determining unit, configured to determine whether there is an object to be grasped based on the perceived image information; a second acquiring unit, configured to acquire image information to be analyzed of the object to be grasped when the object to be grasped exists; a second determining unit, configured to determine grasping parameter information of the object to be grasped based on the image information to be analyzed; A control unit is used to control the grasping robot arm to perform a grasping operation on the object to be grasped based on the grasping parameter information.

8. An electronic device comprising: A memory and a processor, wherein the processor is configured to implement the steps of the cleaning equipment control method according to any one of claims 1 to 6 when executing a computer program stored in the memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the cleaning device control method according to any one of claims 1 to 6 are implemented.

10. A cleaning device, characterized in that: Comprising the electronic device as claimed in claim 8.

Images