Sweeping robot control method and device, related equipment and computer program product

Abstract

The invention discloses a sweeping robot control method, a sweeping robot control device, related equipment and a computer program product, surrounding environment information is collected through an environment sensing sensor on a sweeping robot, and accordingly the type and the position of an interactive object existing in the environment can be recognized. Wherein the interactive objects are objects under any category in a category set specified by a user in advance, the categories of all the interactive objects can be counted in advance, the interactive objects are divided into different categories according to a certain mode, and a matched interactive cleaning strategy is configured for each category. Therefore, the cleaning task can be executed by adopting the matched interactive cleaning strategy for different types of interactive objects, and the intelligent level of the sweeping robot is greatly improved. Moreover, the method can support the user to configure matched interactive cleaning strategies according to the characteristics of different types of interactive objects, so that the cleaning effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of smart home control technology, and more specifically, to a sweeping robot control method, device, related equipment and computer program product. Background Art

[0002] A sweeping robot is an intelligent device that has the functions of autonomous movement and cleaning. Current sweeping robots generally perform cleaning tasks based on a single cleaning strategy. When encountering obstacles in the environment (such as tables, chairs, slippers, pets, etc.), they mainly avoid the obstacles, and their intelligence level is relatively low. Summary of the invention

[0003] In view of the above problems, this application is proposed to provide a sweeping robot control method, device, related equipment and computer program product to improve the intelligence level of the sweeping robot. The specific solution is as follows:

[0004] In a first aspect, a method for controlling a sweeping robot is provided, comprising:

[0005] Obtain the surrounding environment information collected by the environment perception sensor on the sweeping robot;

[0006] Based on the surrounding environment information, identify the category and location of an interactive object in the environment, wherein the interactive object is an object in any category in a specified category set, each category in the specified category set is configured with a matching interactive cleaning strategy, and the specified category set includes more than two categories;

[0007] For each of the interactable objects, the location of the interactable object is cleaned according to a pre-configured interaction cleaning strategy that matches the category of the interactable object.

[0008] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the process of identifying the category and location of the interactive object in the environment based on the surrounding environment information includes:

[0009] The configured large model is called to instruct the large model to refer to the surrounding environment information to identify the category and location of the interactive objects in the environment.

[0010] In one possible design, in another implementation of the first aspect of the embodiments of the present application, the designated category set includes at least a category of movable objects, and the category of movable objects is a category of objects configured to be moved by a sweeping robot.

[0011] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the interactive cleaning strategy that can be matched by the moving object category includes:

[0012] When the first set condition is met, the first interactive object corresponding to the movable object category in the environment is moved to the target position by the sweeping robot, and then the position where the first interactive object was located before the movement is cleaned.

[0013] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the first setting condition includes:

[0014] The sweeping robot moves along the cleaning path to a set range around the first interactive object;

[0015] or,

[0016] After identifying the first interactive object in the environment, the sweeping robot plans a path from the current position to the position of the first interactive object, and moves to the first interactive object along the planned path.

[0017] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, the target location is a location in an environment pre-specified by a user;

[0018] or,

[0019] The target position is the position in the area that has been cleaned by the sweeping robot;

[0020] or,

[0021] The target location is a location outside the cleaning area required for this cleaning task.

[0022] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the interactive cleaning strategy that can be matched by the moving object category further includes:

[0023] When a second set condition is met, restoring or arranging the position of the first interactive object at the target position;

[0024] The position restoration is to move the first interactive object to the position before the movement, and the arrangement is to classify and place one or more of the first interactive objects at the target position according to subcategories of the objects, where the subcategories are categories of different objects under the movable object category.

[0025] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the second setting condition includes:

[0026] After the sweeping robot completes the cleaning task;

[0027] or,

[0028] After the position where the first interactive object was located before the movement is cleaned.

[0029] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the specified category set further includes: at least one of the following categories:

[0030] Dangerous goods category, liquid stain category, particulate matter category, living thing category.

[0031] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the interactive cleaning strategy for dangerous goods category matching includes:

[0032] When performing a cleaning task, controlling the distance between the cleaning robot and the second interactive object corresponding to the dangerous object category to be no less than a first set distance threshold;

[0033] The interactive cleaning strategy for matching the liquid stain category includes:

[0034] Cleaning the location of the third interactive object corresponding to the liquid stain category by first mopping and then sweeping;

[0035] The particle class matching interactive cleaning strategy includes:

[0036] The configured particle cleaning mode is used to clean the location of the fourth interactive object corresponding to the particle category, wherein the particle cleaning mode has a greater suction force and a lower side brush rotation speed than the conventional cleaning mode;

[0037] The interactive cleaning strategy for matching the living creature category includes:

[0038] When a fifth interactable object corresponding to the living thing category is detected within a second set distance threshold in front of the cleaning path, a voice and / or light prompt is issued to prompt the fifth interactable object to leave the current position, and cleaning continues along the cleaning path after the fifth interactable object is detected to leave the current position.

[0039] In one possible design, in another implementation of the first aspect of the embodiments of the present application, the surrounding environment information includes a planar image captured by an image sensor and / or a depth image captured by a depth camera.

[0040] In a second aspect, a sweeping robot control device is provided, comprising:

[0041] An environmental information acquisition unit, used to acquire surrounding environment information collected by an environmental perception sensor on the sweeping robot;

[0042] an object recognition unit, configured to recognize the category and position of an interactive object existing in the environment based on the surrounding environment information, wherein the interactive object is an object of any category in a specified category set, each category in the specified category set is configured with a matching interactive cleaning strategy, and the specified category set includes more than two categories;

[0043] The interactive cleaning unit is used to clean the location of each of the interactive objects according to a pre-configured interactive cleaning strategy that matches the category of the interactive object.

[0044] In a third aspect, a sweeping robot is provided, comprising:

[0045] Equipment body;

[0046] A moving part, a cleaning part and an environment sensing sensor are arranged on the device body, and the environment sensing sensor is used to collect surrounding environment information;

[0047] The processor connected to the moving component, the cleaning component and the environmental perception sensor is used to implement the various steps of the cleaning robot control method described in any one of the first aspects of the present application.

[0048] In a fourth aspect, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the sweeping robot control method described in any one of the first aspects of the present application are implemented.

[0049] In a fifth aspect, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements the various steps of the cleaning robot control method described in any one of the first aspects of the present application.

[0050] By means of the above technical solution, the present application collects surrounding environment information through the environmental perception sensor on the sweeping robot, and can identify the category and location of the interactive objects in the environment based on this. Among them, the interactive object is an object under any category in the category set pre-specified by the user. The present application can count the categories of all interactive objects in advance, divide each interactive object into different categories in a certain way, and configure a matching interactive cleaning strategy for each category, so that the cleaning task can be performed for interactive objects of different categories using a matching interactive cleaning strategy, which greatly improves the intelligence level of the sweeping robot. In addition, the present application can support users to configure matching interactive cleaning strategies according to the characteristics of interactive objects of different categories, so as to improve the cleaning effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0052] Figure 1 A schematic diagram of a system architecture for implementing the sweeping robot control method provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a control method for a sweeping robot provided in an embodiment of the present application;

[0054] Figure 3a-3c Several real-life scene images collected by the sweeping robot are illustrated respectively;

[0055] Figure 4 A schematic diagram of a large model training process provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of a LoRA-based model training strategy provided in an embodiment of the present application;

[0057] Figure 6 This is a schematic diagram of the structure of a sweeping robot control device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0058] Before introducing this application solution, the relevant concepts involved in this article are first explained:

[0059] prompt: Instructions. When interacting with AI (such as an artificial intelligence model), you need to send instructions to AI. It can be a text description, such as "Please recommend me a pop song" when you interact with AI, or it can be a parameter description in a certain format. For example, if you want AI to draw in a certain format, you need to describe the relevant drawing parameters.

[0060] Large models: In the field of artificial intelligence, large models usually refer to large-scale pre-trained models. Such models are called "large" because they can be pre-trained on a large amount of data and can be transferred to a variety of downstream tasks. Their full English name is Large Pre-Trained Models or Large-Scale Pre-Training Models. Large models are characterized by their large scale and contain billions or even more parameters, which help them learn complex patterns in the data. The emerging capabilities of large models include but are not limited to: contextual learning, instruction following, code generation, step-by-step reasoning capabilities, etc. Large models can include large language models (LLMs) and multimodal large models. Large language models are mainly used to process text modal data. Multimodal large models further integrate multimodal capabilities on the basis of large language models and can process information in multiple modalities, such as images, text, audio, etc.

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] The present application provides a control solution for a sweeping robot, which can be applicable to the control process of the sweeping robot in a variety of occasions. It can clean different categories of interactive objects with matching interactive cleaning strategies, thereby improving the intelligence level of the sweeping robot.

[0063] The "interactive objects" referred to in this application can be understood as objects specified by the user in the environment of the sweeping robot. The user can configure matching interactive cleaning strategies for "interactive objects", and different categories of "interactive objects" can be configured with different interactive cleaning strategies, so that different categories of "interactive objects" can be cleaned using different interactive cleaning strategies. The "interaction" here can be understood as the meaning of configuring interactive cleaning strategies for environmental objects.

[0064] Depending on the environment in which the sweeping robot is located, there can be many categories of "interactive objects". Taking the home sweeping robot scenario as an example, "interactive objects" include but are not limited to: shoes, toys, animal feces, liquid stains, melon seed shells, rice grains, people, pets, etc.

[0065] The sweeping robot control method of the present application can be applied to Figure 1 As shown in the system architecture, the system may include a cleaning robot 100 and a control terminal 200 .

[0066] The cleaning robot 100 or the control terminal 200 can be used alone to execute the cleaning robot control method provided in the embodiment of the present application. In addition, the cleaning robot 100 and the control terminal 200 can also be used in collaboration to execute the cleaning robot control method provided in the embodiment of the present application.

[0067] Among them, the sweeping robot, also known as the automatic cleaning robot or the intelligent sweeping robot, is a smart home device specially designed to automatically complete the cleaning work on the ground. It integrates technologies such as microcomputer system, sensor, motor drive system and dust collection system, and can autonomously plan the cleaning path without manual operation or with less manual intervention, and clean and vacuum the ground, and some models even have the function of mopping the floor. Many sweeping robots also support remote control, setting cleaning plans, monitoring cleaning status, etc. through mobile phone APP, Bluetooth speakers, etc., which improves the convenience and intelligence level of use.

[0068] In a possible embodiment, the cleaning robot 100 may send data to the control terminal 200, and receive the processing result sent by the control terminal 200. Exemplarily, the cleaning robot 100 sends the surrounding environment information collected by the environment perception sensor to the control terminal 200, and the control terminal 200 identifies the category and location of the interactive objects in the environment based on the surrounding environment information, and feeds back to the cleaning robot 100. The cleaning robot 100 cleans the location of each interactive object according to the pre-configured interactive cleaning strategy that matches the category of the interactive object.

[0069] The control terminal 200 can be a base station device of the cleaning robot 100, or it can be a smart phone, a smart speaker, a server or other device.

[0070] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, usage scenarios, etc. of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0071] The present application provides a method for controlling a sweeping robot. Figure 1 Taking the computer device in the example as an example, the computer device can be Figure 1 The cleaning robot 100 or the system consisting of the cleaning robot 100 and the control terminal 200. Figure 2 , the sweeping robot control method specifically includes the following steps:

[0072] Step S100: Acquire surrounding environment information collected by the environment perception sensor on the sweeping robot.

[0073] Specifically, the sweeping robot may be provided with an environment perception sensor, which can be used to collect information about the surrounding environment. There may be various types of environment perception sensors, such as image sensors and depth cameras. For an image sensor, the surrounding environment information collected may be a planar image, such as an RGB image, a grayscale image, an infrared image, etc. For a depth camera, it may be a TOF camera, and the surrounding environment information collected may be a depth image.

[0074] In this embodiment, the environment perception sensor can use any one or more different types of environment perception sensors, and can collect one or more different types of surrounding environment information. In the subsequent embodiments of this application, the collected surrounding environment information includes RGB images and depth images as an example for explanation.

[0075] In this step, the surrounding environment information can be collected by the environment perception sensor while the sweeping robot is performing cruising or other non-cleaning tasks. In addition, the surrounding environment information can also be collected by the environment perception sensor while the sweeping robot is performing the cleaning task.

[0076] Step S110: Based on the surrounding environment information, identify the category and position of the interactive object in the environment, wherein the interactive object is an object in any category in the specified category set.

[0077] Specifically, the present application can pre-set different interactive objects, and this process can be a system default setting, such as setting which environmental objects belong to interactive objects according to the configuration logic. Alternatively, an interactive interface can be provided to the user, and the user can manually set (select in a list or manually enter) the interactive objects on the interactive interface. For all interactive objects, they can be divided into more than two different categories, and the categories of all interactive objects constitute a specified category set. Each category can include one or more interactive objects of different subcategories.

[0078] Furthermore, the present application can configure a matching interaction cleaning strategy for each category in the specified category set, so as to clean up the interactive objects of the corresponding category.

[0079] Optionally, the interaction sweeping strategies for different category matches may be different.

[0080] When all interactive objects are classified, the specific classification strategy can be set according to the system default settings or can be set by the user.

[0081] In a possible implementation, different interactive objects may be classified according to their interaction cleaning strategies, and the interactive objects with the same interaction cleaning strategy may be classified into the same category, and the category name may be defined according to the characteristics of the interactive objects in the same category.

[0082] Refer to Table 1 below, which illustrates a classification method for interactive objects.

[0083] Table 1

[0084]

[0085] Table 1 above illustrates several categories of interactive objects, including movable objects, dangerous items, liquid stains, particles, living things, etc.

[0086] Among them, the movable object is a category of objects that can be moved by the sweeping robot. It can be understood that the movable object does not mean that the object itself has the ability to move autonomously, but that the object can be moved by the sweeping robot. Of course, the movable object can be an object that has the ability to move autonomously, such as a toy car with rollers, and the movable object can also be an object that does not have the ability to move autonomously, such as slippers.

[0087] The sweeping robot in this application has the ability to move autonomously and can rely on the side walls of the body to push some light objects to move.

[0088] Dangerous items refer to items that may cause danger if touched by the robot vacuum cleaner, including but not limited to damage to the robot vacuum cleaner, contamination of the ground, etc. Dangerous items include animal feces, wires, etc.

[0089] Liquid stains refer to stains that exist on the ground in liquid form, such as water, milk, etc.

[0090] Particulate matter refers to objects that exist on the ground in granular form, such as melon seed shells, rice grains, etc.

[0091] Living things refer to living individuals, such as humans, pets, etc.

[0092] In this embodiment, each category of interactive objects may be configured by the system by default, such as automatically configuring each category of interactive objects according to the factory configuration, or each category of interactive objects may be manually set by the user, such as manually selecting or inputting each category of interactive objects by the user.

[0093] Step S120: For each interactive object, clean the location of the interactive object according to a pre-configured interactive cleaning strategy that matches the category of the interactive object.

[0094] For each interactive object in the environment identified in the aforementioned steps, the present application can clean the location of the interactive object according to the interactive cleaning strategy that matches the category of the pre-configured interactive object.

[0095] The matching relationship between the category of the interactive object and the interactive cleaning strategy may be stored in the sweeping robot itself, or in other devices that have a communication relationship with the sweeping robot.

[0096] This application supports users to configure the matching relationship between the categories of interactive objects and the interaction cleaning strategy.

[0097] The sweeping robot control method provided in the embodiment of the present application collects surrounding environment information through the environmental perception sensor on the sweeping robot, and can identify the category and position of the interactive objects in the environment based on this. Among them, the interactive object is an object under any category in the category set pre-specified by the user. The present application can count the categories of all interactive objects in advance, divide each interactive object into different categories in a certain way, and configure a matching interactive cleaning strategy for each category, so that the cleaning task can be performed for interactive objects of different categories using a matching interactive cleaning strategy, which greatly improves the intelligence level of the sweeping robot. In addition, the present application can support users to configure matching interactive cleaning strategies according to the characteristics of interactive objects of different categories, so as to improve the cleaning effect.

[0098] In some embodiments of the present application, the process of identifying the categories and locations of interactive objects in the environment based on the surrounding environment information in the aforementioned step S110 is introduced.

[0099] In one possible implementation, the present application can pre-train a deep neural network model as an object recognition model. Before training, a large number of environmental information samples are collected, and the categories and location labels of the interactive objects are annotated to form training data, and then the deep neural network model is trained using the training data. The trained model can be used as an object recognition model and deployed in a production environment for reasoning. On this basis, the currently acquired surrounding environment information can be fed into the object recognition model to obtain the category and location of the interactive objects output by the model.

[0100] Among them, the surrounding environment information can be a combination of RGB images and depth images. The combination of multimodal data can improve the accuracy of the recognition results.

[0101] In another possible implementation, the present application can use the capability of a large model to implement the object recognition task of step S110.

[0102] Specifically, the present application may call the configured large model to instruct the large model to refer to the surrounding environment information to identify the categories and locations of the interactive objects existing in the environment.

[0103] Taking the surrounding environment information as RGB images and depth images as an example, the large model can specifically adopt a multimodal large model that can process RGB images and depth images.

[0104] In one possible implementation, the present application is pre-configured with a prompt instruction prompt template, which includes a task instruction, an RGB image slot and a depth image slot. The task instruction is used to instruct the model to refer to the RGB image in the RGB image slot and the depth image in the depth image slot to identify the category and position of the interactive object in the image.

[0105] An optional example of a prompt template is:

[0106] “Given the given RGB image and depth image, identify the category and location of the interactive objects in the image.

[0107] RGB image: [image slot]

[0108] Depth Image: [text slot]".

[0109] Furthermore, the RGB image obtained in the aforementioned step S100 is filled into the image slot, and the depth image is filled into the depth image slot to obtain a first prompt instruction prompt, and the first prompt instruction prompt is input into the configured large model to obtain the category and position of the output interactive object.

[0110] Furthermore, the large model used in this embodiment can be fine-tuned and trained in advance through training data, that is, the general large model can be fine-tuned and trained through training data to be more suitable for the processing tasks in this embodiment. The training data includes samples of surrounding environment information and the categories and locations of annotated interactive objects. As shown in Figure 3, it illustrates several RGB image samples in a home scene, in which the interactive objects contained therein are water stains, slippers, and particulate matter.

[0111] In the method provided in this embodiment, by calling the capabilities of the multimodal large model and cooperating with the prompt instruction prompt designed in this application, the large model can be guided to accurately generate the categories and positions of the interactive objects existing in the image.

[0112] In another possible implementation, the RGB image and the depth image can be encoded by an encoder, and the encoded features can be sent to the feature processing layer of the configured large model to output the categories and positions of the interactive objects in the image through the large model.

[0113] When encoding RGB images and depth images through the encoder:

[0114] An optional method is to first concatenate the RGB image and the depth image, then encode the concatenated image, and send the encoded features to the feature processing layer of the large model.

[0115] Another optional method is to encode the collected RGB image and depth image separately, and then concatenate the encoded features to obtain concatenated features. The concatenated features are sent to the feature processing layer of the configured large model to output the category and position of the interactive objects in the image through the large model.

[0116] The process of encoding the RGB image and the depth image may use a configured encoder, for example, using a first image encoder to encode the RGB image to obtain a first encoding feature, and using a second encoder to encode the depth image to obtain a second encoding feature.

[0117] In order to guide the large model to output the categories and locations of the interactive objects in the image, the large model can be fine-tuned in advance using training data, which includes samples of surrounding environment information and the categories and locations of the annotated interactive objects. As shown in Figure 3, several RGB image samples in a home scene are shown, in which the interactive objects included are water stains, slippers, and particles.

[0118] Take the surrounding environment information sample including RGB image and depth image as an example:

[0119] In the fine-tuning training process, the RGB image can be encoded through the first encoder, and the depth image can be encoded through the second encoder. The encoded features of the two are spliced ​​and sent to the feature processing layer of the large model. The loss function is calculated based on the category and position of the interactive objects in the image generated by the large model, as well as the labels of the surrounding environment information samples, and the large model is fine-tuned and updated according to the loss function.

[0120] For the process of fine-tuning large models, a variety of training strategies can be used, such as Figure 4 and Figure 5 This example shows a training method using LoRA. The output of the large model fine-tuning training process is the category and location of the interactive object.

[0121] The LoRA (Low-Rank Adaptation) model training strategy is a lightweight model fine-tuning method that aims to adapt to specific tasks or data through a small amount of parameter adjustments while maintaining the capabilities of the original large model. LoRA assumes that the changes in the model parameters when adapting the task are low-rank, and uses low-rank decomposition to simulate these changes, thereby achieving fine-tuning of large models with extremely small parameters.

[0122] LoRA adds a small model based on the basic large model. Figure 5 As shown in the figure, the rectangular box on the left represents the parameters of one layer of the large model (Pretrained Weights), and the two trapezoidal boxes on the right represent the parameters of one layer of the small model. x and h represent the input and output features, respectively. Specifically, a bypass can be added next to the linear transformation module (such as the fully connected layer, the Embedding layer, and the convolutional layer) of the large model. This bypass is composed of the inner product of two small matrices A and B. By adjusting the parameters of these two small matrices, the change of the parameters of the large model can be simulated without modifying the parameters of the original model. Assume that the parameters of the linear transformation module of the basic large model are a d×d matrix, the bypass matrix A is a d×r matrix, and the matrix B is an r×d matrix (the value of r is usually very small, generally 2 or 3). The linear transformation module only participates in the forward calculation and does not participate in the gradient update. The matrices A and B participate in both the forward calculation and the gradient update, so only the small model (matrices A and B) needs to be updated.

[0123] It should be noted that in addition to the above-mentioned large model training strategy, other training strategies can also be adopted, which will not be described one by one in this application.

[0124] In some embodiments of the present application, several interactive cleaning strategies are introduced.

[0125] With respect to the several categories of interactive objects exemplified in Table 1 above, this embodiment introduces a possible example of an interactive cleaning strategy matched to each category.

[0126] For ease of description, the five categories of interactive objects in Table 1 from top to bottom are: a first interactive object, a second interactive object, a third interactive object, a fourth interactive object, and a fifth interactive object.

[0127] 1. For the categories of movable objects

[0128] The matching interactive cleaning strategies provided in this embodiment may include:

[0129] When the first set condition is met, the first interactive object corresponding to the movable object category in the environment is moved to the target position by the sweeping robot, and then the position where the first interactive object was before the movement is cleaned.

[0130] Specifically, in order to avoid interference with cleaning caused by the first interactive object in the environment, before cleaning the position where the first interactive object is located, the first interactive object can be first moved to the target position by the sweeping robot, and then the position where the first interactive object was before the movement can be cleaned to ensure the integrity of the cleaning.

[0131] In some optional scenarios, the present application may be to first identify all first interactive objects in a room before cleaning the room, and then move all first interactive objects in the room to the target position one by one, and then perform the cleaning task for the room.

[0132] In other optional scenarios, the sweeping robot in the present application may perform the cleaning task according to the planned cleaning path. When a first interactive object is detected in front of the path, the cleaning task is first paused, the identified first interactive object is moved to the target position, and then the cleaning task is continued after returning to the breakpoint position.

[0133] In some possible implementations, the first setting condition may be multiple, for example:

[0134] The first set condition may be: the sweeping machine moves along the cleaning path to within a set range around the first interactive object. The set range may be a circular area with a set radius r centered on the location of the first interactive object, or a rectangular area with a set side length centered on the location of the first interactive object, etc.

[0135] In addition, the first set condition may also be: after the sweeping robot recognizes the first interactive object in the environment, it plans a path from the current position to the position of the first interactive object, and moves to the first interactive object along the planned path.

[0136] Of course, the above only illustrates several optional examples of the first setting condition. In addition to these, other optional conditions may also be set, which will not be listed one by one in this embodiment.

[0137] In some possible implementations, the first interactive object is moved to a target position in the above interactive sweeping strategy. The target position may have multiple situations, for example:

[0138] a. The target location is the location in the environment pre-specified by the user.

[0139] Specifically, the user may pre-set a position in the human-computer interaction interface as the target position here, for example, setting a position in a balcony as the target position, etc.

[0140] b. The target position is the position in the area that the sweeping robot has cleaned.

[0141] Specifically, the cleaning robot can also autonomously select a position in the area that has been cleaned as the target position, so as to ensure that the subsequent cleaning of the target position will not be affected by moving the first interactive object to the target position.

[0142] c. The target location is outside the cleaning area required for this cleaning task.

[0143] Specifically, if the cleaning area required for this cleaning task is not the entire map area, the sweeping robot can select a location as the target location in the area outside the cleaning area required for this cleaning task in the entire map area, which can also avoid affecting the subsequent cleaning of the target location.

[0144] Of course, the above examples only illustrate several methods for selecting the target position. In addition, other strategies may be used to set the target position to ensure that the target position does not overlap with the position of the first interactive object before it is moved.

[0145] In some possible implementations, the interactive cleaning strategy that can be matched by the moving object category may further include:

[0146] When the second setting condition is met, the first interactive object at the target position is restored or arranged.

[0147] The position restoration is to move the first interactive object to the position before the movement.

[0148] For example, the first interactive object is at position A before being moved, and is moved to target position B. When the second setting condition is met, the first interactive object can be moved from target position B back to position A to restore the position of the first interactive object.

[0149] The above process of organizing the first interactable object specifically includes:

[0150] Considering that there may be more than one first interactive object in the environment, multiple first interactive objects may be moved to the target position according to the aforementioned strategy. In this embodiment, more than one first interactive object at the target position can be classified and placed according to the subcategories of the object to achieve object organization.

[0151] The subcategories of the objects are categories of different objects under the movable object category. For example, the “slippers” and “toys” in Table 1 both belong to different subcategories under the movable object category.

[0152] The second setting condition in the above interactive cleaning strategy may be multiple, including but not limited to:

[0153] After the position where the first interactive object was located before the push is cleaned, after the sweeping robot completes the current cleaning task, and other conditions.

[0154] That is, after the cleaning robot completes the cleaning task of the position of a first interactive object before the movement, it can perform the position restoration or arrangement step for the first interactive object. Alternatively, the cleaning robot can perform the position restoration or arrangement step for each first interactive object one by one after completing the cleaning task.

[0155] The above only illustrates several optional examples of the second set condition. In addition, other optional conditions can be set to ensure that the operation of restoring or sorting the position of the first interactive object is after the target time point in time, where the target time point is: the time when the cleaning task of the position of the first interactive object before the shift is completed.

[0156] 2. For dangerous goods categories

[0157] The matching interactive cleaning strategies provided in this embodiment may include:

[0158] When performing a cleaning task, the cleaning robot is controlled to stay away from the second interactive object corresponding to the dangerous object category.

[0159] In a possible implementation, the distance between the sweeping robot and the second interactive object is controlled to be no less than a first set distance threshold, where the first set distance threshold is a preset value, such as 30 cm or other values.

[0160] By controlling the cleaning robot to stay away from the second interactive object, it is prevented that the cleaning robot contacts the second interactive object and causes danger.

[0161] 3. For liquid stains

[0162] The matching interactive cleaning strategies provided in this embodiment may include:

[0163] The location of the third interactive object corresponding to the liquid stain category is cleaned by using a method of first mopping and then sweeping.

[0164] That is, the sweeping robot first uses the mopping mode to clean the liquid stains, and then uses the sweeping mode to clean, so as to avoid directly using the sweeping mode to disperse the liquid.

[0165] Since the mop of a sweeping robot is generally located at the rear of the body, the tail of the sweeping robot can be aimed at the liquid stain in the mopping mode, and the tail mop can be moved backwards to make the tail mop contact the liquid stain first, and then clean the liquid stain to prevent the liquid stain from contaminating other parts of the sweeping robot.

[0166] After the liquid stains are cleaned using the mopping mode, the sweeping mode is turned on, and the sweeping robot is controlled to turn around and clean the area where the liquid stains are located.

[0167] 4. For particle types

[0168] The matching interactive cleaning strategies provided in this embodiment may include:

[0169] The configured particle cleaning mode is used to clean the location of the fourth interactive object corresponding to the particle category.

[0170] In particle cleaning mode, the robot vacuum cleaner has a stronger suction force and a lower side brush rotation speed than in normal cleaning mode. By increasing the suction force, particles can be sucked into the storage box. At the same time, the side brush rotation speed is reduced to prevent particles from being scattered.

[0171] 5. For living creatures

[0172] The matching interactive cleaning strategies provided in this embodiment may include:

[0173] When a fifth interactive object corresponding to the living thing category is detected within a second set distance threshold in front of the cleaning path, a voice and / or light prompt is issued to prompt the fifth interactive object to leave the current position, and cleaning continues along the cleaning path after the fifth interactive object is detected to leave the current position.

[0174] Different voice prompts may be used for fifth interactive objects of different subcategories. For example, for humans, the voice prompt may be "Please move aside so that I can clean better." For pets, the voice prompt may be to make a corresponding pet sound or other set audio content, hoping to get the pet to leave the current location.

[0175] Of course, in addition to voice prompts, light prompts can also be used, the purpose of which is to prompt the fifth interactive object to leave the current position.

[0176] If it is detected that the fifth interactive object has left the current position, the cleaning robot can continue cleaning along the cleaning path. Conversely, if it is detected that the fifth interactive object has not left the current position, the fifth interactive object can be regarded as an obstacle and the cleaning path can be replanned using an obstacle avoidance algorithm.

[0177] In the above embodiment, several examples of matching interactive cleaning strategies are given for the categories of several interactive objects provided in Table 1. For other categories, matching interactive cleaning strategies can be formulated according to the characteristics of the interactive objects of the corresponding categories. By adopting the scheme of the present application, a matching interactive cleaning strategy is adopted for cleaning different interactive objects in the environment, which greatly improves the intelligence level of the sweeping robot.

[0178] The sweeping robot control device provided in an embodiment of the present application is described below. The sweeping robot control device described below and the sweeping robot control method described above can refer to each other.

[0179] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a sweeping robot control device disclosed in an embodiment of the present application.

[0180] like Figure 6 As shown, the device may include:

[0181] The environment information acquisition unit 11 is used to acquire the surrounding environment information collected by the environment perception sensor on the sweeping robot;

[0182] An object recognition unit 12 is used to recognize the category and position of an interactive object in the environment based on the surrounding environment information, wherein the interactive object is an object in any category of a specified category set, each category in the specified category set is configured with a matching interactive cleaning strategy, and the specified category set includes more than two categories;

[0183] The interactive cleaning unit 13 is used to clean the location of each of the interactive objects according to a pre-configured interactive cleaning strategy that matches the category of the interactive object.

[0184] In a possible implementation, the process of the object recognition unit identifying the category and location of the interactive object in the environment based on the surrounding environment information includes:

[0185] The configured large model is called to instruct the large model to refer to the surrounding environment information to identify the category and location of the interactive objects in the environment.

[0186] In a possible implementation, the designated category set includes at least a movable object category, and the movable object category is a category of objects configured to be moved by the sweeping robot.

[0187] In a possible implementation, the interactive cleaning strategy that can be matched by the moving object category includes:

[0188] When the first set condition is met, the first interactive object corresponding to the movable object category in the environment is moved to the target position by the sweeping robot, and then the position where the first interactive object was located before the movement is cleaned.

[0189] In a possible implementation, the first setting condition includes:

[0190] The sweeping robot moves along the cleaning path to a set range around the first interactive object;

[0191] or,

[0192] After identifying the first interactive object in the environment, the sweeping robot plans a path from the current position to the position of the first interactive object, and moves to the first interactive object along the planned path.

[0193] In a possible implementation, the target location is a location in an environment pre-specified by a user;

[0194] or,

[0195] The target position is the position in the area that has been cleaned by the sweeping robot;

[0196] or,

[0197] The target location is a location outside the cleaning area required for this cleaning task.

[0198] In a possible implementation, the interactive cleaning strategy that can be matched by the moving object category further includes:

[0199] When a second set condition is met, restoring or arranging the position of the first interactive object at the target position;

[0200] The position restoration is to move the first interactive object to the position before the movement, and the arrangement is to classify and place one or more of the first interactive objects at the target position according to subcategories of the objects, where the subcategories are categories of different objects under the movable object category.

[0201] In a possible implementation, the second setting condition includes:

[0202] After the sweeping robot completes the cleaning task;

[0203] or,

[0204] After the position where the first interactive object was located before the movement is cleaned.

[0205] In a possible implementation, the specified category set further includes: at least one of the following categories:

[0206] Dangerous goods category, liquid stain category, particulate matter category, living thing category.

[0207] In a possible implementation, the interactive cleaning strategy for dangerous goods category matching includes:

[0208] When performing a cleaning task, controlling the distance between the cleaning robot and the second interactive object corresponding to the dangerous object category to be no less than a first set distance threshold;

[0209] The interactive cleaning strategy for matching the liquid stain category includes:

[0210] Cleaning the location of the third interactive object corresponding to the liquid stain category by first mopping and then sweeping;

[0211] The particle class matching interactive cleaning strategy includes:

[0212] The configured particle cleaning mode is used to clean the location of the fourth interactive object corresponding to the particle category, wherein the particle cleaning mode has a greater suction force and a lower side brush rotation speed than the conventional cleaning mode;

[0213] The interactive cleaning strategy for matching the living creature category includes:

[0214] When a fifth interactable object corresponding to the living thing category is detected within a second set distance threshold in front of the cleaning path, a voice and / or light prompt is issued to prompt the fifth interactable object to leave the current position, and cleaning continues along the cleaning path after the fifth interactable object is detected to leave the current position.

[0215] In a possible implementation, the surrounding environment information includes a planar image captured by an image sensor and / or a depth image captured by a depth camera.

[0216] In some embodiments of the present application, a sweeping robot is further provided, which includes:

[0217] Equipment body;

[0218] A moving part, a cleaning part and an environment sensing sensor are arranged on the device body, and the environment sensing sensor is used to collect surrounding environment information;

[0219] The processor connected to the moving component, the cleaning component and the environment perception sensor is used to implement each step of the cleaning robot control method of any of the aforementioned embodiments.

[0220] Furthermore, the sweeping robot may also be provided with components such as a voice module and a radar module.

[0221] The sweeping robot provided by the present application collects information about the surrounding environment through an environmental perception sensor, and can identify the categories and locations of interactive objects in the environment based on this information. A matching interactive cleaning strategy is used to perform cleaning tasks for different categories of interactive objects, which greatly improves the intelligence level of the sweeping robot. In particular, when identifying the categories and locations of interactive objects in the environment, the ability of the large model can be used to improve the accuracy of the object recognition results.

[0222] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the sweeping robot control methods provided in the embodiments of the present application.

[0223] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the sweeping robot control methods provided in the embodiments of the present application.

[0224] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0225] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0226] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for controlling a sweeping robot, characterized in that: include: Obtain the surrounding environment information collected by the environment perception sensor on the sweeping robot; Based on the surrounding environment information, identifying the category and location of an interactive object in the environment, the interactive object being an object in any category in a specified category set, each category in the specified category set being configured with a matching interactive cleaning strategy, and the specified category set containing more than two categories; For each of the interactable objects, the location of the interactable object is cleaned according to a pre-configured interaction cleaning strategy that matches the category of the interactable object.

2. The method according to claim 1, characterized in that The process of identifying the type and location of the interactive objects in the environment based on the surrounding environment information includes: The configured large model is called to instruct the large model to refer to the surrounding environment information to identify the category and location of the interactive objects in the environment.

3. The method according to claim 1, characterized in that The designated category set includes at least a movable object category, and the movable object category is a category of objects configured to be moved by the cleaning robot.

4. The method according to claim 3, characterized in that The interactive cleaning strategies that can be matched by the moving object category include: When the first set condition is met, the first interactive object corresponding to the movable object category in the environment is moved to the target position by the sweeping robot, and then the position where the first interactive object was located before the movement is cleaned.

5. The method according to claim 4, characterized in that The first setting condition includes: The sweeping robot moves along the cleaning path to a set range around the first interactive object; or, After identifying the first interactive object in the environment, the sweeping robot plans a path from the current position to the position of the first interactive object, and moves to the first interactive object along the planned path.

6. The method according to claim 4, characterized in that The target location is a location in an environment pre-specified by a user; or, The target position is the position in the area that has been cleaned by the sweeping robot; or, The target location is a location outside the cleaning area required for this cleaning task.

7. The method according to claim 4, characterized in that The interactive cleaning strategy that can be matched by the moving object category also includes: When a second set condition is met, restoring or arranging the position of the first interactive object at the target position; The position restoration is to move the first interactive object to the position before the movement, and the arrangement is to classify and place one or more of the first interactive objects at the target position according to subcategories of the objects, where the subcategories are categories of different objects under the movable object category.

8. The method according to claim 7, characterized in that The second setting condition includes: After the sweeping robot completes the cleaning task; or, After the position where the first interactive object was located before the movement is cleaned.

9. The method according to claim 3, characterized in that: The specified category set also includes: at least one of the following categories: Dangerous goods category, liquid stain category, particulate matter category, living thing category.

10. The method according to claim 9, characterized in that The interactive cleaning strategies for dangerous goods category matching include: When performing a cleaning task, controlling the distance between the cleaning robot and the second interactive object corresponding to the dangerous object category to be no less than a first set distance threshold; The interactive cleaning strategy for matching the liquid stain category includes: Cleaning the location of the third interactive object corresponding to the liquid stain category by mopping first and then sweeping; The particle class matching interactive cleaning strategy includes: The configured particle cleaning mode is used to clean the location of the fourth interactive object corresponding to the particle category, wherein the particle cleaning mode has a greater suction force and a lower side brush rotation speed than the conventional cleaning mode; The interactive cleaning strategy for matching the living creature category includes: When a fifth interactable object corresponding to the living thing category is detected within a second set distance threshold in front of the cleaning path, a voice and / or light prompt is issued to prompt the fifth interactable object to leave the current position, and cleaning continues along the cleaning path after the fifth interactable object is detected to leave the current position.

11. The method according to any one of claims 1 to 10, characterized in that: The surrounding environment information includes a planar image captured by an image sensor and / or a depth image captured by a depth camera.

12. A sweeping robot control device, characterized in that: include: An environmental information acquisition unit, used to acquire surrounding environment information collected by an environmental perception sensor on the sweeping robot; an object recognition unit, configured to recognize the category and position of an interactive object existing in the environment based on the surrounding environment information, wherein the interactive object is an object of any category in a specified category set, each category in the specified category set is configured with a matching interactive cleaning strategy, and the specified category set includes more than two categories; The interactive cleaning unit is used to clean the location of each of the interactive objects according to a pre-configured interactive cleaning strategy that matches the category of the interactive object.

13. A sweeping robot, characterized in that: include: Equipment body; A moving part, a cleaning part and an environment sensing sensor are arranged on the device body, and the environment sensing sensor is used to collect surrounding environment information; The processor connected to the moving component, the cleaning component and the environment perception sensor is used to implement each step of the cleaning robot control method as described in any one of claims 1 to 11.

14. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, each step of the cleaning robot control method as described in any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the cleaning robot control method as described in any one of claims 1 to 11 is implemented.

Images