Cleaning method and device, electronic equipment and storage medium

Abstract

The invention provides a cleaning method and device, electronic equipment and a storage medium. The method comprises the steps that a layout map of a to-be-cleaned area and an area image of the to-be-cleaned area are acquired; the area image is acquired by the sweeping robot in a to-be-swept area at a corresponding pose, and the layout map is obtained by mapping of the sweeping robot; determining a velocity field of the to-be-cleaned area based on the layout map, the area image and a pose text of a pose corresponding to the area image; based on the velocity field, the sweeping robot is controlled to sweep the to-be-swept area, the defects that in a traditional scheme, sweeping modes of the sweeping robot are all-in-one and special areas cannot be emphatically swept are overcome, the corresponding sweeping strategy can be determined based on understanding of the scene, and therefore the sweeping process is proper, and the sweeping efficiency is improved. Flexible adjustment of cleaning modes in different scenes is achieved, various scenes can be effectively handled, intelligent cleaning is achieved, cleaning is more flexible, and efficiency is higher.

Description

Technical Field

[0001] The present invention relates to the field of smart home technology, and in particular to a cleaning method, device, electronic equipment and storage medium. Background Art

[0002] With the rapid development of smart home technology, sweeping robots, as an efficient and convenient household cleaning tool, have been widely used in modern family life. Traditional sweeping robots mostly clean the whole house based on a preset whole-house cleaning mode.

[0003] However, although this whole-house cleaning mode meets daily cleaning needs to a certain extent, in the current era of rapid development of intelligence, this fool-proof cleaning mode obviously can no longer meet the needs of users, that is, it lacks intelligence and personalization. Summary of the invention

[0004] The present invention provides a cleaning method, device, electronic device and storage medium, which are used to solve the problems in the prior art that the cleaning mode of the sweeping robot is fixed and lacks intelligence and personalization, and realize intelligent cleaning based on specific scenarios.

[0005] The present invention provides a cleaning method, comprising: Acquire a layout diagram of the area to be cleaned and a regional image of the area to be cleaned; the regional image is acquired by the sweeping robot in the area to be cleaned in a corresponding posture, and the layout diagram is obtained by mapping the sweeping robot; Determine the velocity field of the area to be cleaned based on the layout diagram, the area image, and the posture text corresponding to the posture of the area image; Based on the velocity field, the cleaning robot is controlled to clean the area to be cleaned.

[0006] According to a cleaning method provided by the present invention, the velocity field of the area to be cleaned is determined based on the layout diagram, the area image, and the posture text corresponding to the posture of the area image, including: Extracting features from the regional image and the posture text respectively to obtain image features and posture features; A velocity field prediction is performed based on the image features, the posture features and the layout diagram to obtain a velocity field of the area to be cleaned.

[0007] According to a cleaning method provided by the present invention, the velocity field prediction is performed based on the image features, the posture features and the layout diagram to obtain the velocity field of the area to be cleaned, including: Performing target detection based on the image features to obtain a target detection result; A velocity field prediction is performed based on the target detection result, the posture feature and the layout diagram to obtain a velocity field of the area to be cleaned.

[0008] According to a cleaning method provided by the present invention, the velocity field prediction is performed based on the target detection result, the posture feature and the layout diagram to obtain the velocity field of the area to be cleaned, including: Perform perspective mapping based on the posture features and the layout diagram, as well as the category and coordinates of each detection frame in the target detection result to obtain a target layout diagram; Perform velocity field prediction based on the target layout diagram to obtain the velocity field; The speed field includes the cleaning speeds of the areas corresponding to the detection frames and the cleaning speeds of other areas in the area to be cleaned.

[0009] According to a cleaning method provided by the present invention, the velocity field of the area to be cleaned is determined based on the layout diagram, the area image, and the posture text corresponding to the posture of the area image, including: Obtaining a demand description text, where the demand description text is used to characterize the cleaning demand of the user; Extracting features from the regional image to obtain image features; Extracting features from the posture text and the requirement description text respectively to obtain posture features and requirement features; A velocity field prediction is performed based on the image features, the posture features, the demand features, and the layout diagram to obtain a velocity field of the area to be cleaned.

[0010] According to a cleaning method provided by the present invention, the velocity field prediction is performed based on the image features, the posture features and the layout diagram to obtain the velocity field of the area to be cleaned, including: Based on the image features, the posture features and the layout diagram, a velocity field prediction model is applied to perform velocity field prediction to obtain a velocity field of the area to be cleaned; The velocity field prediction model is obtained by applying a sample layout diagram of a sample area, a sample area image of the sample area, a sample posture text corresponding to the posture of the sample area image, and a pseudo velocity field training of the sample area on the basis of the twin model; The twin model includes a multimodal large model and a multimodal small model, wherein the multimodal small model is constructed on the basis of the multimodal large model by a low-rank adaptation method; the parameters of the multimodal large model are not updated during the training of the velocity field prediction model.

[0011] According to a cleaning method provided by the present invention, the velocity field prediction model is obtained by training the sample layout diagram, the first sample area image of the sample area, the first sample posture text corresponding to the posture of the first sample area image, and the first pseudo velocity field of the sample area on the basis of the twin model; The twin model is trained based on the sample layout diagram, a second sample area image of the sample area, a second sample posture text corresponding to the posture of the second sample area image, and a second pseudo velocity field of the sample area; the first pseudo velocity field and the second pseudo velocity field are determined in different ways.

[0012] The present invention also provides a cleaning device, comprising: an acquisition unit, configured to acquire a layout diagram of the area to be cleaned and an area image of the area to be cleaned; the area image is acquired by the sweeping robot in the area to be cleaned in a corresponding position and posture, and the layout diagram is acquired by mapping by the sweeping robot; A prediction unit, configured to determine a velocity field of the area to be cleaned based on the layout diagram, the area image, and a posture text corresponding to the posture of the area image; A cleaning unit is used to control the cleaning robot to clean the area to be cleaned based on the velocity field.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the cleaning methods described above when executing the computer program.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the cleaning method described in any one of the above is implemented.

[0015] The cleaning method, device, electronic device and storage medium provided by the present invention predict the velocity field according to the layout diagram and the area image of the area to be cleaned, obtain the velocity field of the area to be cleaned, and control the sweeping robot to clean the area to be cleaned according to the velocity field, thereby overcoming the defects of the sweeping robot in the traditional scheme that the cleaning modes are all the same and the special areas cannot be cleaned in a focused manner, and can determine the corresponding cleaning strategy based on the understanding of the scene, so that the cleaning process is appropriate, and the cleaning mode can be flexibly adjusted in different scenes, which can effectively cope with various scenes, realize intelligent cleaning, and make cleaning more flexible and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a schematic flow chart of the cleaning method provided by the present invention; Figure 2 is an example diagram of a layout diagram of a home scene provided by the present invention; Figure 3 is an example diagram of a regional image of a home scene provided by the present invention; Figure 4 is an example diagram of the velocity field prediction process provided by the present invention; Figure 5 It is a schematic diagram of the velocity field prediction model training process provided by the present invention; Figure 6 is an example diagram of the determination process of the second pseudo velocity field provided by the present invention; Figure 7 It is a schematic diagram of the process of the cleaning device provided by the present invention; Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] At present, most sweeping robots clean based on a preset whole-house cleaning mode. That is, after the user gives the whole-house cleaning command to the sweeping robot, the sweeping robot will systematically clean all cleanable areas of the house according to the preset algorithm and path.

[0020] However, although this whole-house cleaning mode meets daily cleaning needs to a certain extent, it still has significant deficiencies in intelligence and personalization. Specifically, the existing sweeping robot technology solutions lack a deep understanding of home scenes and user needs. It is precisely because of its lack of scene recognition and user intention analysis capabilities that it cannot be flexibly adjusted according to the specific conditions of the home environment and the special needs of users, and can only perform stereotyped cleaning according to fixed procedures and paths, with poor flexibility and low cleaning efficiency.

[0021] In this regard, the present invention provides a cleaning method, which aims to predict the velocity field according to the layout map of the area to be cleaned obtained by mapping and the regional image of the area to be cleaned, so as to obtain the velocity field of the area to be cleaned, formulate a cleaning strategy according to the specific situation of the actual scene, and control the sweeping robot to clean based on this, thereby avoiding the problem of monotonous cleaning modes and inability to focus on cleaning special areas, realizing intelligent cleaning with higher efficiency.

[0022] Figure 1 It is a schematic diagram of the cleaning method provided by the present invention, such as Figure 1 As shown, the method can be applied to a sweeping robot and can provide it with an intelligent cleaning strategy. The specific execution subject can be a processor built into the sweeping robot. The method can also be applied to a server that directly controls the sweeping robot, which is not specifically limited in the embodiments of the present invention.

[0023] Step 110, obtaining a layout diagram of the area to be cleaned and a regional image of the area to be cleaned; the regional image is acquired by the sweeping robot in the area to be cleaned with a corresponding posture, and the layout diagram is obtained by mapping by the sweeping robot; Step 120, determining a velocity field of the area to be cleaned based on the layout diagram, the area image, and the posture text corresponding to the posture of the area image; Step 130: Based on the velocity field, control the cleaning robot to clean the area to be cleaned.

[0024] Considering that the current cleaning of sweeping robots is limited to set modes, they can only clean the whole house according to preset paths. On the one hand, this cleaning method has low cleaning efficiency, and on the other hand, it has poor flexibility and cannot distinguish between special areas and other areas. As a result, areas that should be cleaned are easily passed over, and areas that should be quickly passed over are cleaned instead. The "lightness" of the cleaning process is not properly controlled, and flexible cleaning is not possible, which can easily lead to poor user experience.

[0025] Based on this, in an embodiment of the present invention, it is proposed that a sweeping robot can be endowed with human-like thinking ability, so that it can clean intelligently, formulate corresponding cleaning strategies according to specific circumstances, realize flexible, intelligent and personalized cleaning, better fit the home scene, better adapt to user needs, and can give users a better user experience.

[0026] Specifically, before controlling the sweeping robot to clean, in the embodiment of the present invention, the area to be cleaned, i.e., the area to be cleaned, must first be determined. The area to be cleaned here can be a large area, such as a shopping mall, a square, etc., or a relatively small area, such as a home. The embodiment of the present invention does not specifically limit this. In actual applications, the sweeping robot is used as a smart home and is mostly used in home scenarios.

[0027] After determining the area to be cleaned, before cleaning, it is often necessary to build a map first. That is, the sweeping robot needs to first understand the house type and layout of the entire area to be cleaned, so that it can avoid obstacles, clean the area, and return to the original position accurately during the subsequent cleaning process. In other words, before cleaning, the sweeping robot needs to build a map of the area to be cleaned. Figure 2 is an example diagram of the layout diagram of a home scene provided by the present invention, such as Figure 2 As shown, by building a map, the layout structure of the area to be cleaned can be clarified. For example, it is clear which part of the home is the living room, which part is the bedroom, kitchen, and the relative position of the bedroom and the living room, etc., so as to obtain a layout map of the area to be cleaned. When corresponding to a home scene, this local map can also be called a floor plan.

[0028] Furthermore, after determining the layout map, the sweeping robot can perform image acquisition to collect images of the area to be cleaned from its perspective, thereby obtaining a regional image of the area to be cleaned. Specifically, after building a map to obtain the layout map, the sweeping robot can select multiple "viewing points" based on the layout map, and then perform image acquisition at these viewing points to collect images of the area to be cleaned. Figure 3 is an example diagram of a regional image of a home scene provided by the present invention, such as Figure 3 As shown, the degree of dirtiness and the location of objects in the home can be determined through the regional image, so as to facilitate the cleaning of the sweeping robot. It should be noted that while collecting the regional image, the sweeping robot will also record its own posture, and each regional image corresponds to a posture of the sweeping robot.

[0029] Taking the home scene as an example, the sweeping robot builds a map of the home scene. After obtaining the house plan, when starting the cleaning mode, it will select some viewing points in the home based on the house plan, and then collect images of the home scene at these viewing points, so that the actual image of the home can be obtained.

[0030] After obtaining the layout diagram and area image of the area to be cleaned, in an embodiment of the present invention, a velocity field prediction can be performed based on this to formulate a corresponding cleaning strategy, thereby facilitating the sweeping robot to clean based on this and achieving flexible and personalized cleaning.

[0031] Specifically, here, the layout diagram can be used as a reference, and the velocity field prediction can be performed based on the regional image to determine the cleaning mode of each sub-area in the layout diagram, so as to obtain the velocity field of the entire area to be cleaned. The velocity field here includes the cleaning speed of each sub-area in the area to be cleaned. The division of the sub-area can be determined according to the actual situation, or the user can pre-set it. The embodiment of the present invention does not make specific limitations on this. Preferably, in the embodiment of the present invention, the area to be cleaned is divided into sub-areas according to the layout diagram with a side length of 0.5 meters, and each sub-area in the area to be cleaned is obtained.

[0032] Specifically, here, the cleaning mode of each sub-region can be determined based on the objects in each sub-region, the types of objects, the degree of dirtiness of each sub-region, the types of stains, the positions of each sub-region, etc. reflected in the collected regional images. The cleaning mode may include the cleaning speed, cleaning time, number of cleaning times, etc. of each sub-region. For example, for areas with serious dirtiness and difficult to clean such as particle images and liquid stains, they can be set as key areas and given a lower cleaning speed and a longer cleaning time for thorough cleaning. For another example, for clean and tidy areas, areas that do not need to be specially cleaned, and areas that will not be easily involved, a higher cleaning speed and a shorter cleaning time can be given for quick cleaning. In this way, intelligent cleaning can be achieved based on the specific conditions of the area to be cleaned.

[0033] After determining the velocity field of the area to be cleaned, in an embodiment of the present invention, the sweeping robot can be controlled to perform intelligent cleaning, specifically, in accordance with the cleaning mode set in the velocity field, to clean each sub-area in the area to be cleaned. It should be noted that the sub-area cleaned each time can correspond to the entire area to be cleaned. Although the whole house is cleaned at this time, it is different from the traditional fool-style whole house cleaning mode, but it is an intelligent cleaning after understanding the actual scene. Correspondingly, the sub-area cleaned each time can be only a part of the entire area to be cleaned. For example, according to the regional image collected on the spot, only the living room is dirty. At this time, only the living room can be cleaned and set to the cleaning mode, and other areas can be set to the non-cleaning mode; for example, according to the user's instructions, it is set that the bedroom is not cleaned this time, then other areas outside the bedroom can be set to the cleaning mode, and the bedroom can be set to the non-cleaning mode.

[0034] The cleaning method provided by the present invention predicts the velocity field according to the layout diagram of the area to be cleaned and the regional image of the area to be cleaned, obtains the velocity field of the area to be cleaned, and controls the sweeping robot to clean the area to be cleaned according to the velocity field, thereby overcoming the defect that the sweeping robot in the traditional scheme has the same cleaning mode and cannot focus on cleaning special areas, and can determine the corresponding cleaning strategy based on the understanding of the scene, so that the cleaning process is appropriate, and the flexible adjustment of the cleaning mode in different scenes can be realized, so as to effectively cope with various scenes, realize intelligent cleaning, and make cleaning more flexible and more efficient.

[0035] Based on the above embodiment, step 120 includes: Feature extraction is performed on the regional image and the posture text respectively to obtain image features and posture features; The velocity field is predicted based on image features, posture features and layout diagram to obtain the velocity field of the area to be cleaned.

[0036] Specifically, in step 120, the process of determining the velocity field of the area to be cleaned according to the layout diagram, the area image, and the posture text corresponding to the posture of the area image may specifically include: First, feature extraction can be performed on the regional image of the area to be cleaned to extract information about placed objects, floor stains, etc., thereby encoding image features; at the same time, feature extraction can be performed on the posture of the sweeping robot corresponding to the regional image of the area to be cleaned to extract posture features, that is, to extract the features of the posture text corresponding to the regional image, which is referred to as posture features here.

[0037] Here, a feature extractor is specifically used, such as a text encoder and an image encoder in a multimodal model CLIP (Contrastive Language-Image Pretraining), to respectively extract features from the regional image and the pose text of the area to be cleaned. That is, the collected multi-frame regional images can be input into the CLIP image encoder, so that the input multi-frame regional images are encoded by the CLIP image encoder, and the features of the encoded multi-frame images are fused, such as adding and splicing, to obtain image features; the corresponding pose text is input into the CLIP text encoder, and the input pose text is encoded by the CLIP text encoder to obtain pose features.

[0038] Subsequently, the velocity field prediction can be performed based on the extracted image features, posture features, and the layout map obtained by mapping to obtain the velocity field of the area to be cleaned. That is, based on the layout map, the velocity field prediction is performed based on the image features to determine the cleaning mode of each sub-area, thereby obtaining the velocity field of the area to be cleaned. Here, specifically, based on the image features of the regional image, the objects in each sub-area, the type of objects, the degree of dirtiness of each sub-area, the type of stains, etc. can be analyzed, and the position of each sub-area in the layout map can be confirmed based on the posture features, and the cleaning mode of each sub-area can be determined on this basis.

[0039] Based on the above embodiment, velocity field prediction is performed based on image features, posture features and layout diagram to obtain the velocity field of the area to be cleaned, including: Perform target detection based on image features to obtain target detection results; The velocity field is predicted based on the target detection results, posture features and layout diagram to obtain the velocity field of the area to be cleaned.

[0040] Specifically, the process of predicting the velocity field according to the image features, the posture features and the layout diagram to obtain the velocity field of the area to be cleaned may include: After extracting the image features of the regional image, you can first perform object recognition and stain clarification based on the image features. That is, you can perform target detection based on the image features to detect the target in the regional image, thereby obtaining a target detection result. The target detection result here can include the detection frame of the target, as well as the coordinates and category of the detection frame. Here, target detection can be achieved through target detection models and algorithms, such as the target detection algorithm of the YOLO (You Only Look Once) series.

[0041] After obtaining the target detection result, in an embodiment of the present invention, the velocity field prediction can be performed based on the target detection result, as well as the posture characteristics and the layout diagram to obtain the velocity field of the area to be cleaned. That is, the layout diagram can be used as a reference to perform velocity field prediction based on the target detection result to determine the cleaning mode of each sub-area, thereby obtaining the velocity field of the entire area to be cleaned. Here, specifically, the objects, stains and their types in each sub-area can be determined through the target detection result, the position of each sub-area in the layout diagram can be confirmed based on the posture characteristics, and the cleaning mode of each sub-area can be determined based on this.

[0042] Based on the above embodiment, the velocity field is predicted based on the target detection result, the posture feature and the layout diagram to obtain the velocity field of the area to be cleaned, including: Based on the pose features and layout diagram, as well as the categories and coordinates of each detection box in the target detection result, perspective mapping is performed to obtain the target layout diagram; Predict the velocity field based on the target layout diagram to obtain the velocity field; The speed field includes the cleaning speed of the area corresponding to each detection frame and the cleaning speed of other areas in the area to be cleaned.

[0043] Specifically, when predicting the velocity field based on the target detection results, posture features, and layout diagram, the posture of the sweeping robot when collecting the corresponding area image, as reflected by the posture features, can be used to map each detection frame in the target detection result to the layout diagram, thereby determining the position of each detection frame on the layout diagram. That is, based on the layout diagram, each detection frame in the target detection result can be perspective mapped according to the posture features. Specifically, each detection frame can be mapped to the layout diagram through an inverse perspective transformation (IPM), and the category of each detection frame can be marked on the layout diagram, thereby obtaining a mapped layout diagram, that is, a target layout diagram.

[0044] After that, the speed field can be predicted based on the target layout diagram to obtain the speed field of the area to be cleaned; that is, the corresponding cleaning speed, duration, etc. can be assigned to the corresponding sub-area according to each detection frame on the target layout diagram and the category of each detection frame, so as to obtain the cleaning mode of each area. For example, for areas with a large number of particles, liquid stains and other serious dirt and difficult to clean, they can be set as key areas, and given a lower cleaning speed and a longer cleaning time for thorough cleaning; while for clean and tidy areas, areas that do not need to be specially cleaned and will not be easily involved, they can be given a higher cleaning speed and a shorter cleaning time for quick cleaning; for books, shoes and other areas that cannot be crossed, they can be set as restricted areas and given a negative speed to indicate that they are not cleaned; other areas can be given a normal speed and normal cleaning, so that intelligent cleaning can be achieved based on the specific conditions of the area to be cleaned.

[0045] Based on the above embodiment, step 120 includes: Obtain the requirement description text, which is used to represent the user's cleaning requirements; Extract features from the regional image to obtain image features; Feature extraction is performed on the posture text and the requirement description text respectively to obtain posture features and requirement features; The velocity field is predicted based on image features, posture features, demand features and layout diagram to obtain the velocity field of the area to be cleaned.

[0046] Specifically, in step 120, the process of determining the velocity field of the area to be cleaned according to the layout diagram, the area image, and the posture text corresponding to the posture of the area image may specifically include: Taking into account the actual application process, in addition to scene adaptability, in order to better improve user satisfaction and better adapt to user needs, in the embodiment of the present invention, it can also be given intelligence so that it can "understand" the language and clarify the needs, so as to carry out targeted cleaning, thereby realizing personalized cleaning and improving the user experience.

[0047] Specifically, the user's needs, that is, the need description text, can be obtained first. The description text can ensure the user's cleaning needs, such as which area in the cleaning area should be cleaned with emphasis, which area does not need to be cleaned, which area is simply cleaned, etc. The need description text can be directly input by the user. For example, the user issues a command to the sweeping robot by voice. After receiving the voice input, the sweeping robot can convert the voice command into text, thereby obtaining the user's need description text.

[0048] After that, the velocity field can be predicted based on the user's demand description text, area image and corresponding posture features, as well as the layout diagram to obtain the velocity field of the area to be cleaned. Figure 4 is an example diagram of the velocity field prediction process provided by the present invention, such as Figure 4 As shown, for two types of input, namely, posture text and demand description text, as well as regional image and layout diagram, they can first be input into the CLIP encoder for feature extraction, thereby obtaining the image features output by the CLIP image encoder, as well as the posture features and demand features output by the CLIP text encoder. Then, the velocity field prediction can be performed based on the extracted image features, posture features, demand features and layout diagram to obtain the velocity field of the area to be cleaned; here, specifically, the objects in each sub-region, the type of objects, the degree of dirtiness of each sub-region, and the type of stains reflected by the image features can be used to confirm the position of each sub-region in the layout diagram based on the posture features, and the areas such as key cleaning and simple cleaning can be determined based on the demand features, and the cleaning mode of each sub-region can be determined based on this. The cleaning mode may include the cleaning speed, cleaning time, number of cleaning times, etc. of each sub-region.

[0049] Here, the process of velocity field prediction can be realized by a multimodal large model, that is, the posture features, demand features and image features obtained by feature extraction, as well as the layout map can be input into the multimodal large model and the multimodal small model, so as to perform velocity field prediction by the multimodal large model and the multimodal small model, thereby obtaining the output velocity field of the area to be cleaned, which may include the cleaning mode of each sub-area. For example, for areas with serious dirt such as particle images and liquid stains, as well as areas that users set as key cleaning, they can be set as key areas, given a lower cleaning speed and a longer cleaning time, so as to clean them thoroughly; for clean and tidy areas, as well as areas that users mention can be cleaned simply, they can be given a higher cleaning speed and a shorter cleaning time for quick cleaning; for areas that cannot be crossed, such as books and shoes, they can be set as restricted areas, given a negative speed, indicating that they are not cleaned; other areas can be given a normal speed and normal cleaning, so that intelligent cleaning can be completed based on an accurate understanding of the scene and user instructions.

[0050] Among them, the multimodal small model is built on the basis of the multimodal large model. Compared with the multimodal large model, it greatly reduces the scale of parameters, making the training process more efficient. The large model and the small model can be pre-trained through samples and labels.

[0051] The following uses a home scenario as an example to explain the intelligent cleaning process of a sweeping robot: First, the sweeping robot will map the home scene and obtain a floor plan. Then, before cleaning, it will select some viewing points based on the floor plan, collect images of the home scene at these viewing points, and record the posture during the collection. After completing the image collection, the sweeping robot will send an interactive voice, such as "This cleaning begins, may I ask if the owner has any special requirements?" If there is voice input, the input content will be recognized as text as the demand description text. After being processed by the CLIP encoder, the extracted features can be used as the input of the multimodal large model and the multimodal small model. If not, the demand description text is empty. After that, the velocity field of the entire floor plan can be predicted by the multimodal large model and the multimodal small model. After obtaining the velocity field, the sweeping robot can clean based on the velocity field.

[0052] Based on the above embodiment, step 120 includes: Based on the image features, posture features and layout diagram, the velocity field prediction model is used to predict the velocity field and obtain the velocity field of the area to be cleaned; The velocity field prediction model is obtained by training the sample layout diagram of the sample area, the sample area image of the sample area, the sample posture text corresponding to the posture of the sample area image, and the pseudo velocity field of the sample area on the basis of the twin model; The twin model includes a multimodal large model and a multimodal small model. The multimodal small model is constructed on the basis of the multimodal large model through a low-rank adaptation method; the parameters of the multimodal large model are not updated during the training of the velocity field prediction model.

[0053] Specifically, the process of predicting the velocity field based on the layout diagram, the area image, and the posture text corresponding to the posture of the area image can be implemented by a model. That is, the image features, posture features, and the layout diagram can be input into the velocity field prediction model, so that the velocity field prediction model predicts based on the input information and outputs the predicted velocity field of the area to be cleaned.

[0054] It should be noted that the velocity field prediction model here is obtained by pre-training with samples and labels on the basis of the twin model. The twin model here includes a multimodal large model and a multimodal small model. The multimodal small model is constructed on the basis of the multimodal large model through low-rank adaptation. The multimodal large model is a currently open source multimodal large model.

[0055] During the training process, in order to make the general multimodal large model better adapt to the current task without significantly increasing the amount of training data, in the embodiment of the present invention, the model training can be performed in the LORA (Low-Rank Adaptation) mode. Figure 5 is a schematic diagram of the velocity field prediction model training process provided by the present invention, such as Figure 5 As shown in the figure, a small model is first constructed by LORA based on the multimodal large model. The constructed multimodal small model is a low-rank structure relative to the large model, including two low-rank matrices, namely low-rank matrix A and low-rank matrix B; then the twin model is trained through the sample layout diagram of the sample area, the sample area image of the sample area, the sample pose text corresponding to the pose of the sample area image, and the pseudo velocity field of the sample area to obtain the trained twin model, that is, the velocity field prediction model.

[0056] It should be noted that during the training of the twin models, the parameters of the multimodal large model only participate in the forward calculation and do not participate in the reverse gradient update; the parameters in the two low-rank matrices of the multimodal small model participate in both the forward calculation and the gradient update. In this way, the parameters of the multimodal large model are not updated, but only the parameters of the small model are updated. This allows the multimodal large model to adapt to the velocity field prediction task, thereby improving the scene adaptability and performance of the large model without increasing large-scale training data, thereby outputting an accurate velocity field in actual applications.

[0057] Based on the above embodiment, the velocity field prediction model is obtained by training on the basis of the twin model by applying the sample layout diagram, the first sample region image of the sample region, the first sample position text corresponding to the position of the first sample region image, and the first pseudo velocity field of the sample region; The twin model is trained based on a sample layout diagram, a second sample area image of the sample area, a second sample posture text corresponding to the posture of the second sample area image, and a second pseudo velocity field of the sample area; the first pseudo velocity field and the second pseudo velocity field are determined in different ways.

[0058] Specifically, in order to make the performance of the velocity field prediction model better and provide a more accurate and more adaptive cleaning strategy, when training the velocity field prediction model, the training process can be divided into two steps, namely coarse finetuning and fine finetuning.

[0059] Among them, coarse finetuning is to use the sample layout diagram of the sample area, the second sample area image of the sample area, the second sample posture text of the corresponding posture of the second sample area image, and the second pseudo velocity field of the sample area to pre-supervise the multimodal large model and the multimodal small model in the twin model to obtain a trained twin model. Fine finetuning is based on the trained twin model, and uses the data that has been manually annotated, that is, the sample layout diagram, the first sample area image of the sample area, the first sample posture text of the corresponding posture of the first sample area image, and the first pseudo velocity field of the sample area for training, and finally a trained model, that is, a velocity field prediction model, can be obtained. Here, the first sample area image and the second sample area image can be the same or different.

[0060] It is worth noting that, unlike the training data in the fine tuning stage that is manually annotated, the coarse fine tuning stage is a "pre-training" stage and requires more training data so that the model can learn the mapping relationship between the input image and text and the output velocity field. Therefore, in this stage, manual annotation to obtain training data is not advisable because it is too time-consuming, costly and labor-intensive. Based on this, in an embodiment of the present invention, it is proposed that the training data for the coarse fine tuning stage can be generated by automatic annotation.

[0061] Figure 6 is an example diagram of the determination process of the second pseudo velocity field provided by the present invention, such as Figure 6As shown in the figure, after obtaining the second sample area image in the coarse fine-tuning stage, target detection can be first performed based on it to detect objects, stains, etc., so as to obtain the corresponding target detection results, which are called sample detection results; then, based on the second sample pose text, coordinate mapping can be performed, specifically, each detection box and category in the target detection result is mapped to the sample layout diagram through inverse perspective transformation, so as to obtain the mapped sample layout diagram; then, the corresponding cleaning mode can be assigned to each sub-area according to the mapped sample layout diagram, so that automatic label generation can be realized, and a large amount of training data can be quickly obtained, thereby ensuring the adequacy and effectiveness of the training in the coarse fine-tuning stage, and further ensuring the model performance.

[0062] The cleaning device provided by the present invention is described below. The cleaning device described below and the cleaning method described above can be referenced to each other.

[0063] Figure 7 Schematic diagram of the structure of the cleaning device provided by the present invention. Figure 7 As shown, the device comprises: The acquisition unit 710 is used to acquire a layout diagram of the area to be cleaned and a regional image of the area to be cleaned; the regional image is acquired by the cleaning robot in the area to be cleaned in a corresponding posture, and the layout diagram is obtained by mapping by the cleaning robot; A prediction unit 720, configured to determine a velocity field of the area to be cleaned based on the layout diagram, the area image, and a posture text corresponding to the posture of the area image; The cleaning unit 730 is used to control the cleaning robot to clean the area to be cleaned based on the velocity field.

[0064] The cleaning device provided by the present invention predicts the velocity field according to the layout diagram of the area to be cleaned and the area image of the area to be cleaned, obtains the velocity field of the area to be cleaned, and controls the sweeping robot to clean the area to be cleaned according to the velocity field, thereby overcoming the defect that the sweeping robot in the traditional scheme has the same cleaning mode and cannot focus on cleaning special areas, and can determine the corresponding cleaning strategy based on the understanding of the scene, so that the cleaning process is appropriate, and the cleaning mode can be flexibly adjusted in different scenes, which can effectively cope with various scenes, realize intelligent cleaning, and make cleaning more flexible and efficient.

[0065] Based on the above embodiment, the prediction unit 720 is used for: Extracting features from the regional image and the posture text respectively to obtain image features and posture features; A velocity field prediction is performed based on the image features, the posture features and the layout diagram to obtain a velocity field of the area to be cleaned.

[0066] Based on the above embodiment, the prediction unit 720 is used for: Performing target detection based on the image features to obtain a target detection result; A velocity field prediction is performed based on the target detection result, the posture feature and the layout diagram to obtain a velocity field of the area to be cleaned.

[0067] Based on the above embodiment, the prediction unit 720 is used for: Perform perspective mapping based on the posture features and the layout diagram, as well as the category and coordinates of each detection frame in the target detection result to obtain a target layout diagram; Perform velocity field prediction based on the target layout diagram to obtain the velocity field; The speed field includes the cleaning speeds of the areas corresponding to the detection frames and the cleaning speeds of other areas in the area to be cleaned.

[0068] Based on the above embodiment, the prediction unit 720 is used for: Obtaining a demand description text, where the demand description text is used to characterize the cleaning demand of the user; Extracting features from the regional image to obtain image features; Extracting features from the posture text and the requirement description text respectively to obtain posture features and requirement features; A velocity field prediction is performed based on the image features, the posture features, the demand features, and the layout diagram to obtain a velocity field of the area to be cleaned.

[0069] Based on the above embodiment, the prediction unit 720 is used for: Based on the image features, the posture features and the layout diagram, a velocity field prediction model is applied to perform velocity field prediction to obtain a velocity field of the area to be cleaned; The velocity field prediction model is obtained by applying a sample layout diagram of a sample area, a sample area image of the sample area, a sample posture text corresponding to the posture of the sample area image, and a pseudo velocity field training of the sample area on the basis of the twin model; The twin model includes a multimodal large model and a multimodal small model, wherein the multimodal small model is constructed on the basis of the multimodal large model by a low-rank adaptation method; the parameters of the multimodal large model are not updated during the training of the velocity field prediction model.

[0070] Based on the above embodiment, the velocity field prediction model is obtained by training on the basis of the twin model by applying the sample layout diagram, the first sample area image of the sample area, the first sample posture text corresponding to the posture of the first sample area image, and the first pseudo velocity field of the sample area; The twin model is trained based on the sample layout diagram, a second sample area image of the sample area, a second sample posture text corresponding to the posture of the second sample area image, and a second pseudo velocity field of the sample area; the first pseudo velocity field and the second pseudo velocity field are determined in different ways.

[0071] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the cleaning method, which includes: obtaining a layout diagram of the area to be cleaned and a regional image of the area to be cleaned; the regional image is acquired by the sweeping robot in the area to be cleaned in a corresponding posture, and the layout diagram is obtained by mapping the sweeping robot; based on the layout diagram, the regional image, and the posture text of the corresponding posture of the regional image, determining the velocity field of the area to be cleaned; based on the velocity field, controlling the sweeping robot to clean the area to be cleaned.

[0072] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0073] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the cleaning methods provided by the above-mentioned methods, and the method includes: obtaining a layout map of the area to be cleaned, and a regional image of the area to be cleaned; the regional image is collected by a sweeping robot in the area to be cleaned in a corresponding posture, and the layout map is obtained by mapping the area by the sweeping robot; based on the layout map, the regional image, and the posture text of the corresponding posture of the regional image, determining the velocity field of the area to be cleaned; based on the velocity field, controlling the sweeping robot to clean the area to be cleaned.

[0074] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the cleaning methods provided by the above-mentioned methods, the method comprising: obtaining a layout map of the area to be cleaned, and a regional image of the area to be cleaned; the regional image is collected by a sweeping robot in the area to be cleaned in a corresponding posture, and the layout map is obtained by mapping the area by the sweeping robot; determining a velocity field of the area to be cleaned based on the layout map, the regional image, and a posture text of the corresponding posture of the regional image; and controlling the sweeping robot to clean the area to be cleaned based on the velocity field.

[0075] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cleaning method, characterized in that: include: Acquire a layout diagram of the area to be cleaned and a regional image of the area to be cleaned; The area image is acquired by the cleaning robot in the area to be cleaned in a corresponding posture, and the layout map is obtained by mapping by the cleaning robot; Determine the velocity field of the area to be cleaned based on the layout diagram, the area image, and the posture text corresponding to the posture of the area image; Based on the velocity field, the cleaning robot is controlled to clean the area to be cleaned.

2. The cleaning method according to claim 1, characterized in that: The determining of the velocity field of the area to be cleaned based on the layout diagram, the area image, and the posture text corresponding to the posture of the area image includes: Extracting features from the regional image and the posture text respectively to obtain image features and posture features; A velocity field prediction is performed based on the image features, the posture features and the layout diagram to obtain a velocity field of the area to be cleaned.

3. The cleaning method according to claim 2, characterized in that: The performing of velocity field prediction based on the image features, the posture features and the layout diagram to obtain the velocity field of the area to be cleaned includes: Performing target detection based on the image features to obtain a target detection result; A velocity field prediction is performed based on the target detection result, the posture feature and the layout diagram to obtain a velocity field of the area to be cleaned.

4. The cleaning method according to claim 3, characterized in that: The performing of velocity field prediction based on the target detection result, the posture feature and the layout diagram to obtain the velocity field of the area to be cleaned includes: Perform perspective mapping based on the posture features and the layout diagram, as well as the category and coordinates of each detection frame in the target detection result to obtain a target layout diagram; Perform velocity field prediction based on the target layout diagram to obtain the velocity field; The speed field includes the cleaning speeds of the areas corresponding to the detection frames and the cleaning speeds of other areas in the area to be cleaned.

5. The cleaning method according to any one of claims 1 to 4, characterized in that: The determining of the velocity field of the area to be cleaned based on the layout diagram, the area image, and the posture text corresponding to the posture of the area image includes: Obtaining a demand description text, where the demand description text is used to characterize the cleaning demand of the user; Extracting features from the regional image to obtain image features; Extracting features from the posture text and the requirement description text respectively to obtain posture features and requirement features; A velocity field prediction is performed based on the image features, the posture features, the demand features, and the layout diagram to obtain a velocity field of the area to be cleaned.

6. The cleaning method according to any one of claims 2 to 4, characterized in that: The performing of velocity field prediction based on the image features, the posture features and the layout diagram to obtain the velocity field of the area to be cleaned includes: Based on the image features, the posture features and the layout diagram, a velocity field prediction model is applied to perform velocity field prediction to obtain a velocity field of the area to be cleaned; The velocity field prediction model is obtained by applying a sample layout diagram of a sample area, a sample area image of the sample area, a sample posture text corresponding to the posture of the sample area image, and a pseudo velocity field training of the sample area on the basis of the twin model; The twin model includes a multimodal large model and a multimodal small model, wherein the multimodal small model is constructed on the basis of the multimodal large model by a low-rank adaptation method; the parameters of the multimodal large model are not updated during the training of the velocity field prediction model.

7. The cleaning method according to claim 6, characterized in that: The velocity field prediction model is obtained by training on the basis of the twin model by applying the sample layout diagram, the first sample region image of the sample region, the first sample position text corresponding to the position of the first sample region image, and the first pseudo velocity field of the sample region; The twin model is obtained by training based on the sample layout diagram, the second sample region image of the sample region, the second sample position text corresponding to the position of the second sample region image, and the second pseudo velocity field of the sample region; The first pseudo velocity field and the second pseudo velocity field are determined in different ways.

8. A cleaning device, characterized in that: include: An acquisition unit, used to acquire a layout diagram of the area to be cleaned and an area image of the area to be cleaned; The area image is acquired by the cleaning robot in the area to be cleaned in a corresponding posture, and the layout map is obtained by mapping by the cleaning robot; A prediction unit, configured to determine a velocity field of the area to be cleaned based on the layout diagram, the area image, and a posture text corresponding to the posture of the area image; A cleaning unit is used to control the cleaning robot to clean the area to be cleaned based on the velocity field.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the cleaning method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the cleaning method according to any one of claims 1 to 7 is implemented.

Images