Cleaning equipment, control method and device thereof, storage medium and controller
By obtaining and identifying stain information in the image of the area to be cleaned, determining the target stain area and category, and controlling the cleaning equipment for cleaning, the problem of high complexity in stain recognition in automatic cleaning equipment is solved, and simple stain treatment and efficient cleaning are achieved.
Patent Information
- Application Number
- CN202410874246.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-01
- Publication Date
- 2024-11-12
AI Technical Summary
Existing automatic cleaning equipment has high complexity in stain identification and treatment, resulting in increased model complexity and difficulty in terminal deployment.
By obtaining images of multiple areas to be cleaned, identifying stain information, including areas and categories, determining target stain areas and categories, controlling cleaning equipment for cleaning, and using pre-trained target detection model and multi-label joint labeling method to reduce model complexity.
It realizes simple stain identification and treatment, reduces model complexity, improves cleaning efficiency and equipment service life, and adapts to application scenarios with limited resources at the terminal.
Smart Images

Figure CN118924196A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cleaning technology, and in particular to a cleaning device and a control method, device, storage medium and controller thereof. Background Art
[0002] With the development of technology, various automatic cleaning devices have emerged, such as sweeping robots and mopping robots. When receiving a cleaning instruction, the automatic cleaning device can automatically execute the cleaning instruction and complete the cleaning work, which not only liberates labor but also saves labor costs. In order to execute the cleaning instruction, the automatic cleaning device can capture visual information such as images and videos based on visual information, and process the visual information through a pre-trained model to perform cleaning.
[0003] However, in the automatic cleaning tasks based on visual information in related technologies, the stain recognition and processing process is relatively complex and difficult to implement in practice. Too much information extraction will increase the complexity of the model, resulting in greater difficulties in terminal deployment. Summary of the invention
[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent. To this end, the first object of the present invention is to provide a control method for a cleaning device to achieve simple stain identification and treatment.
[0005] A second object of the present invention is to provide a storage medium.
[0006] The third object of the present invention is to provide a controller.
[0007] A fourth object of the present invention is to provide a control device for a cleaning device.
[0008] A fifth object of the present invention is to provide a cleaning device.
[0009] To achieve the above-mentioned purpose, the first aspect of the embodiment of the present invention proposes a control method for a cleaning device, the method comprising: acquiring multiple images of the area to be cleaned; identifying each of the images of the area to be cleaned respectively to obtain stain information in each of the images of the area to be cleaned, wherein the stain information includes the area and category of the stain; determining a target stain area in the area to be cleaned and a target stain category corresponding to the target stain area based on the stain information in the multiple images of the area to be cleaned; and controlling the cleaning device to clean the stain in the target stain area based on the target stain area and its corresponding target stain category.
[0010] A second aspect of the present invention provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the control method of the cleaning device described above is implemented.
[0011] The third aspect of the present invention provides a controller, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the control method of the cleaning device described above is implemented.
[0012] The fourth aspect of the present invention provides a control device for a cleaning device, the device comprising: an acquisition module for acquiring multiple images of an area to be cleaned; an identification module for respectively identifying each of the images of the area to be cleaned to obtain stain information in each of the images of the area to be cleaned, wherein the stain information includes the area and category of the stain; a determination module for determining a target stain area in the area to be cleaned and a target stain category corresponding to the target stain area based on the stain information in the multiple images of the area to be cleaned; and a control module for controlling the cleaning device to clean the stain in the target stain area based on the target stain area and its corresponding target stain category.
[0013] A fifth aspect of the present invention provides a cleaning device, including the control device of the above-mentioned cleaning device.
[0014] According to the cleaning equipment and its control method, device, storage medium, and controller of the embodiment of the present invention, it is configured to obtain multiple images of the area to be cleaned; identify each image of the area to be cleaned respectively to obtain the stain information in each image of the area to be cleaned, wherein the stain information includes the area and category of the stain; determine the target stain area in the area to be cleaned and the target stain category corresponding to the target stain area according to the stain information in the multiple images of the area to be cleaned; and control the cleaning equipment to clean the stains in the target stain area according to the target stain area and its corresponding target stain category. By identifying the area and category of stains in the images of the area to be cleaned under different acquisition conditions, determining the target stain area and the target stain category, and cleaning the target stain area according to the target stain category, simple stain recognition and processing can be achieved, the model complexity can be reduced, and it can be deployed relatively simply on the terminal.
[0015] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 is a flow chart of a method for controlling a cleaning device according to an embodiment of the present invention;
[0017] Figure 2is a schematic diagram of a target stain area according to an example of the present invention;
[0018] Figure 3 is a schematic diagram of image annotation of an example of the present invention;
[0019] Figure 4 is a structural block diagram of a controller according to an embodiment of the present invention;
[0020] Figure 5 is a structural block diagram of a control device for a cleaning device according to an embodiment of the present invention;
[0021] Figure 6 4 is a structural block diagram of a cleaning device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0022] The cleaning device and control method, device, storage medium, and controller of the embodiments of the present invention are described below with reference to the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described with reference to the accompanying drawings are exemplary and should not be construed as limiting the present invention.
[0023] Figure 1 4 is a flow chart of a method for controlling a cleaning device according to an embodiment of the present invention.
[0024] like Figure 1 As shown, the control method of the cleaning equipment includes:
[0025] S11, acquiring multiple images of the area to be cleaned.
[0026] Specifically, after receiving the cleaning instruction, the cleaning device starts the cleaning movement, and during the process, it takes pictures through the camera device, and uses the pre-trained model to process the information obtained from the pictures.
[0027] When the cleaning device determines that a stain is photographed, the cleaning device will place itself in multiple acquisition conditions to photograph the stain under multiple acquisition conditions to obtain an image of the area to be cleaned.
[0028] The acquisition conditions of the above-mentioned multiple images of the area to be cleaned may be different. The above-mentioned acquisition conditions include the acquisition position. Specifically, the cleaning device will move around the photographed stain according to preset rules, such as approaching the photographed stain and moving around the stain. Moreover, the cleaning device needs to continuously photograph the stain during the movement, so as to obtain multiple images of the area to be cleaned. For example, the cleaning device can use each frame of the video captured as an image of the area to be cleaned in the process of approaching and moving around the stain.
[0029] Moreover, the above-mentioned collection conditions may also include ambient brightness. If the cleaning equipment identifies stains at a long distance or in poor viewing conditions such as insufficient brightness, the cleaning equipment needs to further confirm the stains. At this time, the cleaning equipment not only needs to get close to the stains to achieve closer observation, but also needs to automatically fill in the light according to the ambient brightness to make a judgment based on more frame recognition results.
[0030] S12, identifying each image of the area to be cleaned respectively to obtain stain information in each image of the area to be cleaned, wherein the stain information includes the area and category of the stain.
[0031] Specifically, after obtaining a plurality of images of the area to be cleaned, the pre-trained target detection model may be used to identify the images of the area to be cleaned, and obtain stain information in the images of the area to be cleaned.
[0032] It should be noted that the above target detection model can be set to only identify the type of stains, and the area of the stains can be obtained based on the detection frame when the stains are detected.
[0033] S13, determining a target stain region in the area to be cleaned and a target stain category corresponding to the target stain region according to stain information in the plurality of images of the area to be cleaned.
[0034] Since there may be multiple stains in the area to be cleaned, after obtaining the stain information in multiple images of the area to be cleaned, it is necessary to obtain the actual category of each stain according to the category of the stains in the multiple images of the area to be cleaned, and to obtain the actual area of each stain according to the area of the stains in the multiple images of the area to be cleaned.
[0035] Moreover, since the cleaning equipment has a single cleanable area, that is, the cleaning equipment will clean stains within a certain range in one cleaning, therefore, after obtaining the actual area and actual category of each stain in the area to be cleaned, it is necessary to determine the stains that the cleaning equipment can clean at one time based on the single cleanable area of the cleaning equipment and the actual area of each stain, and obtain a target stain area based on the area where the stains that the cleaning equipment can clean at one time are located. In this way, clustering can be achieved based on the position and recognition results during the observation process, the number and position of stains can be obtained, and stains that are closer can be planned to be cleaned at the same time according to the single cleanable area, and stains that are farther away can be planned to be cleaned at different times.
[0036] Obviously, there may be multiple stains in a target stain area, and thus there may be multiple stain categories. Therefore, after obtaining a target stain area, it is necessary to cluster the actual categories of all stains in the target stain area, that is, to count the actual categories of all stains in the target stain area, select the stain with the highest probability, and use the stain category with the highest probability as the target stain category corresponding to the target stain area. The standard for selecting the stain with the highest probability can be the area or number of stains, for example, the stain category with the largest single area or the largest total area can be used as the target stain category, and for another example, the stain category with the largest number can be used as the target stain category.
[0037] See also Figure 2 In the example shown, after the cleaning device identifies multiple solid stains, liquid stains, and solid-liquid mixed stains, it divides two target stain areas according to the area that can be cleaned by the cleaning device in a single time, and then counts the stain categories in the two target stain areas respectively, and selects the stain with the highest probability. The standard is the area. For a certain stain, its point area represents the category score given to the stain, so that the category of the target stain area is obtained according to the largest category score. The specific results are as follows: Figure 2 As shown, two target stain areas are identified as solid stain area 1 and liquid stain area 1.
[0038] S14, according to the target stain area and its corresponding target stain category, controlling the cleaning device to clean the stain in the target stain area.
[0039] Among them, the cleaning equipment operates according to the following principles: the original action is maintained in the stain-free area, the solid-liquid mixed stains are avoided, the solid stains are swept and sucked, and the liquid stains are washed and mopped. For example, for solid stains, first lift the mop to ensure that large particles of solid stains will not be adsorbed on the mop. Then use the side brush to gather the solid stains and guide them to the main brush, and then control the main brush to cooperate with vacuuming to suck the solid stains into the garbage collector. After completing the cleaning, check again whether there are any stains remaining in the cleaned area. If there are, clean the solid stains again. If not, use the mop to clean the fine dust residue. For liquid stains, lift the solid stain cleaning component and only use the mop to mop the floor. After mopping, check again. If there are no liquid stains, the cleaning is completed and the mop is returned to the base station to wash and dry. If there are, repeat the mopping, and after reaching a certain number of times, return to the base station to wash and dry the mop and continue working.
[0040] Therefore, it is set to obtain multiple images of the area to be cleaned, wherein the collection conditions of the multiple images of the area to be cleaned are different; each image of the area to be cleaned is identified respectively to obtain the stain information in each image of the area to be cleaned, wherein the stain information includes the area and category of the stain; according to the stain information in the multiple images of the area to be cleaned, the target stain area in the area to be cleaned and the target stain category corresponding to the target stain area are determined; according to the target stain area and its corresponding target stain category, the cleaning equipment is controlled to clean the stains in the target stain area. By identifying the area and category of the stains in the images of the area to be cleaned under different collection conditions, determining the target stain area and the target stain category, and cleaning the target stain area according to the target stain category, simple stain recognition and processing can be achieved, and the model only needs to perform stain category recognition, which reduces the complexity of the model and can be deployed more simply on the terminal.
[0041] In some embodiments of the present invention, a target stain area in the area to be cleaned and a target stain category corresponding to the target stain area are determined based on stain information in multiple images of the area to be cleaned, including: determining the world coordinates of the stains based on the areas of the stains in the multiple images of the area to be cleaned; clustering the stains and determining the target stains based on the world coordinates, and obtaining the target stain area based on the clustering results, wherein the target stains are stains with the same position in multiple images of the area to be cleaned; performing category voting on the target stains based on the stain information in the multiple images of the area to be cleaned to obtain a target category of the target stains; and obtaining a target stain category corresponding to the target stain area based on the target category of the stains in the target stain area.
[0042] Specifically, for each image of the area to be cleaned, the world coordinates of each stain need to be obtained according to the area of each stain therein, and after the world coordinates are obtained, the specific position of the same stain in different images of the area to be cleaned is determined according to the world coordinates.
[0043] Since different images of the area to be cleaned are collected under different collection conditions, that is, when the model recognizes the image of the area to be cleaned, even for the same stain, the recognition results in different images of the area to be cleaned may be different. Therefore, for a specific stain, all images of the area to be cleaned containing the stain can be determined first, and then each image of the area to be cleaned containing the stain can vote for the category of the stain, and the final category of the stain can be determined based on the voting results.
[0044] Therefore, after determining the world coordinates of the stain, it is necessary not only to cluster the stains according to the world coordinates to obtain a target stain area, but also to determine the target stain, which is any stain in the area to be cleaned.
[0045] After the target stain is determined, all images of the area to be cleaned in which the target stain is photographed are obtained. For each image of the area to be cleaned in which the target stain is photographed, a category vote is performed on the target stain according to the stain category of the target stain in the image, and the target category of the target stain is determined according to the voting result, and the target category obtained by the vote is determined as the above-mentioned actual category.
[0046] Therefore, to set up the model, it is only necessary to identify the type of stain in each image of the area to be cleaned, and the correction of the model is obtained by voting on multiple recognition results for the same stain, thereby further reducing the complexity of the model and making it easier to deploy on the terminal.
[0047] In some embodiments of the present invention, the target detection model is trained using a pre-acquired training set, and the training images in the training set are annotated using solid stain labels, liquid stain labels, and solid-liquid mixed stain labels. This stain definition avoids overly specific or abstract stain definitions that make data set collection and annotation difficult, and improves the actual application effect and recognition accuracy of the stain recognition function of the cleaning equipment under limited terminal resources.
[0048] Moreover, when labeling the training set, it is also possible to set solid-liquid mixed stain labels and solid stain labels for solid stains in each training image, solid-liquid mixed stain labels and liquid stain labels for liquid stains in each training image, and solid-liquid mixed stains or unidentified stains in each training image. The above labels can be implemented using bounding boxes. Therefore, setting only three labels can reduce the task difficulty of the model, thereby supporting the use of lightweight target detection models with lower complexity and enabling the target detection model to adapt to application scenarios with limited resources. Moreover, the multi-label labeling method ensures that the stains are detected as solid-liquid mixed stains. When it is impossible to determine whether it is solid or liquid, it can be marked first, thereby improving the recall rate of the target detection model for stain targets, reducing missed recognition of stains, and at the same time ensuring the recognition accuracy of solid stains and liquid stains, and reducing false triggering of incorrect cleaning modes. Furthermore, through the above-mentioned labeling method, it is possible to label each identified solid stain or liquid stain with two labels, thereby providing redundancy for the target detection model through multiple labels. When a label is incorrectly labeled or uncertain, other labels can be used as supplements to help the model make more accurate judgments.
[0049] Optionally, when labeling the training set, the stains in each training image are labeled with a solid-liquid mixed stain label, and further, the solid stains in each training image are labeled with a solid stain label, and the liquid stains in each training image are labeled with a liquid stain label. In this way, labeling can be achieved through a simple algorithm, thereby adapting to a lightweight target detection model. Moreover, by first labeling the solid-liquid mixed stains, the annotator can traverse the image more quickly and label the approximate stain area, and then subdivide these areas and label the specific solid stains and liquid stains. This gradually refined labeling method helps to improve labeling efficiency.
[0050] The above three labels correspond to the sweeping, washing and mopping, and avoidance reminders of conventional cleaning equipment respectively (assuming that the cleaning equipment cannot clean mixed solid-liquid stains when performing this action, if there are other cleaning actions for this scene, execute the corresponding action).
[0051] When labeling the dataset, all stain areas are assigned a label of solid-liquid mixed stains. If the stains can be distinguished as solid stains and liquid stains, another label is assigned. If the stains cannot be distinguished, only the label of solid-liquid mixed stains is retained. Defining only three labels can reduce the difficulty of the model's tasks, which means that a less complex target detection model can be used, which is also in line with the application scenario of limited resources of terminal devices.
[0052] For detailed annotation results, please refer to Figure 3 The example shown in Figure 3 In the example shown, the stain categories are labeled according to the stain morphology. Solid stains are labeled as both solid stains and solid-liquid mixed stains. Liquid stains are labeled as both liquid stains and solid-liquid mixed stains. Solid-liquid mixed stains and those whose exact morphology cannot be determined are only labeled as solid-liquid mixed stains. Figure 3 The four images in the figure are all labeled with bounding boxes of solid-liquid mixed stains. Furthermore, since the stains in the upper left and upper right corner images are identified as solid, the upper left and upper right corner images are labeled with bounding boxes of solid stains. Since the stains in the lower left corner image are identified as liquid stains, the lower left corner image is labeled with bounding boxes of liquid stains. At this time, the bounding boxes in the upper left and upper right corner images are labeled boxes of solid & solid-liquid mixed stains, the bounding box in the lower left corner image is labeled box of liquid & solid-liquid mixed stains, and the bounding box in the lower right corner image is labeled box of solid-liquid mixed stains.
[0053] Therefore, by adopting multi-label annotation, we can solve the problem that the stains are difficult to label due to the various forms of stains and the limited viewing angle of the equipment, ensuring that the stains can be detected as mixed solid and liquid stains. When it is impossible to determine whether it is solid or liquid, it can be marked first. This method improves the recall rate of the target detection model for stain targets, reduces the number of missed stains, and at the same time ensures the accuracy of solid and liquid stains recognition, reducing the false triggering of the wrong cleaning mode.
[0054] After labeling is completed, the training set is converted into the labeling format required by the lightweight model, and the lightweight object detection model is trained. During the training process, the parameters are adjusted to obtain the best model.
[0055] Optionally, during training, stains labeled with liquid stain labels can be directly trained as liquid stains, and stains labeled with solid stain labels can be directly trained as solid stains.
[0056] In some embodiments of the present invention, the stain information also includes confidence, and the image of the area to be cleaned is obtained by an image acquisition device installed on a cleaning device. Category voting is performed on the target stain based on the stain information in multiple images of the area to be cleaned to obtain a target category of the target stain, including: for each target stain in each image of the area to be cleaned, the area of the target stain and the actual distance between the target stain and the cleaning device when the image of the area to be cleaned is obtained according to the area of the target stain, and the category weight of the target stain is obtained according to the category, area, confidence and / or actual distance of the target stain, thereby obtaining multiple category weights of the target stain; and the category corresponding to the maximum value among the multiple category weights of the target stain is used as the target category of the target stain.
[0057] Specifically, since the distance between the image acquisition device and the stain and the size of the stain will affect the model's recognition accuracy of the stain, too far a distance or too small an area will result in a decrease in recognition accuracy. Therefore, when using the model to identify the stain, the distance and the area of the stain need to be taken into consideration.
[0058] Therefore, after determining the target stain, for each target stain, the area of the target stain needs to be determined, and for each image of the area to be cleaned, the actual distance between the cleaning device and the target stain when the image of the area to be cleaned is taken is obtained.
[0059] Moreover, when the model identifies the category of the stain, it not only outputs the identified category, but also outputs the confidence of the category. Therefore, after obtaining the area area and actual distance of the target stain, the category weight of the image of the area to be cleaned for the target stain category can be obtained for each image of the area to be cleaned according to the category, area area, confidence and / or actual distance of the target stain in the image of the area to be cleaned. The category weight is the voting result of the image of the area to be cleaned. After obtaining the voting result of each image of the area to be cleaned that captures the target stain, the category of the target stain in the image of the area to be cleaned corresponding to the maximum category weight is used as the target category of the target stain.
[0060] The above-mentioned area can adopt the pixel area of the stain in the image of the area to be cleaned, that is, the area of the stain is determined according to the number of pixels occupied by the stain in the image. Moreover, the enclosing frame of the stain can also be obtained, and the area of the stain is determined according to the number of pixels occupied by the enclosing frame.
[0061] As an example, assuming that stain 1 is captured in images 1, 2, and 3, it is necessary to obtain the actual distance, area, category, and confidence of the stain 1 in image 1, obtain the voting result of image 1 for the stain 1, obtain the actual distance, area, category, and confidence of the stain 1 in image 2, obtain the voting result of image 2 for the stain 1, obtain the actual distance, area, category, and confidence of the stain 1 in image 3, and obtain the voting result of image 3 for the stain 1. If the category weight obtained based on the information corresponding to image 3 is the largest, the category of stain 1 identified by the model for image 3 is used as the target category of stain 1.
[0062] That is to say, after the cleaning device observes the stain (any type of stain), it will move to the vicinity of the stain for reconfirmation. After successful confirmation, it will perform cleaning actions or avoidance reminders. During the observation movement of approaching and surrounding, the approximate world coordinate position of the stain can be obtained in combination with the detection frame in the detection result. By clustering the positions, it is determined whether the stain area is a multi-target or a single target. For the classification information determined to be the same target, motion voting is performed, that is, the farther away from the device and the fewer effective pixels observed (smaller bounding box area) are given a smaller weight, and vice versa.
[0063] Therefore, the category weight of the stain in each picture is obtained according to the distance between the image acquisition device and the stain and the size of the stain, and the target stain is determined according to the category weight. This makes it possible to capture the details of the stain from different angles and fields of view by taking multiple pictures, determine the category of the stain based on the multiple pictures taken, and increase the accuracy of stain recognition.
[0064] In some embodiments of the present invention, the category weight of the target stain is obtained according to the category, area, confidence and / or actual distance of the target stain, including: calculating a first difference between the area of the target stain and a preset optimal recognition area, and calculating a second difference between the actual distance of the target stain and the preset optimal recognition distance; calculating the category weight of the target stain according to the stain weight, the first difference and the second difference of the target stain, wherein the stain weight of the target stain is determined by the category of the target stain.
[0065] Specifically, in order to calculate the category weight, it is necessary to first determine the stain weight of the target stain according to the category of the target stain, that is, different categories of stains have different weights in the vote, so as to achieve better identification of the stain category. For example, if the categories of stains are determined to include solid stains, liquid stains, and solid-liquid mixed stains, then the stain weight of the solid-liquid mixed stain is determined to be less than the stain weight of the solid stain and the stain weight of the liquid stain, that is, if the model identifies the category of the target stain as solid stains or liquid stains for a certain image of the area to be cleaned, then the category weight calculated based on the image of the area to be cleaned is higher, that is, the image of the area to be cleaned has a higher voting weight, so that the target category of the target stain is more likely to be a solid stain or a liquid stain, thereby better identifying the stain category.
[0066] Moreover, it is also necessary to calculate the first difference between the area of the target stain and the preset optimal recognition area, as well as the second difference between the actual distance of the target stain and the preset optimal recognition distance, that is, to judge the credibility of the recognition result of the image of the area to be cleaned by the model based on the difference between the area of the target stain in a certain image of the area to be cleaned and the preset optimal recognition area and the difference between the actual distance and the preset optimal recognition distance.
[0067] Therefore, by setting the category weight of the target stain according to the first difference between the regional area of the target stain and the preset optimal recognition area and the second difference between the actual distance of the target stain and the preset optimal recognition distance, it is possible to obtain a comparison result between the actual parameters of the target stain and the preset optimal recognition parameters, and more accurately evaluate the pros and cons of the current recognition conditions, and then adjust the category weight based on the difference, which helps to improve the accuracy of stain category recognition.
[0068] In a specific embodiment, the category weight of the target stain is calculated according to the following formula:
[0069]
[0070] Among them, stain_class is the category weight, D is the target stain distance, best_d is the preset optimal recognition distance, σ dσ is the boundary value of the stain recognition distance, and D-best_d is the second difference. A is the area of the target stain, best_a is the preset optimal recognition area, and σ a is the boundary value of the stain recognition area, and w i is the stain weight, s is the stain confidence, and A-best_a is the first difference.
[0071] The above-mentioned boundary value of the stain recognition distance σ d is the maximum distance at which the cleaning device can achieve stain recognition, that is, if the distance exceeds this value, the cleaning device will not be able to recognize the stain. The above-mentioned boundary value of the stain recognition area σ a is the minimum area at which the cleaning device can achieve stain recognition, that is, if the stain area is smaller than this area, the cleaning device will not be able to recognize this area. The above-mentioned best_d, σ d , best_a, σ a can all be obtained according to the data set and test statistics.
[0072] As an example, assume that the target stain is stain 1, and the images of the area to be cleaned where stain 1 is captured include Image 1, Image 2, and Image 3. The stain categories that the model can recognize include solid stains, liquid stains, and solid-liquid mixed stains. When the value of i is 1, 2, or 3, they are w1: the weight of solid stains, w2: the weight of liquid stains, w3: the weight of solid-liquid mixed stains, and 0 < w3 < w1 = w2 < 1, that is, a smaller weight is given to the solid-liquid mixed stain category, and larger weights are given to the solid stain and liquid stain categories. For Image 1, the model outputs the stain confidence weights for recognizing stain 1 as a solid stain, the stain confidence weights for recognizing stain 1 as a liquid stain, and the stain confidence weights for recognizing stain 1 as a solid-liquid mixed stain, and substitute them into the above w d *w a *w i *s, take the largest calculation result as the stain_class of Image 1, and take the stain category corresponding to the largest calculation result as the stain category corresponding to Image 1. Then, use the same method to calculate the stain_class of Image 2 and Image 3, as well as the stain categories corresponding to Image 2 and Image 3. If the calculated stain_class of Image 3 is the largest, then take the stain category corresponding to Image 3 as the target category of stain 1.
[0073] Thus, it is possible to recognize the weight of each image in each stain category, calculate the stain category of each image according to the weight, obtain the target category of the stain according to the stain category corresponding to the image with the largest stain_class, so as to achieve accurate recognition of the stain category and reduce the false alarm rate.
[0074] In a specific embodiment, the class weight of the target stain is calculated according to the following formula:
[0075] stain_class = w d * s a * s i * s,
[0076] As an example, assume that the target stain is stain 1, and the images of the area to be cleaned where stain 1 is captured include Image 1, Image 2, and Image 3. The stain classes that the model can recognize include solid stains, liquid stains, and solid-liquid mixed stains. When the value of i is 1, 2, or 3, they are w1: the weight of solid stains, w2: the weight of liquid stains, w3: the weight of solid-liquid mixed stains, and 0 < w3 < w1 = w2 < 1. For Image 1, the model outputs the recognized stain class of stain 1 and the stain confidence weight of this stain class. Substitute the stain confidence weight and the stain weight corresponding to this stain class into the above w d * w a * w i * s, and take the calculation result as the stain_class of Image 1. Then, use the same method to calculate the stain_class of Image 2 and the stain_class of Image 3. If the calculated stain_class of Image 3 is the largest, then take the stain class corresponding to Image 3 as the target class of stain 1.
[0077] Thus, it is possible to recognize the stain class and weight of each image, obtain the target class of the stain according to the stain class corresponding to the image with the largest stain_class, thereby simplifying the calculation process, reducing the computational complexity, and better adapting to the lightweight object detection model.
[0078] In summary, the control method of the cleaning equipment of the embodiment of the present invention is set to obtain multiple images of the area to be cleaned, wherein the acquisition conditions of the multiple images of the area to be cleaned are different; each image of the area to be cleaned is identified respectively to obtain the stain information in each image of the area to be cleaned, wherein the stain information includes the area and category of the stain; according to the stain information in the multiple images of the area to be cleaned, the target stain area in the area to be cleaned and the target stain category corresponding to the target stain area are determined; according to the target stain area and its corresponding target stain category, the cleaning equipment is controlled to clean the stains in the target stain area. By identifying the area and category of the stains in the images of the area to be cleaned under different acquisition conditions, determining the target stain area and the target stain category, and cleaning the target stain area according to the target stain category, simple stain identification and processing can be achieved, and the model only needs to identify the stain category, which reduces the complexity of the model and can be deployed more simply on the terminal. By identifying the type and location of the stain, the pollution of the device and the floor caused by generalizing the stains can be avoided, and the cleaning efficiency of the equipment can be improved and the service life of the equipment can be increased. In addition, three stain categories can be set to define stains. The three stain category definitions are more concise, in line with mainstream cleaning equipment models and user needs, and are convenient for collecting and annotating data and training models. The multi-label joint annotation method avoids the problem of blurred boundaries during annotation, improves the accuracy of stain recognition and the accuracy of classified stain cleaning actions. The supporting motion voting post-processing method gives full play to the flexibility of cleaning equipment and can obtain more accurate stain categories and locations.
[0079] Based on the control method of the cleaning device in the above embodiment, the present invention proposes a storage medium.
[0080] In an embodiment of the present invention, a storage medium stores a computer program thereon, and when the computer program is executed by a processor, the control method of the cleaning device described above is implemented.
[0081] The storage medium of the embodiment of the present invention, by implementing the control method of the cleaning equipment described above, identifies the area and category of the stains in the image of the area to be cleaned under different acquisition conditions, determines the target stain area and the target stain category, and cleans the target stain area according to the target stain category, so that simple stain recognition and processing can be achieved. The model only needs to identify the stain category, which reduces the complexity of the model and can be deployed relatively simply on the terminal. By identifying the type and location of the stain, the device and floor contamination caused by generalizing the stains can be avoided, and the cleaning efficiency of the equipment can be improved and the service life of the equipment can be increased. Moreover, three stain categories can be set to define the stains. The three stain category definitions are more concise, meet the needs of mainstream cleaning equipment models and users, and are convenient for collecting data, labeling data, and training models. The multi-label joint labeling method avoids the problem of blurred boundaries during labeling, improves the accuracy of stain recognition and the accuracy of classified stain cleaning actions. The matching motion voting post-processing method gives full play to the flexibility of the cleaning equipment and can obtain more accurate stain categories and locations.
[0082] Based on the control method of the cleaning device in the above embodiment, the present invention further proposes a controller.
[0083] Figure 4 4 is a structural block diagram of a controller according to an embodiment of the present invention.
[0084] like Figure 4 As shown, the controller 500 includes: a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, such as through a bus 502. Optionally, the controller 500 may also include a transceiver 504. It should be noted that in actual applications, the transceiver 504 is not limited to one, and the structure of the controller 500 does not constitute a limitation on the embodiments of the present invention.
[0085] Processor 501 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 501 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0086] The bus 502 may include a path to transmit information between the above components. The bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 502 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0087] The memory 503 is used to store a computer program corresponding to the control method of the cleaning device of the above embodiment of the present invention, and the computer program is controlled and executed by the processor 501. The processor 501 is used to execute the computer program stored in the memory 503 to implement the contents shown in the above method embodiment.
[0088] in, Figure 4 The controller 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0089] The controller of the embodiment of the present invention, by implementing the control method of the cleaning equipment of the above embodiment, identifies the area and category of the stains in the image of the area to be cleaned under different acquisition conditions, determines the target stain area and the target stain category, and cleans the target stain area according to the target stain category, so as to realize simple stain recognition and processing. The model only needs to identify the stain category, which reduces the complexity of the model and can be deployed in the terminal more simply. By identifying the type and location of the stain, the pollution of the device and floor caused by generalizing the stains can be avoided, and the cleaning efficiency of the equipment can be improved and the service life of the equipment can be increased. Moreover, three stain categories can be set to define the stains. The three stain category definitions are more concise, meet the mainstream cleaning equipment models and user needs, and are convenient for collecting data, labeling data and training models. The multi-label joint labeling method avoids the problem of blurred boundaries during labeling, improves the accuracy of stain recognition and the accuracy of classified stain cleaning actions. The matching motion voting post-processing method gives full play to the flexibility of the cleaning equipment and can obtain more accurate stain categories and locations.
[0090] Figure 5 4 is a structural block diagram of a control device for a cleaning device according to an embodiment of the present invention.
[0091] like Figure 5As shown, the control device 100 of the cleaning equipment includes: an acquisition module 101, which is used to acquire multiple images of the area to be cleaned; an identification module 102, which is used to identify each image of the area to be cleaned respectively, and obtain the stain information in each image of the area to be cleaned, wherein the stain information includes the area and category of the stain; a determination module 103, which is used to determine the target stain area in the area to be cleaned and the target stain category corresponding to the target stain area according to the stain information in the multiple images of the area to be cleaned; and a control module 104, which is used to control the cleaning equipment to clean the stain in the target stain area according to the target stain area and the target stain category corresponding to the target stain area.
[0092] In some embodiments of the present invention, the acquisition conditions of the multiple images of the area to be cleaned are different, and the acquisition conditions include acquisition position and / or ambient brightness.
[0093] In some embodiments of the present invention, identifying the image of the area to be cleaned to obtain stain information in the image of the area to be cleaned includes: using a pre-trained target detection model to identify the image of the area to be cleaned to obtain stain information in the image of the area to be cleaned.
[0094] In some embodiments of the present invention, a target stain area in the area to be cleaned and a target stain category corresponding to the target stain area are determined based on stain information in multiple images of the area to be cleaned, including: determining the world coordinates of the stains based on the areas of the stains in the multiple images of the area to be cleaned; clustering the stains and determining the target stains based on the world coordinates, and obtaining the target stain area based on the clustering results, wherein the target stains are stains with the same position in multiple images of the area to be cleaned; performing category voting on the target stains based on the stain information in the multiple images of the area to be cleaned to obtain a target category of the target stains; and obtaining a target stain category corresponding to the target stain area based on the target category of the stains in the target stain area.
[0095] In some embodiments of the present invention, the stain information also includes confidence, and the image of the area to be cleaned is obtained by an image acquisition device installed on a cleaning device. Category voting is performed on the target stain based on the stain information in multiple images of the area to be cleaned to obtain a target category of the target stain, including: for each target stain in each image of the area to be cleaned, the area of the target stain and the actual distance between the target stain and the cleaning device when the image of the area to be cleaned is obtained according to the area of the target stain, and the category weight of the target stain is obtained according to the category, area, confidence and / or actual distance of the target stain, thereby obtaining multiple category weights of the target stain; and the category corresponding to the maximum value among the multiple category weights of the target stain is used as the target category of the target stain.
[0096] In some embodiments of the present invention, the category weight of the target stain is obtained according to the category, area, confidence and / or actual distance of the target stain, including: calculating a first difference between the area of the target stain and a preset optimal recognition area, and calculating a second difference between the actual distance of the target stain and the preset optimal recognition distance; calculating the category weight of the target stain according to the stain weight, the first difference and the second difference of the target stain, wherein the stain weight of the target stain is determined by the category of the target stain.
[0097] In some embodiments of the present invention, the category weight of the target stain is calculated according to the following formula:
[0098]
[0099] Among them, stain_class is the category weight, D is the target stain distance, best_d is the preset optimal recognition distance, σ d is the boundary value of the stain recognition distance, D-best_d is the second difference value, A is the target stain area, best_a is the preset optimal recognition area, σ a is the boundary value of the stain identification area, w i is the stain weight, s is the stain confidence, and A-best_a is the first difference.
[0100] In some embodiments of the present invention, the target detection model is trained using a pre-acquired training set, and the training images in the training set are annotated using solid stain labels, liquid stain labels, and solid-liquid mixed stain labels.
[0101] In some embodiments of the present invention, when labeling the training set, solid stains in each training image are labeled with solid-liquid mixed stain labels and solid stain labels, liquid stains in each training image are labeled with solid-liquid mixed stain labels and liquid stain labels, and solid-liquid mixed stains in each training image are labeled with solid-liquid mixed stain labels.
[0102] It should be noted that, for other specific implementations of the control device of the cleaning equipment in the embodiment of the present invention, reference may be made to the control method of the cleaning equipment in the above embodiment.
[0103] The control device of the cleaning equipment of the embodiment of the present invention identifies the area and category of the stains in the image of the area to be cleaned under different acquisition conditions, determines the target stain area and the target stain category, and cleans the target stain area according to the target stain category, which can realize simple stain recognition and processing. The model only needs to identify the stain category, which reduces the complexity of the model and can be deployed more simply on the terminal. By identifying the type and location of the stain, the pollution of the device and floor caused by generalizing the stains can be avoided, which can improve the cleaning efficiency of the equipment and increase the service life of the equipment. In addition, three stain categories can be set to define the stains. The three stain category definitions are more concise, meet the needs of mainstream cleaning equipment models and users, and are convenient for collecting data, labeling data, and training models. The multi-label joint labeling method avoids the problem of blurred boundaries during labeling, improves the accuracy of stain recognition and the accuracy of classified stain cleaning actions. The matching motion voting post-processing method gives full play to the flexibility of the cleaning equipment and can obtain more accurate stain categories and locations.
[0104] The invention also provides a cleaning device.
[0105] Figure 6 4 is a structural block diagram of a cleaning device according to an embodiment of the present invention.
[0106] like Figure 6 As shown, the cleaning device 10 comprises the control device 100 of the cleaning device mentioned above.
[0107] The cleaning device of the embodiment of the present invention, through the control device of the cleaning device of the above embodiment, identifies the area and category of the stains in the image of the area to be cleaned under different acquisition conditions, determines the target stain area and the target stain category, and cleans the target stain area according to the target stain category, which can realize simple stain recognition and processing. The model only needs to identify the stain category, which reduces the complexity of the model and can be deployed more simply on the terminal. By identifying the type and location of the stain, the device and floor pollution caused by generalizing the stains can be avoided, which can improve the cleaning efficiency of the equipment and increase the service life of the equipment. In addition, three stain categories can be set to define the stains. The three stain category definitions are more concise, meet the mainstream cleaning equipment models and user needs, and are convenient for collecting data, labeling data and training models. The multi-label joint labeling method avoids the problem of blurred boundaries during labeling, improves the accuracy of stain recognition and the accuracy of classified stain cleaning actions. The matching motion voting post-processing method gives full play to the flexibility of the cleaning equipment and can obtain more accurate stain categories and locations.
[0108] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.
[0109] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0110] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0111] In the description of this specification, the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and cannot be understood as a limitation on the present invention.
[0112] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0113] In the description of this specification, unless otherwise specified, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0114] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.
[0115] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. A method for controlling a cleaning device, characterized in that: The method comprises: Acquire multiple images of areas to be cleaned; Respectively identifying each of the images of the area to be cleaned to obtain stain information in each of the images of the area to be cleaned, wherein the stain information includes the area and category of the stain; Determining a target stain area in the area to be cleaned and a target stain category corresponding to the target stain area according to stain information in the plurality of images of the area to be cleaned; According to the target stain area and its corresponding target stain category, the cleaning device is controlled to clean the stain in the target stain area.
2. The control method of the cleaning equipment according to claim 1, characterized in that: The acquisition conditions of the multiple images of the area to be cleaned are different, and the acquisition conditions include acquisition position and / or ambient brightness.
3. The control method of the cleaning equipment according to claim 1, characterized in that: Identifying the image of the area to be cleaned to obtain stain information in the image of the area to be cleaned includes: The image of the area to be cleaned is identified using a pre-trained target detection model to obtain stain information in the image of the area to be cleaned.
4. The control method of the cleaning equipment according to claim 3, characterized in that: The determining, based on the stain information in the plurality of images of the area to be cleaned, a target stain area in the area to be cleaned and a target stain category corresponding to the target stain area comprises: Determining the world coordinates of the stain according to the area of the stain in the plurality of images of the area to be cleaned; Based on the world coordinates, the stains are clustered and target stains are determined, and the target stain area is obtained according to the clustering result, wherein the target stain is a stain having the same position in a plurality of images of the area to be cleaned; Performing category voting on the target stain according to the stain information in the plurality of images of the area to be cleaned, to obtain a target category of the target stain; According to the target category of the stain in the target stain area, a target stain category corresponding to the target stain area is obtained.
5. The control method of the cleaning equipment according to claim 4, characterized in that: The stain information also includes a confidence level. The image of the area to be cleaned is obtained by an image acquisition device installed on the cleaning device. The target stain is voted for according to the stain information in the plurality of images of the area to be cleaned to obtain the target category of the target stain, including: For each target stain in each of the images of the area to be cleaned, the area of the target stain and the actual distance between the target stain and the cleaning device when the image of the area to be cleaned is collected are obtained according to the area of the target stain, and the category weight of the target stain is obtained according to the category, area of the target stain, confidence level and / or actual distance, thereby obtaining multiple category weights of the target stain; The category corresponding to the maximum value among the multiple category weights of the target stain is used as the target category of the target stain.
6. The control method of the cleaning equipment according to claim 5, characterized in that: Obtaining the category weight of the target stain according to the category, area, confidence level and / or actual distance of the target stain, including: Calculating a first difference between the area of the target stain and a preset optimal recognition area, and calculating a second difference between the actual distance of the target stain and a preset optimal recognition distance; The category weight of the target stain is calculated according to the stain weight of the target stain, the first difference and the second difference, wherein the stain weight of the target stain is determined by the category of the target stain.
7. The control method of the cleaning equipment according to claim 6, characterized in that: The category weight of the target stain is calculated according to the following formula: Among them, stain_class is the class weight, D is the target stain distance, best_d is the preset optimal recognition distance, σ d is the boundary value of the stain identification distance, D-best_d is the second difference value, A is the target stain area, best_a is the preset optimal recognition area, σ a is the boundary value of the stain identification area, w i is the stain weight, s is the stain confidence, and A-best_a is the first difference.
8. The control method of the cleaning equipment according to claim 3, characterized in that: The target detection model is trained using a pre-acquired training set, and the training images in the training set are annotated using solid stain labels, liquid stain labels and / or solid-liquid mixed stain labels.
9. The control method of the cleaning equipment according to claim 8, characterized in that: When labeling the training set, The solid stains in each of the training images are labeled with a solid-liquid mixed stain label and a solid stain label, labeling the liquid stains in each of the training images with a solid-liquid mixed stain label and a liquid stain label, and The solid-liquid mixed stains or unidentified stains in each of the training images are marked with a solid-liquid mixed stain label.
10. The control method of the cleaning equipment according to claim 8, characterized in that: When labeling the training set, Marking the stains in each of the training images with a solid-liquid mixed stain label; The solid stains in each of the training images are labeled with a solid stain label, and the liquid stains in each of the training images are labeled with a liquid stain label.
11. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the control method of the cleaning device according to any one of claims 1 to 10 is implemented.
12. A controller, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the control method of the cleaning device according to any one of claims 1 to 10 is implemented.
13. A control device for a cleaning device, characterized in that: The device comprises: An acquisition module, used for acquiring multiple images of areas to be cleaned; An identification module, used to identify each of the images of the area to be cleaned, and obtain stain information in each of the images of the area to be cleaned, wherein the stain information includes the area and category of the stain; A determination module, configured to determine a target stain region in the area to be cleaned and a target stain category corresponding to the target stain region according to stain information in the plurality of images of the area to be cleaned; The control module is used to control the cleaning device to clean the stains in the target stain area according to the target stain area and its corresponding target stain category.
14. A cleaning device, characterized in that: A control device comprising the cleaning device as claimed in claim 13.
Citation Information
Cited By
Cleaning robot control method and system for photovoltaic power station
CN121033058A
Indoor space cleaning task processing method combined with image recognition technology
CN121121450A
