Cleaning robot control method, device and cleaning robot

Abstract

The present application provides a cleaning robot control method, device and cleaning robot. The method comprises: generating a to-be-processed image based on a cleaning image collected when the cleaning robot is running; identifying hair in the to-be-processed image to obtain hair position information of the hair in the to-be-processed image; and controlling the cleaning robot to clean the hair in the cleaning image based on the hair position information. The method enables the cleaning robot to clean the hair during the cleaning process, thereby preventing hair from being entangled in the rotating components of the cleaning robot and mopping cloth from sticking to the hair and causing malfunctions, thereby improving the stability and user experience of the cleaning robot.

Description

Technical Field

[0001] The present application relates to the field of robots, and in particular to a cleaning robot control method, device and cleaning robot. Background Art

[0002] With the advancement of science and technology, more and more families use sweeping robots as floor cleaning tools. Intelligent sweeping robots can complete indoor floor cleaning tasks with a certain degree of artificial intelligence.

[0003] In the related art, most sweeping robots use side brushes or roller brushes to gather garbage and suck dust, hair, debris, etc. on the ground into their own garbage collection boxes.

[0004] However, in the related art, the cleaning path of a sweeping robot is usually planned according to the area of ​​the cleaning area. Due to its light weight, hair can easily entangle the rotating parts of the cleaning robot during the cleaning process, causing the rotating parts to rotate unsmoothly, or frequently move positions due to external forces, resulting in the robot skipping over them without cleaning them, resulting in a decrease in the cleaning effect. Summary of the invention

[0005] Based on the above technical problems, the present application provides a cleaning robot control method, device and cleaning robot to avoid hair entanglement in the rotating components of the cleaning robot and mop sticking to hair and causing malfunctions, thereby improving the stability and user experience of the cleaning robot.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0007] According to one aspect of an embodiment of the present application, a cleaning robot control method is provided, comprising:

[0008] Generate an image to be processed according to the cleaning images collected when the cleaning robot is running;

[0009] Identifying the hair in the image to be processed to obtain hair position information of the hair in the image to be processed;

[0010] According to the hair position information, the cleaning robot is controlled to clean the hair in the cleaning image.

[0011] In some embodiments of the present application, based on the above technical solution, the image to be processed includes a first image to be processed at a first scale and a plurality of second images to be processed at a second scale, and the second images to be processed are obtained by dividing the cleaned image; the hair in the image to be processed is identified to obtain the hair position information of the hair in the image to be processed, including:

[0012] Performing hair clump recognition on the first image to be processed by using a first hair recognition model to obtain first position information of the hair clump in the cleaned image;

[0013] Performing hair recognition on the plurality of second images to be processed by using a second hair recognition model to obtain second position information of hair clumps and sparse hair in the cleaned images;

[0014] Hair position information is determined according to the first position information and the second position information.

[0015] In some embodiments of the present application, based on the above technical solution, the hair recognition is performed on the plurality of second images to be processed by a second hair recognition model to obtain second position information of hair clumps and sparse hair in the cleaned image, including:

[0016] Performing image division on each second image to be processed by using a second hair recognition model to obtain a plurality of sub-images;

[0017] Using a second hair recognition model, identifying each second image to be processed and the corresponding multiple sub-images, determining a second image to be processed containing hair and a corresponding hair category, wherein the hair category is used to indicate that the hair in the second image to be processed is a hair clump or sparse hair;

[0018] According to the hair category of the second image to be processed of the hair and the position information in the cleaned image, the second image to be processed is merged to obtain the second position information of the hair clumps and sparse hair in the cleaned image.

[0019] In some embodiments of the present application, based on the above technical solution, before performing hair recognition on the plurality of second images to be processed by using a second hair recognition model to obtain second position information of hair clumps and sparse hair in the cleaned image, the method further includes:

[0020] Performing image division on each sample image in a sample image set to obtain a sample sub-image, wherein the sample image set includes the sample image, the hair category and position information of the hair in the sample image;

[0021] For each sub-sample image, whether the sub-sample image contains hair clumps or sparse hair is marked according to the hair category and position information of the hair in the sample image to which it belongs, so as to obtain a sample label of each sub-sample image and obtain a training sample set;

[0022] Performing sample amplification on the sample sub-images containing hair in the training sample set to obtain an amplified training sample set;

[0023] The recognition model to be trained is trained according to the amplified training sample set to obtain a second hair recognition model.

[0024] In some embodiments of the present application, based on the above technical solution, before the hair clumps are identified in the first to-be-processed image by using the first hair recognition model to obtain the first position information of the hair clumps in the cleaned image, the method further includes:

[0025] Acquire a hair clump image and mark the hair clump position in the hair clump image to obtain a sample image set;

[0026] For each sample image in the sample image set, the cleaned image is cropped according to a preset cropping rule to obtain a training sample set;

[0027] The model to be trained is trained according to the training sample set to obtain a first hair recognition model.

[0028] In some embodiments of the present application, based on the above technical solution, generating the image to be processed according to the cleaning image collected during the operation of the cleaning robot includes:

[0029] Acquire cleaning images collected when the cleaning robot is running;

[0030] According to a preset cropping rule, the cleaned image is cropped to obtain an image to be recognized at a first scale;

[0031] Performing image segmentation on the image to be identified at the first scale to obtain an image to be identified at a second scale, wherein the second scale is smaller than the first scale;

[0032] The image to be recognized at the first scale and the image to be recognized at the second scale are determined as images to be processed.

[0033] In some embodiments of the present application, based on the above technical solution, controlling the cleaning robot to retreat toward the hair and pass through the hair according to the hair position information to clean the hair in the cleaning image includes:

[0034] According to the hair position information of the hair in the image to be processed, controlling the cleaning robot to rotate and move and continuously collect images, so that the hair is located at a predetermined position in the collected image;

[0035] According to a preset angle parameter, the cleaning robot is controlled to rotate so that the hair is located in the backward direction of the cleaning robot;

[0036] The cleaning robot is controlled to retreat and perform cleaning to clean the hair in the image to be processed.

[0037] In some embodiments of the present application, based on the above technical solution, the step of controlling the cleaning robot to rotate and move and continuously collect images according to the hair position information of the hair in the image to be processed, so that the hair is located at a predetermined position in the collected image, includes:

[0038] According to the hair position information, controlling the cleaning robot to rotate so that the hair is located on the center line of the captured image;

[0039] The cleaning robot is controlled to move toward the hair so that the hair is located at the bottom edge of the captured image.

[0040] In some embodiments of the present application, based on the above technical solution, controlling the cleaning robot to rotate according to a preset angle parameter so that the hair is located in the backward direction of the cleaning robot includes:

[0041] Controlling the cleaning robot to rotate 180 degrees so that the hair is located directly behind the cleaning robot;

[0042] The controlling the cleaning robot to retreat and clean the hair in the image to be processed includes:

[0043] The cleaning robot is controlled to retreat along the central axis to pass through the hair and clean the hair.

[0044] According to one aspect of an embodiment of the present application, there is provided a cleaning robot control device, comprising:

[0045] An image generation module is used to generate an image to be processed based on the cleaning images collected when the cleaning robot is running;

[0046] A hair recognition module, used for recognizing hair in the image to be processed and obtaining hair position information of the hair in the image to be processed;

[0047] A cleaning control module is used to control the cleaning robot to clean the hair in the cleaning image according to the hair position information.

[0048] According to one aspect of an embodiment of the present application, a cleaning robot is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the cleaning robot control method in the above technical solution by executing the executable instructions.

[0049] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the cleaning robot method in the above technical solution is implemented.

[0050] In the embodiment of the present application, a multi-scale image is generated based on the cleaning image collected when the cleaning robot is running, and the hair position in the image is identified through the trained hair recognition model, and then the cleaning robot is controlled to clean the hair according to the hair position. In the above manner, the cleaning robot can clean the hair during the cleaning process, thereby avoiding hair entanglement with the rotating components of the cleaning robot and mop adhesion to the hair to cause malfunctions, thereby improving the stability and user experience of the cleaning robot.

[0051] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0053] Figure 1 This is a schematic structural diagram of a cleaning robot in an application scenario in an embodiment of the present application;

[0054] Figure 2 This is a schematic flow chart of the overall process of the cleaning robot control method in the embodiment of the present application;

[0055] Figure 3 is a schematic flow chart of the hair recognition process in the embodiment of the present application;

[0056] Figure 4 This is a schematic flow chart of a cleaning robot control method in an embodiment of the present application;

[0057] Figure 5 Schematic diagram of the structure of the second hair recognition model in the embodiment of the present application;

[0058] Figure 6 is a schematic flow chart of a hair recognition process using a second hair recognition model;

[0059] Figure 7 A schematic diagram of a process for generating an image to be processed in an embodiment of the present application;

[0060] Figure 8 Schematic process of the cleaning robot cleaning hair in the embodiment of the present application;

[0061] Fig. 9 The composition block diagram of the cleaning robot control device in the embodiment of the present application is schematically shown;

[0062] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0063] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0064] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0065] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0066] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0067] It should be understood that the solution of the present application can be applied to cleaning robot scenarios, and is specifically applied to scenarios in which the cleaning robot is controlled to clean hair. In such scenarios, the cleaning robot usually does not perform targeted cleaning of hair, especially pet hair, during the cleaning process. Instead, it plans a cleaning route according to the area and shape of the current cleaning space, or directly cleans according to a predetermined cleaning route. The middle sweep of the cleaning robot usually has the function of cutting and sucking hair, so as to prevent the hair from being entangled in the rotating shaft of the middle sweep and affecting the rotation of the middle sweep. However, during the cleaning process, the hair on the ground is often entangled in the universal wheels, side sweeps and drive wheels of the sweeping robot, thereby affecting the operation of the cleaning robot. Applying the solution of the present application, the cleaning robot will identify the hair on the ground and clean the hair, so as to avoid the hair being ignored or being entangled in the universal wheels, side sweeps and drive wheels, thereby avoiding the cleaning efficiency being affected.

[0068] The following describes the application scenarios in the embodiments of the present application. Figure 1 , Figure 1 FIG. 1 is a schematic diagram of a cleaning robot in an application scenario in an embodiment of the present application. Figure 1 As shown, the cleaning robot 100 mainly includes a universal wheel 101, a side sweep 102, a middle sweep 103, a driving wheel 104 and a mop 105. Among them, the universal wheel 101 is used to control the direction of travel of the cleaning robot 100, and the side sweep 102 is used to sweep the dust, hair and debris in the direction of travel into the middle sweep 103. The middle sweep 103 is used to suck the target object into the cleaning robot 100. Among them, the middle sweep 103 also has an anti-entanglement function such as cutting off hair or hair. The driving wheel 140 is used to provide the cleaning robot 100 with power to move forward or backward. The mop 105 is used to wipe the cleaned floor. In an embodiment of the present application, the cleaning robot 100 also includes a camera device, which is usually installed in the center of the robot's front bump, so as to shoot the image in the direction of travel of the cleaning robot. The camera device can also be installed on both sides of the front bump, and various suitable camera devices such as a monocular camera or a binocular camera can be used. The camera device is usually installed at a horizontal angle or slightly tilted toward the ground to obtain images in the forward direction. It is understandable that the above cleaning robot structure is only a schematic structure, and each component thereof can be omitted or replaced by other components, and the position of each component can also be appropriately adjusted.

[0069] The following is an introduction to the overall process of the cleaning robot control method in the embodiment of the present application. Figure 2 , Figure 2 FIG. 1 is a schematic flow chart of the overall process of the cleaning robot control method in the embodiment of the present application. Figure 2 As shown, in the process of cleaning, the cleaning robot will obtain the surrounding image through the camera in step 201, and then, in step 202, for the captured image, the control device will intercept the lower half of the image. Then, for the intercepted image, in step 203, the control device performs hair recognition through the hair clump recognition algorithm and the sparse hair recognition algorithm. If a hair clump or sparse hair is recognized, the control device will control the cleaning robot to align with the recognized hair in step 204, and then control the cleaning robot to move near the hair in step 205. In step 206, the control device will control the cleaning robot to rotate 180 degrees and control the cleaning robot to lift the mop and retreat, so as to directly clean the hair through the middle sweep. In one embodiment, when performing hair recognition, pet recognition will also be performed based on the captured picture. If the presence of a pet is recognized, the control device will increase the rotation speed of the middle sweep to ensure the anti-entanglement effect for the hair.

[0070] The following is a detailed description of the process of hair recognition using the hair clump recognition algorithm and the sparse hair recognition algorithm in step 203. Figure 3 , Figure 3 FIG. 1 is a schematic flow chart of the hair recognition process in the embodiment of the present application. Figure 3 As shown, after step 202, the control device performs hair detection by calling the target detection model in step 301, and determines the recognition result of hair group recognition in step 302. In step 303, the control device divides the captured image into image sub-regions, and then performs hair recognition on each sub-region through the sub-region classification model in step 304. Subsequently, in step 305, the recognition result of sub-region hair recognition is determined. In step 306, the recognition result of hair group recognition and the recognition result of sub-region hair recognition are combined to determine the final hair recognition result. In addition, in one embodiment, during the operation of the cleaning robot, the control device also uses the target detection model to identify household pets. If a pet is identified, the rotation speed control for hair is performed according to the recognition result during cleaning, such as speeding up the rotation speed of the middle sweep, so as to perform targeted hair cutting and other processing.

[0071] The following is a detailed description of the technical solution provided by this application in conjunction with the specific implementation method. For ease of introduction, please refer to Figure 4 , Figure 4 : is a schematic flow chart of a cleaning robot control method in an embodiment of the present application. The method can be applied to the control system of the cleaning robot. In the embodiment of the present application, the fault detection method may include the following steps S410 to S430:

[0072] Step S410, generating an image to be processed according to the cleaning image collected when the cleaning robot is running.

[0073] In this embodiment, the control device generates an image to be processed based on the cleaning image collected when the cleaning robot is running. Specifically, the cleaning image is usually an image obtained by the cleaning robot in real time through a camera or other shooting equipment during the cleaning process. The image will include an image of the direction of advance of the cleaning robot. Therefore, the cleaning image will include an image of the area that the cleaning robot is about to clean. The image to be processed can be a single image or multiple images of multiple scales. The multi-scale image is usually an image obtained by scaling, intercepting or dividing the cleaning image, including multiple images of different sizes or different clarity. In one embodiment, the cleaning robot also pre-processes the image to be processed, such as sharpening it. The multi-scale image is conducive to the recognition of small hair targets, thereby improving the precision and accuracy of the recognition.

[0074] Step S420: identifying the hair in the image to be processed to obtain hair position information of the hair in the image to be processed.

[0075] In this embodiment, the control device is configured with a trained hair recognition model or image recognition algorithm. The hair recognition model or image recognition algorithm is trained for hair recognition and is used to recognize the hair in the image to be processed and the position of the hair in the image to be recognized and in the cleaned image. The image to be processed is input into the trained hair recognition model or image recognition algorithm for recognition, so that the hair position information of the hair in the image to be processed can be obtained.

[0076] Step S430: According to the hair position information, the cleaning robot is controlled to retreat toward the hair and pass through the hair, so as to clean the hair in the cleaning image.

[0077] Specifically, according to the hair position information, the control device can calculate the relative position between the cleaning robot and the hair in the identified cleaning image. According to the relative position, the control device can control the cleaning robot to place the hair in the direction of the cleaning robot's retreat according to a predetermined program, and retreat through the hair. There is usually no side sweeper or wheel or other components in the direction of the cleaning robot's retreat, so the cleaning robot can directly use the middle sweeper to clean the identified hair when retreating. In one embodiment, a mop is provided in the direction of the cleaning robot's retreat. When retreating to clean the hair, the cleaning robot will raise the mop to prevent the hair from sticking to the mop. Specifically, the predetermined program usually performs a specific cleaning method for the hair, so as to prevent the hair from being entangled in rotating components such as the driving wheel during cleaning. For example, the cleaning robot is controlled to face the hair, and the side sweeper is stopped before cleaning, so as to prevent the hair from being entangled in components such as the side sweeper and the driving wheel.

[0078] In the embodiment of the present application, a multi-scale image is generated based on the cleaning image collected when the cleaning robot is running, and the hair position in the image is identified through the trained hair recognition model, and then the cleaning robot is controlled to clean the hair according to the hair position. In the above manner, the cleaning robot can clean the hair during the cleaning process, thereby avoiding hair entanglement with the rotating components of the cleaning robot and mop adhesion to the hair to cause malfunctions, thereby improving the stability and user experience of the cleaning robot.

[0079] In one embodiment of the present application, based on the above technical solution, the image to be processed includes a first image to be processed at a first scale and a plurality of second images to be processed at a second scale, the second image to be processed is obtained by dividing the cleaned image, and the above step S420, identifying the hair in the image to be processed to obtain the hair position information of the hair in the image to be processed, specifically includes the following steps:

[0080] Performing hair clump recognition on the first image to be processed by using a first hair recognition model to obtain first position information of the hair clump in the cleaned image;

[0081] Performing hair recognition on the plurality of second images to be processed by using a second hair recognition model to obtain second position information of hair clumps and sparse hair in the cleaned images;

[0082] Hair position information is determined according to the first position information and the second position information.

[0083] In this embodiment, the image to be processed includes a first image to be processed at a first scale and a plurality of second images to be processed at a second scale. The first image to be processed at the first scale is usually obtained by directly taking a screenshot based on the clean image, while the second image to be processed at the second scale is obtained by segmenting the first image to be processed. Therefore, the second scale is usually smaller than the first scale. Specifically, the control device will perform hair group recognition on the first image to be processed through the first hair recognition model to obtain the first position information of the hair group in the clean image. The first hair recognition model is used to recognize the hair group in the image to be processed. The hair group is usually relatively large in volume. Therefore, the first image to be processed at a relatively large first scale can be used to recognize the hair group, so as to obtain the position of the hair group in the first image to be processed, and thus the first position information of the hair group in the clean image can be calculated. The control device will also perform hair recognition on the plurality of second images to be processed through the second hair recognition model to obtain the second position information of the hair group and sparse hair in the clean image. Specifically, the control device will identify each second image to be processed through the second hair recognition model, thereby identifying whether each second image to be processed contains hair clumps or sparse hair. Subsequently, according to the position of the second image to be processed containing hair clumps and sparse hair in the cleaned image, the second position information of the hair clumps and sparse hair in the cleaned image is obtained. Finally, the control device can make a comprehensive judgment based on the first position information and the second position information, such as removing duplicate recognition results and accurately locating overlapping recognition results, thereby determining the hair position information.

[0084] In the embodiments of the present application, different models are used to respectively identify hair clumps and sparse hair, which is helpful for accurately identifying hair in the image to be processed and improving the accuracy of recognition.

[0085] In one embodiment of the present application, based on the above technical solution, the above steps of performing hair recognition on the plurality of second images to be processed by a second hair recognition model to obtain second position information of hair clumps and sparse hair in the cleaned images specifically include the following steps:

[0086] Performing image division on each second image to be processed by using a second hair recognition model to obtain a plurality of sub-images;

[0087] Using a second hair recognition model, identifying each second image to be processed and the corresponding multiple sub-images, determining a second image to be processed containing hair and a corresponding hair category, wherein the hair category is used to indicate that the hair in the second image to be processed is a hair clump or sparse hair;

[0088] According to the hair category of the second image to be processed of the hair and the position information in the cleaned image, the second image to be processed is merged to obtain the second position information of the hair clumps and sparse hair in the cleaned image.

[0089] In this embodiment, the control device divides the second image to be processed before processing. The control device first divides each second image to be processed by a second hair recognition model to obtain a plurality of sub-images, and then identifies each second image to be processed and the corresponding plurality of sub-images by the second hair recognition model to determine the second image to be processed containing hair and the corresponding hair category. The hair category includes hair clumps or sparse hair. Finally, the control device merges the second image to be processed according to the hair category of the second image to be processed and the position information in the cleared image, to obtain the second position information of the hair clumps and sparse hair in the cleared image. Specifically, for ease of introduction, please refer to Figure 5 , Figure 5 FIG. 2 is a schematic diagram of the structure of the second hair recognition model in the embodiment of the present application. Figure 5 As shown, the second hair recognition model will fuse the global region and local region image depth features. Specifically, the input second image to be processed will first be segmented, for example, by dividing it into sub-region images through 2x2 equal division, 4x4 equal division or 3x3, 6x6 division, and the underlying visual features will be extracted through convolution operations. Subsequently, the divided sub-region images and the global and local regions (or feature maps) of the second image to be processed are taken as separate samples and enter the shared feature extraction backbone (such as MobileNet, ResNet, etc.) to avoid having a separate backbone network for each region, thereby reducing the model size and facilitating deployment. After the feature vectors of each region are concatenated, they are input into the fully connected feature layer to reduce the dimension of the feature vector and remove redundant and invalid feature information. Finally, the sub-region hair recognition is performed to obtain the recognition information of the sub-region image. The results of multiple second images to be processed are combined to obtain the overall result of hair recognition in the image to be processed. For details, please refer to Figure 6 , Figure 6 FIG. 4 is a schematic flow chart of a hair recognition process using a second hair recognition model. Figure 6As shown, in step 601, the control device will intercept the lower half of the cleaned image, and then in step 602, the sub-region images with overlapping parts are obtained from the image of the lower half as detection frames, and then in step 603, the second image to be processed is used for recognition, so as to obtain the classification results of each sub-region image in step 604, including the category and confidence of each sub-region image, and the confidence is used to indicate the probability that the sub-region image contains hair. According to the classification results of each sub-region image, the control device summarizes the identification information of the hair category in step 605, and then in step 606, the overlapping redundant detection frames are removed by maximum value suppression and other methods, and finally in step 607, the hair detection result of the lower half of the image is determined.

[0090] In one embodiment, in the above step, before performing hair recognition on the plurality of second images to be processed by using a second hair recognition model to obtain second position information of hair clumps and sparse hair in the cleaned image, the following step is also included:

[0091] Performing image division on each sample image in a sample image set to obtain a sample sub-image, wherein the sample image set includes the sample image, the hair category and position information of the hair in the sample image;

[0092] For each sub-sample image, whether the sub-sample image contains hair clumps or sparse hair is marked according to the hair category and position information of the hair in the sample image to which it belongs, so as to obtain a sample label of each sub-sample image and obtain a training sample set;

[0093] Performing sample amplification on the sample sub-images containing hair in the training sample set to obtain an amplified training sample set;

[0094] The recognition model to be trained is trained according to the amplified training sample set to obtain a second hair recognition model.

[0095] Specifically, since sparse hair is relatively small, its visual features are not obvious in the image and it occupies a small proportion in the image, the accuracy of hair recognition based on the whole image is poor. In this embodiment, sparse hair recognition is performed based on the idea of ​​image sub-regions, thereby improving the visual significance of sparse hair. For example, the lower half of the image is divided into NxN grids to obtain a set of sample sub-images. There may be a certain overlap between the sample sub-images. Subsequently, the sample sub-images are annotated with or without hair, thereby obtaining a training sample set. Considering that a hair group is too small and may be missed by the recognition model, this category is added to the hair recognition algorithm based on the sample sub-image to avoid missing the hair group and also improve the accuracy of hair recognition. The hair annotation categories are divided into three categories: sparse hair, hair group, and no hair. In one embodiment, the no hair category can be further subdivided to avoid misidentification of hair. The annotated sample sub-images are amplified, for example, the sample sub-images containing hair are rotated, flipped (left and right, up and down), and the brightness is adjusted, thereby increasing the number of sample sub-images containing hair. Finally, the amplified sample training sample set is used to train the recognition model to be trained to obtain a second hair recognition model.

[0096] In one embodiment, in the above step, before the hair clumps are identified in the first image to be processed by using the first hair recognition model to obtain the first position information of the hair clumps in the cleaned image, the following steps are also included:

[0097] Acquire a hair clump image and mark the hair clump position in the hair clump image to obtain a sample image set;

[0098] For each sample image in the sample image set, the cleaned image is cropped according to a preset cropping rule to obtain a training sample set;

[0099] The model to be trained is trained according to the training sample set to obtain a first hair recognition model.

[0100] In one embodiment of the present application, based on the above technical solution, the above steps generate the image to be processed according to the cleaning image collected when the cleaning robot is running, and specifically include the following steps:

[0101] Acquire cleaning images collected when the cleaning robot is running;

[0102] According to a preset cropping rule, the cleaned image is cropped to obtain an image to be recognized at a first scale;

[0103] Performing image segmentation on the image to be identified at the first scale to obtain an image to be identified at a second scale, wherein the second scale is smaller than the first scale;

[0104] The image to be recognized at the first scale and the image to be recognized at the second scale are determined as images to be processed.

[0105] In an embodiment of the present application, the control device divides the cleaning image into images of multiple scales. Specifically, the control device obtains the cleaning image collected when the cleaning robot is running, and then crops the collected cleaning image according to a preset cropping rule to obtain an image to be identified at a first scale. Usually, the image to be identified at the first scale is the lower half of the cleaning image, that is, the area containing the ground image. Subsequently, the control device divides the image to be identified at the first scale to obtain an image to be identified at a second scale, and the second scale is smaller than the first scale. Usually, when the division is performed, there will be a certain overlap between the images to be identified at the second scale. Finally, the image to be identified at the first scale and the image to be identified at the second scale are determined together as the image to be processed. For details, please refer to Figure 7 , Figure 7 Schematic diagram of the process of generating an image to be processed in the embodiment of the present application. Figure 7 As shown, the cleaned image is segmented into a first-scale image 701 to be identified according to a predetermined area, including hair and hair clumps to be identified. Subsequently, the first-scale image 701 to be identified is segmented to obtain a plurality of second-scale images 702 to be identified.

[0106] In one embodiment of the present application, based on the above technical solution, the above step S430, according to the hair position information, controls the cleaning robot to retreat toward the hair and pass through the hair to clean the hair in the cleaning image, specifically including the following steps:

[0107] According to the hair position information of the hair in the image to be processed, controlling the cleaning robot to rotate and move and continuously collect images so that the hair is located at a predetermined position in the collected image;

[0108] According to a preset angle parameter, the cleaning robot is controlled to rotate so that the hair is located in the backward direction of the cleaning robot;

[0109] The cleaning robot is controlled to retreat and perform cleaning to clean the hair in the image to be processed.

[0110] In this embodiment, the control device controls the cleaning robot to clean in a backward manner according to the hair position information of the hair, so as to prevent the hair from being caught in the side sweeper, universal wheel and driving wheel. Figure 8 , Figure 8The schematic process of the cleaning robot cleaning hair in the embodiment of the present application. Specifically, the control device controls the cleaning robot to rotate and move and continuously collect images according to the hair position information of the hair in the image to be processed, so that the hair is located at a predetermined position in the collected image. Figure 8 As shown in step 801 and step 803 in the above, the cleaning robot first rotates to align the hair with the center line of the cleaning robot, and then moves forward until the hair is located near the lower edge of the captured image. Then, the control device controls the cleaning robot to rotate according to the preset angle parameters so that the hair is located in the backward direction of the cleaning robot. Specifically, Figure 8 As shown in step 804 in , the cleaning robot will rotate 180 degrees so that the hair is located in the direction in which the cleaning robot retreats. It is understandable that, depending on the alignment position, the rotation angle may be other angles, such as 90 degrees or 150 degrees, and the specific degree of rotation may also be calculated according to the actual position and alignment method. For example, the position at which the cleaning robot aligns the hair may not be the center line but may be offset by a certain angle, and the degree of rotation may be calculated based on the offset angle. The alignment angle is 30 degrees offset from the center line. For example, the degree of rotation is subtracted from 180 degrees by 30 degrees. Then, the control device controls the cleaning robot to retreat and clean to clean the hair in the image to be processed. Specifically, as Figure 8 As shown in step 805, the cleaning robot retreats and cleans, thereby cleaning the hair through a middle sweep.

[0111] In the embodiment of the present application, by controlling the cleaning robot to move backward to clean the identified hair, it is possible to prevent the hair from being caught in the rotating structure of the side sweeper, universal wheel, driving wheel and other components arranged in the forward direction of the cleaning robot during cleaning, thereby preventing the components from being deformed or rotating unsmoothly, thereby improving the cleaning efficiency.

[0112] In one embodiment of the present application, based on the above technical solution, the above steps, according to the hair position information of the hair in the image to be processed, control the cleaning robot to rotate and move and continuously collect images so that the hair is located at a predetermined position in the collected image, specifically include the following steps:

[0113] According to the hair position information, controlling the cleaning robot to rotate so that the hair is located on the center line of the captured image;

[0114] The cleaning robot is controlled to move toward the hair so that the hair is located at the bottom edge of the captured image.

[0115] In one embodiment of the present application, based on the above technical solution, according to a preset angle parameter, the cleaning robot is controlled to rotate so that the hair is located in the backward direction of the cleaning robot, which specifically includes the following steps:

[0116] Controlling the cleaning robot to rotate 180 degrees so that the hair is located directly behind the cleaning robot;

[0117] The above step of controlling the cleaning robot to retreat and clean the hair in the image to be processed specifically includes the following steps:

[0118] The cleaning robot is controlled to retreat along the central axis to pass through the hair and clean the hair.

[0119] Specifically, the cleaning robot will control the cleaning robot to rotate according to the relative relationship between the hair position of the hair position information and the centerline position of the image acquisition frame used to acquire the image, so that the hair is located on the centerline of the acquired image. For example, when the hair is located on the right side of the centerline in the image to be identified, the control device will control the cleaning robot to turn right until the hair is located on the centerline of the image in the acquired image. At this time, in the actual space, the hair will be located directly in front of the cleaning robot. Figure 8 For example, after the self-rotation, the central axis of the cleaning robot in the direction of travel will be aligned with the direction of the hair. Subsequently, the control device will control the cleaning robot to move forward, that is, move in the direction of the hair, and continue to detect the position of the hair in the captured image until the hair is already at the bottom of the image in the captured image. It can be understood that the bottom of the image capture area of ​​the cleaning robot and the distance between the cleaning robot and the hair are pre-set, so that when the hair is already at the bottom of the captured image, there is still enough distance between the cleaning robot and the hair, and the hair will not be caught in any part of the cleaning robot.

[0120] Then, the control device controls the cleaning robot to rotate 180 degrees so that the hair is located directly behind the cleaning robot. Specifically, since the cleaning robot has been facing the hair before, the 180-degree rotation will cause the hair to be located directly behind the cleaning robot, that is, the central axis of the cleaning robot in the backward direction will be aligned with the direction of the hair.

[0121] Afterwards, the control device will control the cleaning robot to retreat along the central axis to pass through the hair and clean the hair. Specifically, in the previous process, when the robot places the hair at the bottom of the captured image, the distance between the cleaning robot and the hair is set according to parameters such as the angle of the camera that captures the image. Therefore, after the self-rotation, when the cleaning robot is pointing the rear at the hair, the distance between it and the hair is also known. Based on the known distance and parameters such as the size and middle sweep position of the cleaning robot itself, the distance that the middle sweep of the cleaning robot needs to retreat to clean the hair can be determined. Therefore, the cleaning robot can be controlled to retreat according to the distance so that the middle sweep of the cleaning robot passes through the hair, thereby using the middle sweep to clean the hair.

[0122] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0123] The following introduces the device implementation of the present application, which can be used to execute the cleaning robot control method in the above-mentioned embodiment of the present application. Fig. 9 The following schematically shows a block diagram of the cleaning robot control device in an embodiment of the present application. Fig. 9 As shown, the cleaning robot control device 900 may mainly include:

[0124] An image generation module 910 is used to generate an image to be processed based on the cleaning images collected when the cleaning robot is running;

[0125] A hair recognition module 920 is used to recognize the hair in the image to be processed and obtain the hair position information of the hair in the image to be processed;

[0126] The cleaning control module 930 is used to control the cleaning robot to retreat toward the hair and pass through the hair according to the hair position information, so as to clean the hair in the cleaning image.

[0127] In some embodiments of the present application, based on the above technical solution, the image to be processed includes a first image to be processed at a first scale and a plurality of second images to be processed at a second scale, and the second images to be processed are obtained by dividing the cleaned image; the hair recognition module 920 includes:

[0128] A first recognition unit, configured to recognize hair clumps on the first image to be processed by using a first hair recognition model, and obtain first position information of the hair clumps in the cleaned image;

[0129] A second recognition unit, configured to perform hair recognition on the plurality of second images to be processed by using a second hair recognition model, and obtain second position information of hair clumps and sparse hair in the cleaned images;

[0130] A position determination unit is used to determine hair position information according to the first position information and the second position information.

[0131] In some embodiments of the present application, based on the above technical solution, the second identification unit includes

[0132] An image division subunit, configured to divide each second image to be processed into multiple sub-images by using a second hair recognition model;

[0133] a category determination subunit, configured to identify each second image to be processed and the corresponding plurality of sub-images by using a second hair recognition model, and determine a second image to be processed containing hair and a corresponding hair category, wherein the hair category is used to indicate that the hair in the second image to be processed is a hair mass or sparse hair;

[0134] The position determination subunit is used to merge the second image to be processed according to the hair category of the second image to be processed and the position information in the cleaned image, so as to obtain the second position information of the hair clumps and sparse hair in the cleaned image.

[0135] In some embodiments of the present application, based on the above technical solution, the cleaning robot control device 900 further includes:

[0136] A sample module, used for performing image segmentation on each sample image in a sample image set to obtain a sample sub-image, wherein the sample image set includes the sample image, the hair category and position information of the hair in the sample image;

[0137] A first labeling module is used to label each sub-sample image, according to the hair category and position information of the hair in the sample image to which it belongs, whether the sub-sample image contains hair clumps or sparse hair, to obtain a sample label of each sub-sample image, and to obtain a training sample set;

[0138] an amplification module, configured to amplify the sample sub-images containing hair in the training sample set to obtain an amplified training sample set;

[0139] The first training module is used to train the recognition model to be trained according to the amplified training sample set to obtain a second hair recognition model.

[0140] In some embodiments of the present application, based on the above technical solution, the cleaning robot control device 900 further includes:

[0141] A second labeling module is used to obtain a hair clump image and label the hair clump position in the hair clump image to obtain a sample image set;

[0142] A cropping module, configured to crop the cleaned image according to a preset cropping rule for each sample image in the sample image set to obtain a training sample set;

[0143] The second training module is used to train the model to be trained according to the training sample set to obtain a first hair recognition model.

[0144] In some embodiments of the present application, based on the above technical solution, the image generation module 910 includes:

[0145] An image acquisition unit, used to acquire cleaning images collected when the cleaning robot is running;

[0146] An image cropping unit, used for cropping the cleaned image according to a preset cropping rule to obtain an image to be recognized at a first scale;

[0147] an image division unit, configured to divide the image to be identified at the first scale to obtain the image to be identified at a second scale, wherein the second scale is smaller than the first scale;

[0148] The image determination unit is used to determine the image to be recognized at the first scale and the image to be recognized at the second scale as images to be processed.

[0149] In some embodiments of the present application, based on the above technical solution, the cleaning control module 930 includes:

[0150] A mobile control unit, used for controlling the cleaning robot to rotate and move and continuously collect images according to the hair position information of the hair in the image to be processed, so that the hair is located at a predetermined position in the collected image;

[0151] A rotation control unit, used for controlling the cleaning robot to rotate according to a preset angle parameter so that the hair is located in the backward direction of the cleaning robot;

[0152] A cleaning control unit is used to control the cleaning robot to retreat and perform cleaning so as to clean the hair in the image to be processed.

[0153] In some embodiments of the present application, based on the above technical solution, the mobile control unit includes:

[0154] A first rotation control subunit, used for controlling the cleaning robot to rotate according to the hair position information so that the hair is located on the center line of the collected image;

[0155] The first movement control subunit is used to control the cleaning robot to move toward the hair so that the hair is located at the bottom edge of the captured image.

[0156] In some embodiments of the present application, based on the above technical solution, the rotation control unit includes:

[0157] a second rotation control subunit, used for controlling the cleaning robot to rotate 180 degrees so that the hair is located directly behind the cleaning robot;

[0158] The cleaning control unit includes:

[0159] The retreat control subunit is used to control the cleaning robot to retreat along the central axis to pass through the hair and clean the hair.

[0160] It should be noted that the apparatus provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0161] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0162] It should be noted that Fig.10 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0163] like Fig.10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003. Various programs and data required for system operation are also stored in the RAM 1003. The CPU 1001, ROM 1002 and RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0164] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0165] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 1009, and / or installed from a removable medium 1011. When the computer program is executed by a central processing unit (CPU) 1001, various functions defined in the system of the present application are executed.

[0166] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0167] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0168] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0169] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0170] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0171] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A cleaning robot control method, characterized in that: include: Generate an image to be processed according to the cleaning images collected when the cleaning robot is running; Identifying the hair in the image to be processed to obtain hair position information of the hair in the image to be processed; According to the hair position information, controlling the cleaning robot to rotate so that the hair is located on the center line of the captured image; Controlling the cleaning robot to move toward the hair so that the hair is located at the bottom edge of the captured image; According to a preset angle parameter, the cleaning robot is controlled to rotate so that the hair is located in the backward direction of the cleaning robot; The cleaning robot is controlled to retreat and perform cleaning to clean the hair in the image to be processed.

2. The method according to claim 1, characterized in that The to-be-processed images include a first to-be-processed image of a first scale and a plurality of second to-be-processed images of a second scale, wherein the second to-be-processed images are obtained by dividing the cleaned images; Identifying the hair in the image to be processed to obtain hair position information of the hair in the image to be processed includes: Performing hair clump recognition on the first image to be processed by using a first hair recognition model to obtain first position information of the hair clump in the cleaned image; Performing hair recognition on the plurality of second images to be processed by using a second hair recognition model to obtain second position information of hair clumps and sparse hair in the cleaned images; Hair position information is determined according to the first position information and the second position information.

3. The method according to claim 2, characterized in that The method of performing hair recognition on the plurality of second images to be processed by using a second hair recognition model to obtain second position information of hair clumps and sparse hair in the cleaned images includes: Performing image division on each second image to be processed by using a second hair recognition model to obtain a plurality of sub-images; Using a second hair recognition model, identifying each second image to be processed and the corresponding multiple sub-images, determining a second image to be processed containing hair and a corresponding hair category, wherein the hair category is used to indicate that the hair in the second image to be processed is a hair clump or sparse hair; According to the hair category of the second image to be processed of the hair and the position information in the cleaned image, the second image to be processed is merged to obtain the second position information of the hair clumps and sparse hair in the cleaned image.

4. The method according to claim 2, characterized in that: Before performing hair recognition on the plurality of second images to be processed by using a second hair recognition model to obtain second position information of hair clumps and sparse hair in the cleaned images, the method further comprises: Performing image division on each sample image in a sample image set to obtain a sample sub-image, wherein the sample image set includes the sample image, the hair category and position information of the hair in the sample image; For each sub-sample image, whether the sub-sample image contains hair clumps or sparse hair is marked according to the hair category and position information of the hair in the sample image to which it belongs, so as to obtain a sample label of each sub-sample image and obtain a training sample set; Performing sample amplification on the sample sub-images containing hair in the training sample set to obtain an amplified training sample set; The recognition model to be trained is trained according to the amplified training sample set to obtain a second hair recognition model.

5. The method according to claim 2, characterized in that: Before the first image to be processed is subjected to hair clump identification using the first hair identification model to obtain first position information of the hair clump in the cleaned image, the method further includes: Acquire a hair clump image and mark the hair clump position in the hair clump image to obtain a sample image set; For each sample image in the sample image set, the cleaned image is cropped according to a preset cropping rule to obtain a training sample set; The model to be trained is trained according to the training sample set to obtain a first hair recognition model.

6. The method according to claim 2, characterized in that The step of generating an image to be processed according to the cleaning image collected when the cleaning robot is running includes: Acquire cleaning images collected when the cleaning robot is running; According to a preset cropping rule, the cleaned image is cropped to obtain an image to be recognized at a first scale; Performing image segmentation on the image to be identified at the first scale to obtain an image to be identified at a second scale, wherein the second scale is smaller than the first scale; The image to be recognized at the first scale and the image to be recognized at the second scale are determined as images to be processed.

7. The method according to claim 1, characterized in that The controlling the cleaning robot to rotate according to a preset angle parameter so that the hair is located in the backward direction of the cleaning robot comprises: Controlling the cleaning robot to rotate 180 degrees so that the hair is located directly behind the cleaning robot; The controlling the cleaning robot to retreat and clean the hair in the image to be processed includes: The cleaning robot is controlled to retreat along the central axis to pass through the hair and clean the hair.

8. A cleaning robot control device, characterized in that: include: An image generation module is used to generate an image to be processed based on the cleaning images collected when the cleaning robot is running; A hair recognition module, used for recognizing hair in the image to be processed and obtaining hair position information of the hair in the image to be processed; A first rotation control subunit, used for controlling the cleaning robot to rotate according to the hair position information so that the hair is located on the center line of the collected image; a first movement control subunit, used for controlling the cleaning robot to move toward the hair so that the hair is located at the bottom edge of the captured image; A rotation control unit, used for controlling the cleaning robot to rotate according to a preset angle parameter so that the hair is located in the backward direction of the cleaning robot; A cleaning control unit is used to control the cleaning robot to retreat and perform cleaning so as to clean the hair in the image to be processed.

9. A cleaning robot, characterized in that: include: processor; A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the cleaning robot control method according to any one of claims 1 to 7 by executing the executable instructions.

Images