Construction method and device, recognition method and device, computer equipment and cleaning robot
By constructing and applying room classification models, the existing cleaning robots have solved the problems of low cleaning efficiency and poor accuracy in different apartment types, achieving more efficient cleaning and personalized cleaning strategies, and improving user experience.
Patent Information
- Application Number
- CN202310791324.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-30
- Publication Date
- 2024-11-12
AI Technical Summary
Existing cleaning robots adopt indiscriminate cleaning strategies when cleaning rooms in different types of rooms, resulting in low cleaning efficiency, waste of energy and poor user experience, and the accuracy of room classification is affected by users' habits of placing items and detection algorithms for specific target objects.
Build a room classification model, generate a training sample set by obtaining and preprocessing the original data of room types for different apartment types, and training it with preset models such as MobileNetv2, to generate a room classification model that can be used to clean the robot to quickly identify room types And develop appropriate cleaning strategies.
It improves the cleaning efficiency and user experience of cleaning robots, enhances the accuracy of room type recognition, and can distinguish and formulate cleaning strategies based on more features in the environment, reducing dependence on specific target objects.
Smart Images

Figure CN118941933A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent cleaning robots, and in particular to a method for constructing a room classification model, an identification method and device thereof, computer equipment, a cleaning robot and a computer-readable storage medium. Background Art
[0002] With the development of smart home, people are choosing cleaning robots to complete their home cleaning tasks, which puts forward higher requirements on the adaptability of cleaning robots in different types of houses. In the process of cleaning robots completing cleaning tasks in different types of rooms, the indiscriminate cleaning strategy leads to low cleaning efficiency, energy waste and poor user experience. Summary of the invention
[0003] The present application provides a method for constructing a room classification model, a method for identifying a room category, a construction device, an identification device, a cleaning robot, and a computer-readable storage medium.
[0004] The method for constructing the room classification model of the present application includes:
[0005] Obtaining pre-collected raw data of room types of multiple different apartment types;
[0006] Processing the original data according to preset rules to obtain a training sample set, wherein the training sample set includes data samples of multiple room types, and the data samples include original data of multiple room types;
[0007] The training sample set is trained using a preset model to generate the room classification model.
[0008] In this way, in this application, the original data of room types of different apartment types can be obtained, and the original data can be processed according to certain rules to obtain a training sample set. The data samples in the training sample set include a variety of original data of room types that can be manually labeled based on room types. Finally, by training the training sample set, a room classification model is generated to distinguish the current room type. The method for constructing the room classification model of the present application can be trained based on the original data captured by the camera of the cleaning robot to obtain a room classification model, so that the cleaning robot can quickly distinguish the current room type based on more possible features in the environment, and formulate appropriate cleaning strategies according to the room type, which is conducive to improving the cleaning efficiency of the sweeper and improving the user experience.
[0009] The processing of the original data according to the preset rules to obtain the training sample set includes:
[0010] Performing data cleaning on the raw data according to the preset rules to obtain a data set;
[0011] The data set is divided according to a preset ratio to obtain the training sample set.
[0012] In this way, the data set obtained by cleaning the original data according to certain rules can be divided into a certain proportion to obtain training samples for subsequent model training.
[0013] The step of dividing the data set according to a preset ratio to obtain the training sample set includes:
[0014] Classifying the cleaned data set according to room type, so as to divide the data belonging to the same room type into the same sample subset;
[0015] Each sample subset is divided according to a preset ratio to obtain a plurality of training sample subsets, and the plurality of training sample subsets constitute the training sample set.
[0016] In this way, the cleaned data set can be classified according to the room type, and the data belonging to the same room type can be divided according to a preset ratio to obtain a training sample set consisting of multiple sample subsets to serve as training data to generate a room classification model.
[0017] The step of training the training sample set by using a preset model to generate the room classification model includes:
[0018] Performing data preprocessing on the training sample set;
[0019] The room classification model is generated by training the training sample set after data preprocessing through a preset model.
[0020] In this way, the preprocessed training sample set can be trained through the preset model to generate a room classification model so that the cleaning robot has the ability to identify the room type when working.
[0021] The step of training the training sample set by using a preset model to generate the room classification model includes:
[0022] The training sample set is trained by using a MobileNetv2 network training model to generate the room classification model.
[0023] In this way, the training sample set can be trained through the MobileNetv2 network training model to generate a room classification model so that the cleaning robot has the ability to identify the room type when working.
[0024] The method of training the training sample set after data preprocessing by using a preset model to generate the room classification model includes:
[0025] The training sample set after data preprocessing is trained by the preset model to obtain the initial model;
[0026] The initial model is subjected to model conversion processing to generate the room classification model that can be deployed and run on the cleaning robot.
[0027] In this way, the preset model can be stored to obtain the initial model, and a room classification model that can be run in the cleaning robot is generated based on the initial model to distinguish the current room type and formulate a suitable cleaning strategy.
[0028] The room category identification method of the present application includes:
[0029] When the cleaning robot performs a cleaning task for the first time and establishes a spatial map, image information of the current space is obtained;
[0030] Determining the room type of the current room according to the pre-established room classification model and the image information;
[0031] The constructed space map is labeled with room types according to the room types.
[0032] In this way, when the cleaning robot performs the cleaning task for the first time and establishes the space map, the room type can be determined and marked on the map, avoiding the influence of different object layouts on the room classification results.
[0033] The determining the type of the current room according to the pre-established room classification model and the image information includes:
[0034] Calculating the room type probability of the current room according to the pre-established room classification model and current frame image information;
[0035] The room type corresponding to the probability value that is greater than a preset threshold and has the largest value in the calculation result is determined as the type of the current room.
[0036] In this way, calculations can be performed based on the room classification model and the current frame image information, and the room type of the current room can be determined based on whether the calculation results meet certain standards, thereby improving the accuracy of room type judgment.
[0037] The method further comprises:
[0038] When the calculation results are all smaller than the preset threshold, the current frame image is discarded.
[0039] In this way, calculations can be performed based on the room classification model and the current frame image information, and the current frame image whose calculation results do not meet the standards will be discarded, thereby improving the accuracy of room type judgment.
[0040] The determining the type of the current room according to the pre-established room classification model and the image information includes:
[0041] After completing the construction of the space map of the current room, the type of the current room is determined according to the statistical results of the room types of the current room determined in the process of building the map.
[0042] In this way, the current room type can be determined based on the statistical results of the construction of the room type, thereby improving the operating efficiency of the cleaning robot.
[0043] The room classification model construction device of the present application includes:
[0044] An acquisition module, used to acquire pre-collected raw data of room types of multiple different apartment types;
[0045] A processing module, configured to process the raw data according to a preset rule to obtain a training sample set, wherein the training sample set includes data samples of multiple room types, and the data samples include raw data of multiple room types that can be manually labeled by room type;
[0046] The training module is used to train the training sample set through a preset model to generate the room classification model.
[0047] The room category identification device of the present application includes:
[0048] An acquisition module is used to acquire image information of the current space when the cleaning robot performs a cleaning task for the first time to establish a space map;
[0049] A confirmation module, used to determine the room type of the current room according to the pre-established room classification model and the image information;
[0050] a labeling module, configured to label the constructed space map with room types according to the room types; and detect user behavior in the vehicle cabin;
[0051] The computer device of the present application includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method for constructing the room classification model is implemented.
[0052] The cleaning robot of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the above-mentioned room category recognition method is implemented.
[0053] The computer-readable storage medium of the present application stores a computer program, and when the computer program is executed by one or more processors, the method for constructing the room classification model and the method for identifying the room category are implemented.
[0054] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0056] Figure 1 It is a flowchart of the method for constructing the room classification model of the present application;
[0057] Figure 2 It is a module diagram of the method for constructing the room classification model of the present application;
[0058] Figure 3 It is a flowchart of the method for constructing the room classification model of the present application;
[0059] Figure 4 It is a flowchart of the method for constructing the room classification model of the present application;
[0060] Figure 5 is a schematic diagram of a training sample set of a room classification model of the present application;
[0061] Figure 6 It is a flowchart of the method for constructing the room classification model of the present application;
[0062] Figure 7 It is a flowchart of the method for constructing the room classification model of the present application;
[0063] Figure 8 It is a flowchart of the method for constructing the room classification model of the present application;
[0064] Fig. 9 It is a flowchart of the method for constructing the room classification model of the present application;
[0065] Fig.10 It is a flowchart of the method for identifying the room category of the present application;
[0066] Fig.11 It is a module diagram of the room category identification method of the present application;
[0067] Fig.12 It is a flowchart of the method for identifying the room category of the present application;
[0068] Fig.13 It is a flowchart of the method for identifying the room category of the present application;
[0069] Fig.14 It is a flowchart of the method for identifying the room category of the present application. DETAILED DESCRIPTION
[0070] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.
[0071] Artificial intelligence technology has promoted the development of the smart home field. More and more families choose to use cleaning robots, which puts higher requirements on the adaptability of cleaning robots in different house types. At present, most cleaning robots are based on lidar or visual sensors to build a whole house map. After the whole house map is built, the room area can be divided to realize the whole house cleaning and partition cleaning functions. In the related technology, the room classification method of the cleaning robot is mainly based on the detection or segmentation algorithm of specific target objects in different rooms, which has high requirements on the computing power of the cleaning robot, but it is actually difficult to meet. In addition, the accuracy of the cleaning robot's room classification is also affected by the user's habit of placing items.
[0072] Based on the above problems that may be encountered, please refer to Figure 1 , the method for constructing the room classification model of the present application includes:
[0073] 01: Obtain the pre-collected raw data of multiple room types of different apartment types;
[0074] 02: Process the original data according to the preset rules to obtain the training sample set;
[0075] 03: Train the training sample set using the preset model to generate a room classification model.
[0076] See also Figure 2, the present application also provides a device 100 for constructing a room classification model. The method for constructing a room classification model of the present application can be implemented by the device 100 for constructing a room classification model of the present application. Specifically, the device 100 for constructing a room classification model includes an acquisition module 101, a processing module 102 and a training module 103. Among them, the acquisition module 101 is used to obtain the original data of room types of multiple different apartment types collected in advance. The processing module 102 is used to process the original data according to preset rules to obtain a training sample set, and the training sample set includes data samples of multiple room types, and the data samples include original data of multiple room types that can be manually labeled based on room types. The training module 103 is used to train the training sample set through a preset model to generate a room classification model.
[0077] The present application also provides a computer device, which includes a memory and a processor. The method for constructing a room classification model of the present application can be implemented by the computer device of the present application. Specifically, a computer program is stored in the memory, and the processor is used to obtain pre-collected raw data of room types of multiple different apartment types, and to process the raw data according to preset rules to obtain a training sample set, and to train the training sample set through a preset model to generate a room classification model.
[0078] Specifically, the method for constructing the room classification model proposed in this application is as follows: Figure 3 As shown, first of all, it is necessary to collect the raw data of the room types of multiple different types of apartments. The source of the collected data includes images collected by a cleaning robot specifically for training or two-dimensional images uploaded by other cleaning robots collected through a computer device connected to the Internet, etc., which are not limited here. Among them, the raw data is the image data that has not been manually labeled obtained through various channels. In other words, the raw data is the data basis for subsequent manual labeling. Subsequently, manual labeling can be performed on the basis of the raw image data, and the corresponding room type is marked for the manually distinguishable image data. In order to facilitate image acquisition, a camera can be configured on the cleaning robot, and the camera has a certain height from the ground, for example, about 6 cm, so as to ensure the visual range of the cleaning robot, so as to collect as much valid data as possible. The collected raw data can be an RGB image of a two-dimensional room. Among them, the collected apartment types can include different apartment types such as one-bedroom, two-bedroom, three-bedroom and above. The collected raw data set may include at least two-dimensional images of more than 100 rooms of different types. It can be understood that the more raw image data, the more accurate the subsequent model training results.
[0079] After obtaining the pre-collected raw data of different room types, in order to ensure the accuracy of the room classification results, the acquired data needs to be processed according to the preset rules to obtain a training sample set. For example, the raw data is classified by room type, and the room types of a residence can be roughly divided into multiple types such as living room, bedroom, dining room, kitchen, bathroom, balcony, etc. In addition, in some functional places, such as commercial places, office places, hospitals, schools, etc., the classification of room types may be different. The training sample set may include the raw data of the above multiple room types. The preset rules may also include discarding image data that does not meet any room type. For example, when the cleaning robot captures an outdoor scene or the image does not contain any furniture, the image needs to be discarded and does not participate in the subsequent model training process. The raw data processed according to the preset rules constitutes data samples of multiple room types, thereby constructing a training sample set for training the room classification model. And the training set is divided from the data that can be manually labeled for room type from all the raw data, and these data are further processed to obtain the training sample set. That is, the sample data of multiple room types included in the training sample set, the sample data includes raw data of multiple room types.
[0080] Finally, the training sample set needs to be trained through a preset model. The preset model can select various deep learning models based on the Pytorch deep learning framework. Through the model, transfer learning is performed, and model parameters such as weights are continuously pre-trained, so that the model has the ability to recognize the commonality of images, that is, a room classification model is generated. The room classification model recognizes the commonality of some images in the training sample set, and has the ability to learn autonomously, and can classify images with certain commonalities as images of the same type of room. In other words, the room classification model in the embodiment of the present application has a certain reasoning ability, and can summarize and deduce images of different types of rooms based on the data of the training sample set, autonomously identify and learn the characteristics of different room types, and then when the room type is actually identified, even if there are no certain specific target objects in the acquired image, the room type can be identified. For example, a certain image does not include a bed, but the image can be classified as an image of a bedroom based on other feature information in the image. The establishment of the room classification model enables the cleaning robot to judge the room type when entering a room that has not been entered to perform a cleaning task, adopt appropriate cleaning strategies, improve cleaning efficiency, and facilitate users to customize personalized cleaning strategies.
[0081] It should be noted that in the implementation of the present application, room annotation is no longer performed only for frame images including specific target objects, but as long as frame images that can identify the room type based on artificial means are annotated. For example, in the related art, for a frame image, the type of the room will be marked as a bedroom only if and only if there are target objects such as a bed and a wardrobe in it. However, in actual situations, based on factors such as the placement habits of different users, the apartment type, or the shooting angle of the cleaning robot, the target object may not exist in the acquired image. In this case, based on the method of the related art, the room type will not be quickly identified. It may be necessary to identify the room type based on the second frame, the third frame, etc. images acquired later. It may even be impossible to identify the room type. In other words, for the related art, when the room classification is performed later, a certain room type can only be confirmed when the target object is identified in the spatial environment, or in other words, the recognition of the room type in the related art is based on a specific target object, and the recognition will fail if the target object does not exist in the image.
[0082] In the implementation mode of the present application, for all the acquired raw data, in the manual labeling process, as long as the frame images that can manually identify the room type are labeled, it is not limited to labeling images that include target objects. For example, for a frame image of the current room, which contains bedside tables, dressing tables, etc., which can be confirmed as features of a bedroom based on common sense, although the frame image does not include target objects such as beds and wardrobes, it can still be confirmed or confirmed with a high probability that the room type of the room is a bedroom. On this basis, according to the features in the images labeled as the same room type, further induction and deduction are performed. In this way, in the actual application process, it is not necessary to be limited to specific target objects in the room to identify the room type, and the recognition efficiency of the room type is improved as much as possible.
[0083] In summary, in this application, the original data of room types of different apartment types can be obtained, and the original data can be processed according to certain rules to obtain a training sample set. The data samples in the training sample set include a variety of original data of room types that can be manually labeled based on room types. Finally, by training the training sample set, a room classification model is generated to distinguish the current room type. The method for constructing the room classification model of the present application can be trained based on the original data captured by the camera of the cleaning robot to obtain a room classification model, so that the cleaning robot can quickly distinguish the current room type based on more possible features in the environment, and formulate appropriate cleaning strategies according to the room type, which is conducive to improving the cleaning efficiency of the sweeper and improving the user experience.
[0084] In certain embodiments, see Figure 3 , step 02 includes:
[0085] 021: Perform data cleaning on the original data according to preset rules to obtain a data set;
[0086] 022: Divide the data set according to the preset ratio to obtain the training sample set.
[0087] In some embodiments, the processing module 102 is used to perform data cleaning processing on the original data according to preset rules to obtain a data set, and to divide the data set according to a preset ratio to obtain a training sample set.
[0088] In some embodiments, the processor is used to perform data cleaning processing on the original data according to preset rules to obtain a data set, and divide the data set according to a preset ratio to obtain a training sample set.
[0089] Specifically, the raw RGB image data of the room collected by the camera configured by the cleaning robot is cleaned to obtain a data set differentiated by room type. The data cleaning process can find and correct identifiable errors in the data file, including checking data consistency, processing invalid values and missing values, etc.
[0090] In some examples, the room type classification method of residential houses can be roughly divided into six categories: living room, bedroom, dining room, kitchen, bathroom, balcony, etc. The room type classification method can be selected according to the actual function of the room type. For example, when the cleaning robot is in a multi-story building such as a commercial place, the room type needs to be distinguished between elevators, escalators and other areas to prevent the cleaning robot from falling due to improper routes, causing safety hazards.
[0091] In order to highlight the characteristics of the above room types, there are preset rules in the data cleaning process. In some examples, the room type classification method of residential houses is as follows: Each room type may include a predetermined number of relatively clear room type features, such as a sofa, coffee table, TV, TV cabinet, carpet, floor lamp, recliner, etc., which can be photographed in the living room at the same time. At least two objects such as the bed, bedside table, complete wardrobe, stool at the end of the bed, and table lamp on the bedside table need to be photographed at the same time. The dining room can be photographed based on dining tables and chairs, chandeliers, candlesticks, vases, etc. The kitchen can be photographed with closets, range hoods, kitchen utensils, kitchen appliances, etc. The bathroom needs to be photographed with toilets, showers, sinks, etc. The balcony needs to be photographed with exterior scenes, railings, potted plants, laundry tables, etc. In addition, for image data such as local scenes that cannot be manually annotated, they should be filtered out as invalid values. It should be noted that the above examples are only for schematic illustration, and the number of feature objects in the preset rules can be adjusted according to actual needs. Feature objects are not limited to the types disclosed in the above examples. As long as the room type can be accurately identified manually based on the feature objects, they fall into this category. It should be explained again that in the embodiments of the present application, in the process of identifying the room type according to the room classification model, it is not limited to the features included in the image marked as a certain room type in the manual labeling process, but on the basis of these features, the image marked as a certain room type can be further learned, and then the room type can be identified based on the features that are not present in the labeling process. That is, in the labeling process, although the room type is manually labeled based on certain features in the image, the machine can learn other features in the image labeled as the room type.
[0092] The data set obtained after data cleaning is divided according to a preset ratio. When the magnitude of the data set is not large, for example, when the data set is in the order of 10,000 to tens of thousands, the data set can be divided into a training set and a validation set at a ratio of 8:2. If the magnitude of the data set is large, for example, when the data set is in the order of millions, a sufficient amount of data can be reserved as a validation set, such as reserving 10,000 data as a validation set. The selection of the ratio of the training set and the validation set can be arbitrarily selected within the range that does not affect the generalization of the model, and is not limited here.
[0093] In this way, the data set obtained by cleaning the original data according to certain rules can be divided into a certain proportion to obtain training samples for subsequent model training.
[0094] In certain embodiments, see Figure 4 , step 022 comprises:
[0095] 0221: Classify the cleaned data set according to the room type to divide the data belonging to the same room type into the same sample subset;
[0096] 0222: Divide each sample subset according to a preset ratio to obtain multiple training sample subsets, and the multiple training sample subsets constitute a training sample set.
[0097] In some embodiments, the processing module 102 is used to classify the cleaned data set according to the room type, so as to divide the data belonging to the same room type into the same sample subset, and divide each sample subset according to a preset ratio to obtain multiple training sample subsets, and multiple training sample subsets constitute a training sample set.
[0098] In some embodiments, the processor is used to classify the cleaned data set according to the room type to divide the data belonging to the same room type into the same sample subset, and divide each sample subset according to a preset ratio to obtain multiple training sample subsets, and the multiple training sample subsets constitute a training sample set.
[0099] Specifically, the data set obtained after data cleaning is classified according to room type, and the data belonging to the same room type is divided into the same sample subset. That is, the images belonging to the same room category after data cleaning are placed in the same level directory, and the folder name to which the directory belongs is the true category of all images in the folder.
[0100] After classifying the data set, each sample subset is divided according to a preset ratio. In order to enhance the generalization of the room type model, the selection of training sets and validation sets should be divided according to the room type, that is, the data of all room types of the same room type should be classified as training sets or validation sets at the same time, and the data of different room types in the same room type cannot be classified into training sets and validation sets respectively. For example, if the data of "balcony" of "room type A" is included in the training set, then the data of all room types such as "bedroom" and "dining room" of "room type A" should also be included in the training set.
[0101] In addition, the selection of the preset ratio does not change according to the room type corresponding to the sample subset. For example, when the ratio of the training set to the validation set of the "balcony" type data is 8:2, the ratio of the training set to the validation set corresponding to the data of all room types such as "bedroom" and "dining room" should also be 8:2.
[0102] In particular, in the data set, the ratio of the training set to the validation set can be arbitrarily selected within the range that does not affect the generalization of the model and is not limited here.
[0103] Finally, the training sample set is composed of multiple training sample subsets divided into training sets and validation sets according to a certain ratio, such as Figure 6 As shown, this is a possible training sample.
[0104] In this way, the cleaned data set can be classified according to the room type, and the data belonging to the same room type can be divided according to a preset ratio to obtain a training sample set consisting of multiple sample subsets to serve as training data to generate a room classification model.
[0105] In certain embodiments, see Figure 7 , step 03 includes:
[0106] 031: Perform data preprocessing on the training sample set;
[0107] 032: Train the training sample set after data preprocessing through the preset model to generate a room classification model.
[0108] In some implementations, the training module 103 is used to perform data preprocessing on the training sample set, and to train the preprocessed training sample set using a preset model to generate a room classification model.
[0109] In some embodiments, the processor is used to perform data preprocessing on the training sample set, and train the preprocessed training sample set using a preset model to generate a room classification model.
[0110] Specifically, in order to adapt the image data to the training model and achieve better training results, the training sample set needs to be preprocessed. First, the images in the training set are uniformly scaled to a size of 256×256×3, that is, the length and width of the image are both 256 pixels, and each pixel contains 3 colors, indicating that the image is an RGB image. At the same time, the validation set images are also uniformly scaled to a size of 256×256×3.
[0111] Furthermore, the training set images and the test set images need to undergo image enhancement and normalization operations to mitigate the loss of image features caused by image scaling.
[0112] After data preprocessing of the training sample set, the processed training samples are trained through the preset model. In some examples, the Pytorch deep learning framework can be used to load the training weights of deep learning models in other fields through transfer learning as the pre-training weights for image recognition in this scheme, so that the preset model has the ability to recognize the common features of images in the training sample set. The optimizer of the above deep learning model can be Adam, and the learning rate decay coefficient is set to 0.5. The number of data (batch_size) passed to the model for training each time is 64, and the number of iterations (epoch) is 200. Each iteration represents a traversal of the training set, and it ends after 200 traversals. Among them, the parameters of the model are updated once for each iteration. Parameters such as model optimizer, learning rate decay coefficient, and number of iterations can be changed according to the specific training sample set.
[0113] After preprocessing the data, the training sample set can be trained through the preset model to generate a room classification model. The preset model is converted on the computer side and finally converted into a storage format that can be deployed in the cleaning robot.
[0114] In addition, in order to further improve the accuracy of the room classification model in the room classification process, it is necessary to further verify the accuracy of the model on the cleaning robot and make appropriate optimizations.
[0115] In this way, the preprocessed training sample set can be trained through the preset model to generate a room classification model so that the cleaning robot has the ability to identify the room type when working.
[0116] In certain embodiments, see Figure 8 , step 03 includes:
[0117] 033: Train the training sample set through the MobileNetv2 network training model to generate a room classification model.
[0118] In some embodiments, the training device 100 is used to train the training sample set using the MobileNetv2 network training model to generate a room classification model.
[0119] In some embodiments, the processor is used to train the training sample set using the MobileNetv2 network training model to generate a room classification model.
[0120] Specifically, the training of the model needs to rely on the computing power of the server. In some examples, the training sample set can be batch loaded into the MobileNetv2 network training model, forward propagation can be performed, and the network loss during the forward propagation process can be calculated. The loss function is used for back propagation and gradient calculation, and all the weights of the neurons are updated to obtain a trained room classification model.
[0121] In this way, the training sample set can be trained through the MobileNetv2 network training model to generate a room classification model so that the cleaning robot has the ability to identify the room type when working.
[0122] In certain embodiments, see Fig. 9 , step 032 comprises:
[0123] 0321: Train the training sample set after data preprocessing through the preset model to obtain the initial model;
[0124] 0322: Perform model conversion on the initial model to generate a room classification model that can be deployed and run on a cleaning robot.
[0125] In some embodiments, the training module 103 is used to train the training sample set after data preprocessing through a preset model to obtain an initial model, and perform model conversion processing on the initial model to generate a room classification model that can be deployed and run on a cleaning robot.
[0126] In some embodiments, the processor is used to train the training sample set after data preprocessing through a preset model to obtain an initial model, and perform model conversion processing on the initial model to generate a room classification model that can be deployed and run on a cleaning robot.
[0127] Specifically, the training samples after data preprocessing can be trained through the preset model to obtain the initial model. The initial model is used to store the trained model so that different deep learning frameworks (such as Pytorch) can store model data in the same format. It is an intermediate expression format for the migration model process. The initial model is converted to generate a room classification model that can directly run the cleaning robot.
[0128] In certain embodiments of the present application, the specific steps for converting and deploying the preset model are as follows: Step 1, convert the Pytorch model into an ONNX model on the computer side where the ONNX and onnxruntime environments are configured. ONNX is an open file format designed for machine learning and is used to store trained Pytorch models. The room classification prediction results of the ONNX model should be consistent with Pytorch. Step 2, convert the ONNX model into an RKNN model that can perform reasoning on the cleaning robot, that is, generate a room classification model. The establishment of the room classification model enables the cleaning robot to judge the room type and adopt appropriate cleaning strategies to improve cleaning efficiency when entering a room that has never been entered to perform a cleaning task.
[0129] In addition, the room classification model proposed in the present invention adopts int8 quantization deployment, which has the advantages of small data volume and fast calculation speed, and is suitable for end-side calculation. The present invention evaluates the total running time and total running memory after model quantization. In one example, the total running time is evaluated to be 17ms and the total running memory is 4.44MB, which meets the requirements for calculation inside the cleaning robot.
[0130] In this way, the preset model can be stored to obtain the initial model, and a room classification model that can be run in the cleaning robot is generated based on the initial model to distinguish the current room type and formulate a suitable cleaning strategy.
[0131] See also Fig.10 , the room category identification method of the present application includes:
[0132] 04: When the cleaning robot performs the cleaning task for the first time and establishes the space map, it obtains the image information of the current space;
[0133] 05: Determine the room type of the current room based on the pre-established room classification model and image information;
[0134] 06: Label the constructed space map with room types according to the room types.
[0135] See also Fig.11, the present application also provides a room category recognition device 200. The room category recognition device of the present application can be implemented by the room category recognition device 200 of the present application. Specifically, the room category recognition device 200 includes an acquisition module 201, a confirmation module 202 and a labeling module 203. Among them, the acquisition module 201 is used to obtain image information of the current space when the cleaning robot performs a cleaning task for the first time to establish a space map. The confirmation module 202 is used to determine the room type of the current room based on a pre-established room classification model and image information. The labeling module 203 is used to label the constructed space map with room type according to the room type.
[0136] The cleaning robot of the present application includes a memory and a processor. The method for constructing a room classification model of the present application can be implemented by the cleaning robot of the present application. Specifically, a computer program is stored in the memory, and the processor is used to obtain image information of the current space during the process of the cleaning robot performing a cleaning task for the first time to establish a space map, and determine the room type of the current room according to the pre-established room classification model and image information, and mark the constructed space map with the room type according to the room type.
[0137] Specifically, when the cleaning robot performs a cleaning task for the first time, it needs to build a space map to obtain image information of the current space. After the cleaning robot departs from the base station, it starts the mapping task, keeps the camera set on it turned on, and obtains room images through the camera.
[0138] After the room image is acquired, the room type of the current room can be determined according to the pre-established room classification model. In some embodiments of the present application, the rooms of the residence can be roughly divided into six categories, namely, living room, bedroom, dining room, kitchen, bathroom, balcony, etc. The construction method of the room classification model can refer to the explanation in the above-mentioned related embodiments, which will not be repeated here.
[0139] After determining the room type, the room type of each room needs to be marked on the constructed space map. For example, the room type can be marked with specific text at the corresponding position on the interactive interface of the terminal application, so that the user can choose and customize the personalized cleaning mode through the terminal application.
[0140] In this way, when the cleaning robot performs the cleaning task for the first time and establishes the space map, the room type can be determined and marked on the map, avoiding the influence of different object layouts on the room classification results.
[0141] In certain embodiments, see Fig.12 , step 05 includes:
[0142] 051: Calculate the room type probability of the current room based on the pre-established room classification model and the current frame image information;
[0143] 052: Determine the room type corresponding to the probability value that is greater than a preset threshold and has the largest value in the calculation result as the type of the current room.
[0144] In some embodiments, the confirmation module 202 is used to calculate the room type probability of the current room based on a pre-established room classification model and current frame image information, and determine the room type corresponding to the probability value greater than a preset threshold and with the largest value in the calculation result as the type of the current room.
[0145] In some embodiments, the processor is used to calculate the room type probability of the current room based on a pre-established room classification model and current frame image information, and determine the room type corresponding to the probability value greater than a preset threshold and with the largest value in the calculation result as the type of the current room.
[0146] Specifically, the training sample set can be trained through the MobileNetv2 network training model to generate a room classification model. To ensure the accuracy of the model, it is necessary to use the trained model to predict and evaluate on new room images to enhance the generalization ability of the model.
[0147] The room type probability of the current room can be calculated based on the image information read from the current frame image, including various features included in the current frame image. For example, when a bed, a bedside table, and a complete wardrobe appear in the current frame image, the room is likely to be a bedroom. At the same time, in some special apartment types, beds, bedside tables and other components may appear in the living room, so there is a certain probability that the current frame image is a living room. Therefore, a certain threshold needs to be set to determine the current room type through the relative relationship between the probability value of the current frame image corresponding to the room type and the threshold.
[0148] In one example, the threshold can be set to 0.8. When the calculated probability that the current room is a bedroom is 0.85 and the probability of being other room types is less than 0.8, the current room can be determined to be a bedroom. If the threshold is set to 0.5, when the calculated probability that the current room is a bedroom is 0.85 and the probability of being a balcony is 0.6, the bedroom with the larger calculated result is used as the judgment result of the current room type.
[0149] In this way, calculations can be performed based on the room classification model and the current frame image information, and the room type of the current room can be determined based on whether the calculation results meet certain standards, thereby improving the accuracy of room type judgment.
[0150] In certain embodiments, see Fig.13 , the method further comprises:
[0151] 07: If all calculation results are less than the preset threshold, the current frame image is discarded.
[0152] In certain embodiments, Fig.11 As shown, the recognition device 200 further includes a storage module 204, and the storage module 204 is used to discard the current frame image when all calculation results are smaller than a preset threshold.
[0153] In some implementations, the processor is configured to discard the current frame image when all calculation results are smaller than a preset threshold.
[0154] Specifically, if the calculation results corresponding to all room types are less than the threshold, it is impossible to determine the specific room location of the cleaning robot when the current frame image is captured. In one example, if the current frame image only contains one atypical object, such as an air purifier, or a living object blocks the camera, causing the camera to be unable to capture a specific object in the room, then the calculation result is likely to be less than the threshold, so it is impossible to determine the room type of the cleaning robot based on the image, and the current frame image needs to be discarded.
[0155] In addition, when the cleaning robot is located at the junction of two types of rooms, image data of only a partial scene may be generated. For example, the current frame image contains both a partial image of a toilet in the bathroom and a partial image of a bed in the bedroom. Since there is no complete object in the current frame image, the calculation result is likely to be less than the threshold, so it is impossible to determine the type of room the cleaning robot is in based on the image, and the current frame image needs to be discarded.
[0156] In this way, calculations can be performed based on the room classification model and the current frame image information, and the current frame image whose calculation results do not meet the standards will be discarded, thereby improving the accuracy of room type judgment.
[0157] In certain embodiments, see Fig.14 , step 05 includes:
[0158] 053: After completing the construction of the space map of the current room, determine the type of the current room according to the statistical results of the room type of the current room determined in the process of map building.
[0159] In some implementations, the confirmation module 202 is used to determine the type of the current room according to the statistical results of the room type of the current room determined in the process of building the map after completing the construction of the spatial map of the current room.
[0160] In some embodiments, the processor is configured to determine the type of the current room according to statistical results of the room type of the current room determined during the map building process after completing the construction of the spatial map of the current room.
[0161] Specifically, after the cleaning robot completes the construction of the space map of the current room, the room name can be marked on the map according to the type of the current room determined during the map building process.
[0162] When the spatial map of the room appears in the terminal application, the "marker" of the room name can be displayed to show the divided rooms and their types. It is convenient for users to choose and customize personalized cleaning modes through the terminal application. When the cleaning robot completes the construction of the spatial map, it needs to update and save the map. For the same apartment type, the cleaning robot does not need to rebuild the map when cleaning again, but directly uses the saved map.
[0163] In this way, the current room type can be determined based on the statistical results of the construction of the room type, thereby improving the operating efficiency of the cleaning robot.
[0164] The present application also provides a computer-readable storage medium storing a computer program, which, when executed by one or more processors, implements the above-mentioned method for identifying room categories.
[0165] In the description of this specification, the descriptions with reference to the terms "specifically", "further", "particularly", etc. are intended to mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not intended to refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.
[0166] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0167] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A method for constructing a room classification model, characterized in that: The method comprises: Obtaining pre-collected raw data of room types of multiple different apartment types; Processing the original data according to preset rules to obtain a training sample set, wherein the training sample set includes data samples of multiple room types, and the data samples include original data of multiple room types; The training sample set is trained using a preset model to generate the room classification model.
2. The method according to claim 1, characterized in that: The processing of the original data according to the preset rules to obtain the training sample set includes: Performing data cleaning on the raw data according to the preset rules to obtain a data set; The data set is divided according to a preset ratio to obtain the training sample set.
3. The method according to claim 2, characterized in that The step of dividing the data set according to a preset ratio to obtain the training sample set includes: Classifying the cleaned data set according to room type, so as to divide the data belonging to the same room type into the same sample subset; Each sample subset is divided according to a preset ratio to obtain a plurality of training sample subsets, and the plurality of training sample subsets constitute the training sample set.
4. The method according to claim 1, characterized in that: The step of training the training sample set by using a preset model to generate the room classification model includes: Performing data preprocessing on the training sample set; The room classification model is generated by training the training sample set after data preprocessing through a preset model.
5. The method according to claim 1, characterized in that The step of training the training sample set by using a preset model to generate the room classification model includes: The training sample set is trained by using a MobileNetv2 network training model to generate the room classification model.
6. The method according to claim 1, characterized in that The method of training the training sample set after data preprocessing by using a preset model to generate the room classification model includes: The training sample set after data preprocessing is trained by the preset model to obtain the initial model; The initial model is subjected to model conversion processing to generate the room classification model that can be deployed and run on the cleaning robot.
7. A method for identifying room categories, based on a room classification model generated by the method for generating a room classification model according to any one of claims 1 to 6, characterized in that: The identification method comprises: When the cleaning robot performs a cleaning task for the first time and establishes a spatial map, image information of the current space is obtained; Determining the room type of the current room according to the pre-established room classification model and the image information; The constructed space map is labeled with room types according to the room types.
8. The method according to claim 7, characterized in that The determining the type of the current room according to the pre-established room classification model and the image information includes: Calculating the room type probability of the current room according to the pre-established room classification model and current frame image information; The room type corresponding to the probability value that is greater than a preset threshold and has the largest value in the calculation result is determined as the type of the current room.
9. The method according to claim 8, characterized in that The method further comprises: When the calculation results are all smaller than the preset threshold, the current frame image is discarded.
10. The method according to claim 8, characterized in that The determining the type of the current room according to the pre-established room classification model and the image information includes: After completing the construction of the space map of the current room, the type of the current room is determined according to the statistical results of the room types of the current room determined in the process of building the map.
11. A device for constructing a room classification model, characterized in that: include: An acquisition module, used to acquire pre-collected raw data of room types of multiple different apartment types; A processing module, configured to process the raw data according to a preset rule to obtain a training sample set, wherein the training sample set includes data samples of multiple room types, and the data samples include raw data of multiple room types that can be manually labeled by room type; The training module is used to train the training sample set through a preset model to generate the room classification model.
12. A room category recognition device, characterized in that: include: An acquisition module is used to acquire image information of the current space when the cleaning robot performs a cleaning task for the first time to establish a space map; A confirmation module, used to determine the room type of the current room according to the pre-established room classification model and the image information; The labeling module is used to label the constructed space map with room types according to the room types.
13. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 6 is implemented.
14. A cleaning robot, characterized in that: The cleaning robot comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method described in any one of claims 7 to 10 is implemented.
15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method according to any one of claims 1 to 10 is implemented.
Citation Information
Patent Citations
Deep learning based robot scene identification and semantic navigation map marking method
CN110455306A
Sweeping method and system based on scene recognition, sweeping equipment and storage medium
CN111012261A
Room window classification model construction method and room window classification method and system
CN111291799A
Cleaning method and device based on scene recognition, cleaning robot and storage medium
CN111568314A
Room attribute identification method and device, sweeping robot and storage medium
CN111973076A
