Identification method, control method, identification device, cleaning equipment and storage medium

Abstract

The invention discloses a cleaning degree identification method for a to-be-cleaned object of cleaning equipment, a control method of the cleaning equipment, a cleaning degree identification device, the cleaning equipment and a computer readable storage medium. The cleaning degree identification method comprises the following steps: acquiring a target image comprising a to-be-cleaned object; a first feature map group and a second feature map group with different scales are extracted from the target image, and the scale of the first feature map group is larger than that of the second feature map group; identifying position information of the to-be-cleaned object based on the second feature map group; obtaining a detection frame of the to-be-cleaned object from the first feature map group based on a mapping relation between the position information and the first feature map group; and identifying the cleaning degree of the to-be-cleaned object from the detection frame of the to-be-cleaned object. Through the scheme, the accuracy of the obtained cleaning degree is higher, so that the cleaning equipment can be controlled more accurately by using the cleaning degree information.

Description

Technical Field

[0001] The present application relates to the field of cleaning equipment, and in particular to a method for identifying the cleanliness level of an object to be cleaned for a cleaning equipment, a method for controlling a cleaning equipment, a cleanliness level identifying device, a cleaning equipment, and a computer-readable storage medium. Background Art

[0002] In smart cleaning devices such as smart dishwashers, vegetable washers or smart washing machines, it is often necessary to obtain the cleanliness of the objects to be cleaned. The cleanliness is also called cleanliness, cleanliness, stain information or turbidity information, etc., which is used to characterize the cleanliness of the objects to be cleaned. Depending on the specific type of cleaning device, the objects to be cleaned may be, for example, tableware, vegetables, fruits and clothes.

[0003] The intelligent cleaning equipment needs to determine the specific information of the subsequent cleaning program according to the cleanliness of the object to be cleaned. Therefore, it is of great significance to accurately and quickly determine the cleanliness of the object to be cleaned. Summary of the invention

[0004] The present application provides a method for identifying the cleanliness level of an object to be cleaned for a cleaning device, a method for controlling the cleaning device, a cleanliness level identification device, a cleaning device, and a computer-readable storage medium, so as to more accurately detect the cleanliness level of the object to be cleaned.

[0005] In order to achieve the above technical effects, a technical solution adopted by the present application is: to provide a method for identifying the cleanliness of an object to be cleaned for a cleaning device. The method for identifying the cleanliness includes: obtaining a target image including the object to be cleaned; extracting a first feature map group and a second feature map group of different scales from the target image, wherein the scale of the first feature map group is larger than the scale of the second feature map group; identifying the position information of the object to be cleaned based on the second feature map group; obtaining a detection frame of the object to be cleaned from the first feature map group based on a mapping relationship between the position information and the first feature map group; and identifying the cleanliness of the object to be cleaned from the detection frame of the object to be cleaned.

[0006] In order to achieve the above technical effects, another technical solution adopted by the present application is to provide a control method for a cleaning device. The control method includes: controlling an image acquisition device to acquire a target image of an object to be cleaned; identifying the cleanliness of the object to be cleaned according to the above cleanliness recognition method; determining the corresponding cleaning intensity and cleaning agent type based on the cleanliness of the object to be cleaned; and controlling the cleaning device to clean the object to be cleaned based on the cleaning intensity and cleaning agent type.

[0007] To achieve the above technical effects, another technical solution adopted by the present application is: to provide a device for identifying the cleanliness degree of an object to be cleaned for a cleaning device. The cleanliness degree identification device includes a target image acquisition module, a feature extraction module, a first identification module, a mapping module and a second identification module. The target image acquisition module is configured to acquire a target image including the object to be cleaned. The feature extraction module is configured to extract a first feature map group and a second feature map group of different scales from the target image, and the scale of the first feature map group is greater than the scale of the second feature map group. The first identification module is configured to identify the position information of the object to be cleaned based on the second feature map group. The mapping module is configured to acquire a detection frame of the object to be cleaned from the first feature map group based on the mapping relationship between the position information and the first feature map group. The second identification module is configured to identify the cleanliness degree of the object to be cleaned from the detection frame of the object to be cleaned.

[0008] To achieve the above technical effects, another technical solution adopted by the present application is: to provide a cleaning device. The cleaning device includes a memory and a processor. The memory stores a computer program. The processor is used to execute the computer program to implement the above cleaning degree identification method.

[0009] In order to achieve the above technical effects, another technical solution adopted by the present application is: providing a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the above cleaning degree identification method is implemented.

[0010] Different from the prior art, the present application extracts a first feature map group and a second feature map group of different scales from a target image, wherein the scale of the first feature map group is larger than that of the second feature map group, identifies the position information of the object to be cleaned based on the second feature map group, and obtains a detection frame of the object to be cleaned from the first feature map group based on the mapping relationship between the position information and the first feature map group; and identifies the cleanliness degree of the object to be cleaned from the detection frame of the object to be cleaned. Through this scheme, the present application can obtain the position information of the object to be cleaned from the second feature map group of smaller scale, and map the position information to the first feature map group of larger scale, thereby obtaining the cleanliness degree of the object to be cleaned. Since the first feature map group contains more detailed features, the accuracy of the cleanliness degree obtained by the present application is higher, thereby allowing the use of the cleanliness degree information to implement more precise control of the cleaning equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 A schematic structural diagram of a cleaning device according to an embodiment of the present application is shown.

[0013] Figure 2 A schematic diagram of a cleaning equipment system according to an embodiment of the present application is shown.

[0014] Figure 3 A schematic flow chart of a method for identifying the cleanliness level of an object to be cleaned for use in a cleaning device according to an embodiment of the present application is shown.

[0015] Figure 4 An example of an embodiment of the present application is shown. Figure 3 Schematic diagram of the structure of the neural network model of the cleanliness recognition method.

[0016] Figure 5 A schematic diagram of extracting multiple feature layers according to an embodiment of the present application is shown.

[0017] Figure 6 A schematic diagram is shown of performing feature fusion on at least two feature maps in the second feature map group according to an embodiment of the present application.

[0018] Figure 7 An embodiment according to the present application is shown Figure 4 Schematic diagram of the structure of the mapping module in .

[0019] Figure 8 A schematic diagram is shown of performing feature fusion on feature maps in a first feature map group and generating a detection frame according to an embodiment of the present application.

[0020] Fig. 9 A schematic diagram is shown of generating a detection frame from a feature map in a first feature map group according to an embodiment of the present application.

[0021] Fig.10 An embodiment according to the present application is shown Figure 3 Schematic diagram of the flow chart of step S14 in .

[0022] Fig.11 A schematic diagram showing the execution of maximum pooling on a mapped detection frame according to an embodiment.

[0023] Fig.12 A flow chart of a method for controlling a cleaning device according to an embodiment of the present application is shown.

[0024] Fig.13 A schematic structural diagram of a device for identifying the cleanliness level of an object to be cleaned used in a cleaning device according to an embodiment is shown.

[0025] Fig.14 A schematic structural diagram of a cleaning device according to an embodiment of the present application is shown.

[0026] Fig.15 A schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Before describing the embodiments of the present application in detail, the cleaning device of the present application is briefly described. The cleaning device may be, for example, a dishwasher, a vegetable washing machine, a washing machine, a dry cleaning machine, or other cleaning devices. The corresponding objects to be cleaned may be, for example, tableware, vegetables, fruits, and clothing. The cleaning device may include a closable cavity inside to place and accommodate the objects to be cleaned, so as to perform various operations such as cleaning the objects to be cleaned and optional disinfection and drying. The following description is given by taking a dishwasher as an example.

[0030] See also Figure 1 , Figure 1 1 is a schematic diagram of the structure of a cleaning device 100 provided according to an embodiment of the present invention. The cleaning device 100 may be, for example, a dishwasher. The cleaning device 100 defines a cleaning area inside thereof for placing objects to be cleaned in the area for cleaning. For example, a plurality of baskets or racks may be provided in the cleaning area. Objects to be cleaned, such as tableware, vegetables, fruits, or clothes, may be placed in the baskets or racks.

[0031] At least one image acquisition device 10 may be provided on the inner wall of the cleaning area to acquire the target image of the cleaning area, especially the object to be cleaned. In some embodiments, the cleaning device 100 may include multiple image acquisition devices 10. The image acquisition device 10 may also be provided at the top, side or other position of the cleaning area. The present application does not limit the arrangement position and number of the image acquisition device 10.

[0032] In some embodiments, the cleaning device 100 may further include at least one cleaning device 20. The cleaning device 20 may be, for example, a spray head or a spray arm. The at least one cleaning device 20 may spray a cleaning liquid (e.g., water, a detergent, or a mixture thereof) onto the object to be cleaned. The cleaning liquid spray flow rate, spray direction, etc. of the at least one cleaning device 20 may be adjusted according to the identification of the object to be cleaned in the cleaning device 100.

[0033] In some embodiments, each of the at least one cleaning device 20 can be controlled separately to provide different levels of cleaning services to different parts of the cleaning area.

[0034] See also Figure 2 , Figure 2 A schematic diagram of a cleaning device system 200 according to an embodiment of the present application is shown. The cleaning device system 200 may include at least one cleaning device 100 and a server 300. The server 300 may communicate and exchange data with multiple cleaning devices 100 via a network. In the cleaning device system 200, different users and cleaning devices 200 deployed at different locations may share data and computing models, so as to realize intelligent control of the cleaning devices 100 with the help of the server 300 and big data.

[0035] The network includes but is not limited to the Internet, Ethernet, mobile communication network, wide area network, metropolitan area network, etc. The above mobile communication network includes but is not limited to the third generation (3G) mobile communication network, the third and a half generation (3.5G) mobile communication network, the fourth generation (4G) mobile communication network and the fifth generation (5G) mobile communication network, etc.

[0036] According to different division methods, the server 300 can be a centralized server, a distributed server, a cloud server, etc., and this application does not limit this. Compared with a single cleaning device 100, the server 300 may have more powerful computing power and computing speed. In the specific application of modern intelligent cleaning equipment 100, services with high computing requirements such as target recognition are often performed by the server 300 rather than a single cleaning device 100 to provide better recognition accuracy and speed. In addition, deploying a target detection model, such as a tableware detection model on the server 300, can also better maintain the target detection model, thereby providing the latest and most advanced target recognition services for all cleaning devices 100 that are connected to the server 300 in communication. Of course, the target detection model can also be deployed on a single cleaning device 100, and this application does not limit this.

[0037] As described below, in the present application, the cleaning device system 200 can use the target detection model to identify the number of objects to be cleaned, the type of objects to be cleaned, and / or the cleanliness level of each object to be cleaned, and determine specific cleaning parameters of the objects to be cleaned based on this information, including the amount of water and detergent, cleaning intensity, cleaning time, etc. Specifically, the target detection model can be deployed on either the cleaning device 100 or the server 300.

[0038] refer to Figure 3 and Figure 4 , Figure 3 A schematic diagram showing a flow chart of a method for identifying the cleanliness level of an object to be cleaned by a cleaning device 100 according to an embodiment of the present application is shown. Figure 4 An example of an embodiment of the present application is shown. Figure 3 Schematic diagram of the structure of the neural network model 400 of the cleanliness recognition method.

[0039] like Figure 3 As shown, the method for identifying the cleanliness level of the object to be cleaned for the cleaning device 100 may include the following steps S11 to S15. Specifically, the method for identifying the cleanliness level may be, for example, Figure 4 The neural network model 400 shown in is executed.

[0040] Figure 4 The neural network model 400 shown in the figure may be, for example, a trained neural network model, which may be packaged into an executable program or exist in the form of interpretable code. The neural network model 400 may be stored locally in the cleaning device 100 or in the server 300, and this application does not limit this.

[0041] like Figure 4 As shown, the neural network model 400 may include an input network 401, a backbone network 402, a neck network 403, and a head network 404. In addition, the neural network model 400 in the present application may also include a mapping network 405 and a cleanliness level recognition network 406. The neck network 403 is also called the first feature fusion network. Specifically, the various parts of the neural network model 400 will be described in detail below. Figure 3 In some embodiments, the processor of the cleaning device 100 or the server 300 may be configured to execute the neural network model 400 to implement Figure 3 Method for identifying the degree of cleanliness in .

[0042] Step S11: Acquire a target image including the object to be cleaned.

[0043] Specifically, the input network 401 of the neural network model 400 may receive a target image including an object to be cleaned.

[0044] In some embodiments, the image acquisition device 10 of the cleaning device 100 can capture an image of the cleaning area after the cleaning device 100 starts or completes a stage of cleaning to obtain a target image. The target image can be, for example, a color image, a grayscale image, or an infrared image, etc., which is not limited in this application. The target image generally has a predetermined size, such as 640*640, 800*800, etc. This application does not limit the size of the target image.

[0045] In some embodiments, the input network 401 may perform preprocessing on the target image, such as standardization, normalization, etc. The normalized image may also resist the influence of geometric transformation. In some embodiments, the normalization method may include linear function conversion, logarithmic function conversion, inverse cotangent function conversion, and conversion based on invariant moments. Among them, the image normalization process based on invariant moments may include steps such as coordinate centering, x-shearing normalization, scaling normalization, and rotation normalization. The size of the target image after normalization is, for example, 640*640 or 320*320, etc., and this application does not limit this.

[0046] Optionally, the neural network model 400 may not include a separate input network 401. The target image may be directly acquired by the neural network model 400 and enter the subsequent network.

[0047] Step S12: extracting a first feature map group and a second feature map group of different scales from the target image. The scale of the first feature map group is larger than the scale of the second feature map group.

[0048] Feature extraction of a target image refers to a method of transforming the data of a target image to highlight the representative features of a certain pattern of the target image. Feature extraction of a target image can be performed through data analysis and transformation.

[0049] In some embodiments, the present application may extract a first feature map group and a second feature map group of different scales from the target image. The first feature map group and the second feature map group each include at least one feature map. In the following, the feature map is also referred to as a feature layer. The feature maps in the first feature map group and the second feature map group may have different scales or sizes. For example, the scale of the first feature map group is larger than the scale of the second feature map group. In other words, the feature maps in the first feature map group may be referred to as shallow features or bottom features, while the feature maps in the second feature map group may be referred to as deep features or high-level features. Generally speaking, shallow features have a larger scale and contain more detailed information. The scale of deep features is smaller, but contains more high-level semantic features.

[0050] In some embodiments, reference Figure 5 , Figure 5FIG. 2 shows a schematic diagram of extracting multiple feature layers according to an embodiment of the present application. Figure 5 As shown, the present application can start from the target image and use filters or convolution kernels to extract the first feature layer C1, the second feature layer C2, the third feature layer C3, the fourth feature layer C4 and the fifth feature layer C5 layer by layer. In the present application, feature layer and feature map can be used interchangeably.

[0051] For example, the present application extracts the first feature layer C1 from the target image through convolution, and the number of channels of the first feature layer C1 may be, for example, 64. Further, the present application may extract the second feature layer C2 from the first feature layer C1 using a 3*3 convolution kernel. In the process of extracting from the first feature layer C1 to the second feature layer C2, each 3*3 convolution kernel is also followed by a batch normalization layer and a ReLU activation function. The number of channels of the second feature layer C2 is, for example, 64. Further, the present application extracts the third feature layer C3 from the second feature layer C2 using a 3*3 convolution kernel. In the process of extracting from the second feature layer C2 to the third feature layer C3, each 3*3 convolution kernel is also followed by a batch normalization layer and a ReLU activation function. The number of output channels of the third feature layer is, for example, 128. Further, the present application extracts the fourth feature layer C4 from the third feature layer C3 using a 3*3 convolution kernel. In the process of extracting from the third feature layer C3 to the fourth feature layer C4, each convolution kernel is followed by a batch normalization layer and a ReLU activation function. The number of output channels of the fourth feature layer is, for example, 256. Further, the present application extracts the fifth feature layer C5 from the fourth feature layer C4 by extracting a 3*3 convolution kernel. In the process of extracting from the fourth feature layer C4 to the fifth feature layer C5, each convolution kernel is followed by a batch normalization layer and a ReLU activation function. The number of output channels of the fifth feature layer C5 is, for example, 512.

[0052] In some embodiments, the fourth characteristic layer C4 and the fifth characteristic layer C5 may be collectively referred to as a second characteristic map group. The fourth characteristic layer C4 may be referred to as a third characteristic map in this article, and the fifth characteristic layer C5 may be referred to as a fourth characteristic map in this article. Optionally, the second characteristic map group may also include only one characteristic layer, such as the fifth characteristic layer C5. In some embodiments, the second characteristic map group may also include more than two characteristic layers, which is not limited in this application.

[0053] In some embodiments, the second characteristic layer C2 and the third characteristic layer C3 may be collectively referred to as a first characteristic map group. The second characteristic layer C2 may be referred to as a first characteristic map in this article, and the third characteristic layer C3 may be referred to as a second characteristic map in this article. Optionally, the first characteristic map group may also include only one characteristic layer, such as one of the first characteristic layer C1, the second characteristic layer C2, and the third characteristic layer C3. In some embodiments, the first characteristic map group may also include more than two characteristic layers, which is not limited in this application.

[0054] In some embodiments, the third feature map is obtained from the second feature map through depthwise separable convolution, and / or the fourth feature map is obtained from the third feature map through depthwise separable convolution. Specifically, a depthwise separable convolution includes channel-by-channel convolution and pointwise convolution in sequence. A convolution kernel of channel-by-channel convolution is responsible for one channel, or a channel is convolved by only one convolution kernel. The number of channels of the result produced by channel-by-channel convolution is exactly the same as the number of input channels. Point-by-point convolution is a conventional convolution operation using 1*1*M (M is the number of channels of the previous layer). Point-by-point convolution performs weighted combination of the results obtained by channel-by-channel convolution in the depth direction to generate a new feature layer. Depthwise separable convolution can be performed, for example, through a lightweight network mobilenet. Compared with conventional convolution, depthwise separable convolution is faster and has a smaller computational burden. Since the size of the objects to be cleaned is generally large, depthwise separable convolution can be used to detect macro information of the objects to be cleaned, such as position and type, to speed up the calculation without losing accuracy.

[0055] In some embodiments, the first feature map (eg Figure 5 The second characteristic layer C2 in Figure 5 The third characteristic layer C3 in Figure 5 The third characteristic layer C3 in the figure) and the fourth characteristic map (such as Figure 5 The fourth feature layer C4) in is a feature layer obtained directly or indirectly from the target image through layer-by-layer convolution. Specifically, the second feature map can be directly extracted from the first feature map through convolution, or there may be other intermediate feature maps between the second feature map and the first feature map. The third feature map can be directly extracted from the second feature map through convolution, or there may be other intermediate feature maps between the third feature map and the second feature map. The fourth feature map can be directly extracted from the third feature map through convolution, or there may be other intermediate feature maps between the fourth feature map and the third feature map. The first feature map can be directly extracted from the target image through convolution, or there may be other intermediate feature maps between the first feature map and the target image, such as Figure 4 The first characteristic layer C1 shown in FIG.

[0056] As described above, in some embodiments, the second feature map group may be obtained from the first feature map group by convolution extraction. Of course, those skilled in the art will appreciate that the second feature map group and the first feature map group may also be generated independently of each other, and this application does not limit this.

[0057] refer to Figure 4 In the present application, feature extraction can be performed by the backbone network 402 of the neural network model 400. The backbone network 402 can perform a series of convolutions on the target image using filters, convolution kernels, etc. to obtain feature layers or feature maps of different scales. The backbone network 402 can include at least a first feature extraction network 4021 and a second feature extraction network 4022.

[0058] In some embodiments, the first feature extraction network 4021 may be configured to extract the first feature map group as described above, and the second feature extraction network 4022 may be configured to extract the second feature map group as described above. In some embodiments, the output of the first feature extraction network 4021 may be input into the second feature extraction network 4022.

[0059] In some embodiments, the first feature extraction network 4021 can be configured to use conventional convolution to extract features, and the second feature extraction network 4022 can be configured to use deep separable convolution to extract features. For example, the first feature extraction network 4021 can adopt network designs similar to VGG (Visual Geometry Group), Resnet (Residual Network), Inception, etc., and the second feature extraction network 4022 can adopt network designs similar to mobilenet network, shufflenet network, ghostnet network, etc. In this embodiment, since the objects to be cleaned are generally large in size and have high requirements for details, the first feature extraction network 4021 uses conventional convolution to extract more details and texture information. And since the objects to be cleaned are generally large in size in the target image taken by the cleaning device 100, and small targets rarely appear, the second feature extraction network 4022 can use deep separable convolution to strengthen the extraction speed of deep or high-level semantic information, and can greatly reduce the computational burden.

[0060] It can be understood that a normalization layer and an activation function layer are configured after the conventional convolution layer or the ordinary convolution layer, the depth convolution layer and the point-by-point convolution layer in the backbone network 402. Among them, the normalization layer can be implemented by the existing BatchNormalization algorithm to normalize the data, that is, the mean of the data input to the next layer is 0 and the variance is 1. On the one hand, this can reduce the generalization ability of the network, and on the other hand, it can increase the training speed of the network. The activation function layer can be implemented by the existing ReLU function to increase the nonlinearity of the model and overcome the problem of gradient disappearance.

[0061] In some embodiments, other feature extraction networks may also exist before the first feature extraction network 4021 and between the first feature extraction network 4021 and the second feature extraction network 4022, and the present application does not limit this. In general, the backbone network 402 as the basic feature extraction network will fully learn the input target image.

[0062] Step S13: Identifying the position information of the object to be cleaned based on the second feature map group.

[0063] In the present application, the position information of the object to be cleaned can be identified from the feature map in the second feature map group based on the target detection network. Specifically, the feature map in the second feature map group can be input into the target detection network (head network, also known as head network 404) to perform target recognition and position detection on the object to be cleaned in the feature map.

[0064] Specifically, the output of the head network 404 includes classification output (classification of various objects to be cleaned), position output (position x, y, w, h of various objects to be cleaned in the target) and centrality (or confidence, score) output.

[0065] In some embodiments, the head network 404 may use an anchor frame-based algorithm or an anchor frame-free algorithm to identify the target detection frame. In the head network 404 designed based on the anchor frame, the number of anchor frames needs to be set in advance, and the final output dimension is the product of the number of anchor frames and the number of detections of the corresponding task. In the head network 404 designed without an anchor frame, direct prediction is made for each point on the feature map. Specifically, the anchor frame-based algorithm may be, for example, a YOLO algorithm based on an anchor frame. The anchor frame-free algorithm may be, for example, a YOLOX algorithm or a FCOS algorithm based on an anchor frame-free algorithm.

[0066] In some embodiments, the head network 404 may be designed in a shared weight manner, that is, the same head network 404 is used to output information such as the category, confidence level, and location of the object to be cleaned. This application does not limit this.

[0067] In some embodiments, when the output of the head network 404 includes the classification output, the position output and the confidence output, the head network 404 may adopt a design method that does not share weights. In the design method that does not share weights, the output of the head network 404 can be obtained by different convolutional networks. For example, the head network 404 may include a first head network and a second head network that are independent of each other, i.e., do not share network parameters. The first head network can be used to receive the input feature map to output the classification and corresponding confidence of the object to be cleaned. The second head network can be used to receive the input feature map to output the position information of the object to be cleaned. Any one of the first head network and the second head network can independently adopt any one of the algorithms based on the anchor frame and the algorithms not based on the anchor frame. Compared with the method of sharing weights, the design method of non-shared weights can make the obtained position information of the object to be cleaned more accurate.

[0068] In some embodiments, the head network 404 may use a two-stage (two_stage) detection algorithm and a one-stage (one_stage) detection algorithm. The two-stage detection algorithm first uses the RPN network to generate candidate regions (region proposals) for coarse positioning, and then accurately locates (regresses) and classifies and predicts the candidate regions in the network. The one-stage detection algorithm, on the other hand, directly regresses the category probability and location information of the target object based on the input feature map, without having to generate candidate regions for coarse positioning. Compared with the one-stage detection algorithm, the two-stage detection algorithm has a faster detection speed, which helps to achieve target detection in a real-time environment. The target detection model of the present application may use any of the above-mentioned two-stage detection algorithm and one-stage detection algorithm.

[0069] In some embodiments, the head network 404 may include, for example, a fully connected layer and a softmax layer. The fully connected layer may integrate and calculate the input feature map and output a one-dimensional vector whose dimension is equal to the number of neurons in the fully connected layer. The vector output by the fully connected layer may be input to the softmax layer. The softmax layer includes a softmax function for outputting a one-dimensional predicted label probability distribution based on the input vector, that is, outputting the category and confidence of the object to be cleaned.

[0070] In some embodiments, before the step of identifying the position information of the object to be cleaned based on the second feature map group, the present application also includes the step of fusing at least two feature maps in the second feature map group to generate a first fused feature map group. The feature maps in the first fused feature map group can then be input into the head network 404, the first head network or the second head network to detect the category, position and confidence of the object to be cleaned.

[0071] In some embodiments, the operation of performing feature fusion on at least one feature map in the second feature map group may be performed using Figure 4 The neck network 403 in the embodiment is used for execution. The neck network 403 can receive the first feature map group output by the backbone network 402, perform feature fusion operation, and output the first fused feature map group. After feature fusion, the neck network 403 can enrich the semantic information of the output first fused feature map group and improve the accuracy of subsequent target detection.

[0072] Specifically, refer to Figure 6 , Figure 6 A schematic diagram of feature fusion of at least two feature maps in the second feature map group according to an embodiment of the present application is shown. As shown in the figure, in this embodiment, the second feature map group includes a fourth feature layer C4 and a fifth feature layer C5. The fifth feature layer C5 is convolved to generate a fifth intermediate feature layer I5. After upsampling, the fifth intermediate feature layer I5 is added with the result of convolution with the fourth feature layer C4 to generate the fourth intermediate feature layer I4. The fifth intermediate feature layer I5 is convolved to generate a sixth intermediate feature layer I6. The sixth intermediate feature layer I6 is then convolved to generate a seventh intermediate feature layer I7. The size of the convolution kernel used for the above convolution may be, for example, 3*3 or other sizes, and the present application does not limit this.

[0073] In some embodiments, the fourth intermediate feature layer I4, the fifth intermediate feature layer I5, the sixth intermediate feature layer I6, and the seventh intermediate feature layer I7 may be convolved respectively, for example, using a 3*3 convolution kernel to generate the corresponding fourth fused feature map P4, the fifth fused feature map P5, the sixth fused feature map P6, and the seventh fused feature map P7. The fourth fused feature map P4, the fifth fused feature map P5, the sixth fused feature map P6, and the seventh fused feature map P7 may then be input into the head network 404 to identify information such as the category, confidence, and location information of the object to be cleaned.

[0074] In some embodiments, the fourth fused feature map P4, the fifth fused feature map P5, the sixth fused feature map P6, and the seventh fused feature map P7 may be input into the head network 404 after being spliced ​​into a one-dimensional vector. The fourth fused feature map P4, the fifth fused feature map P5, the sixth fused feature map P6, and the seventh fused feature map P7 may respectively provide feature maps of different scales to facilitate the head network 404 to identify objects to be cleaned of different scales. Objects to be cleaned may be detected from fused feature maps of different scales.

[0075] As an example but not limitation, the size of the fourth fused feature map P4 is 40*40, and the number of channels is 512. The size of the fifth fused feature map P5 is 20*20, and the number of channels is 1024. The size of the sixth fused feature map P6 is 10*10, and the number of channels is 2048. The size of the seventh fused feature map P7 is 5*5, and the number of channels is 4096.

[0076] Optionally, the present application may also use other types of pyramid feature extraction networks to perform feature fusion on the feature layers in the second feature map group, and the present application does not impose any restrictions on this.

[0077] Step S14: based on the mapping relationship between the position information and the first feature map group, a detection frame of the object to be cleaned is obtained from the first feature map group.

[0078] Specifically, in the present application, the position information of the object to be cleaned identified in the head network 404 is mapped to the first feature map group, so as to select or generate a detection frame of the object to be cleaned from the feature map in the first feature map group.

[0079] refer to Figure 7 , Figure 8 and Fig.10 , Figure 7 An embodiment according to the present application is shown Figure 4 A schematic diagram of the structure of the mapping network 405, Figure 8 A schematic diagram showing feature fusion of feature maps in the first feature map group and generation of a detection frame B1 according to an embodiment of the present application, and Fig.10 An embodiment according to the present application is shown Figure 3 Schematic diagram of the process of step S14 in FIG. Step S14 includes step S21 and step S22.

[0080] Step S21: performing feature fusion on the feature maps in the first feature map group to obtain a fused feature map.

[0081] The following Figure 8 The description is made by taking the example that the first feature map group includes the second feature layer C2 and the third feature layer C3. Those skilled in the art should understand that the first feature map group may include other configurations, and the present application does not limit this.

[0082] like Figure 8 As shown, the second feature layer C2 is successively subjected to convolution processing and downsampling processing to generate a second fused feature map P2, and the number of channels and size of the second fused feature map P2 are the same as those of the third feature layer C3. The third feature layer C3 is subjected to convolution processing (for example, convolution processing using a 3*3 convolution kernel) with the number of channels and size kept unchanged, and then added to the second fused feature map P2 to generate a fused third fused feature map P3. This step S21 can be, for example, Figure 7Specifically, the first feature map feature fusion network 4051 receives the output from the backbone network 402, and inputs the generated third fusion feature map P3 into the detection frame generation network 4052 to generate the detection frame B1.

[0083] Step S22: Mapping the position information of the object to be cleaned to the fused feature map to generate a detection frame B1 of the object to be cleaned on the fused feature map.

[0084] Specifically, the position information of the object to be cleaned obtained by the head network 404 is mapped back to the third fused feature map P3, so as to generate a corresponding target area as the detection frame B1 on the third fused feature map P3.

[0085] In some embodiments, the detection frame B1 may be pooled, and the pooling method may be maximum pooling, average pooling, K-max pooling, or the like.

[0086] In some embodiments, step S22 may be performed by algorithms such as ROI pooling or ROI Align, etc. ROIAlign is used as an example for description below.

[0087] First, the position information of the object to be cleaned on the target image is mapped to the third fusion feature map P3 in a manner of retaining floating point numbers.

[0088] Assume that the size of the target image is 800*800, and the size of the third fused feature map P3 is 25*25. The head network 404 detects that the size of an object to be cleaned in the target image (expressed in the form of width*height) is 65*65, and the center point is (80, 168). The size mapped to the third fused feature map P3 is 665*25 / 800=20.78, and the coordinates of the center point (80, 168) mapped in the third fused feature map P3 are (80*25 / 800, 168*25 / 800)=(2.5, 5.25). Since the third fused feature map P3 is obtained by proportionally reducing the target image, the mapping result of the target detection frame of the object to be cleaned on the third fused feature map P3 can be calculated by a proportional transformation method to obtain the detection frame B1 on the third fused feature map P3. In the ROI Align algorithm, the mapping result is expressed in a way that retains floating point numbers, and is not directly rounded and quantized like a general mapping method.

[0089] In some embodiments, the coordinate information of the object to be cleaned in the target image can also be represented by the coordinates of a pair of diagonal points of a bounding box. The coordinates of the diagonal points can also be proportionally changed in a manner that retains floating point numbers, such as in the manner described above, to perform the coordinate mapping process.

[0090] Next, the mapped detection frame B1 can be pooled, such as maximum pooling. Take the case where the detection frame B1 is pooled to a feature map fixed to a size of 2*2. The length and width of the above-mentioned detection frame B1 of size 20.78*20.78 are equally divided into 2 parts. The entire detection frame B1 can be divided into 2*2=4 small areas. 20.78 / 2=10.39, then the size of each small area is 10.39*10.39. ROI Align then calculates the pixel value of each small area.

[0091] refer to Fig.11 , Fig.11 A schematic diagram showing a method of performing maximum pooling on the mapped detection box B1 according to an embodiment. Fig.11 At least a portion of the third fused feature map P3 is shown by a dotted line. The mapped detection box B1 is located on the third fused feature map P3 and is shown by a solid line.

[0092] like Fig.10 As shown, the mapped detection box B1 is divided into 4 small areas. Take the small area R1 in the upper left corner as an example. The small area R1 is further divided into four small boxes. Find the center point of each small box as the sampling point, and calculate the pixel value of the center point. Obviously, the center point is generally not a pixel point of the third fusion feature map P3. In order to calculate the value of the center point, the values ​​of the four pixel points PT1, PT2, PT3 and PT4 closest to the center point on the third fusion feature map P3 can be used for bilinear interpolation to calculate the pixel value of the center point. Calculate the values ​​of the center points of the four small boxes in the small area R1 in turn, and take the maximum value as the pixel value of the small area R1.

[0093] pass Fig.11 By using the method shown in , the pixel values ​​of all four small areas can be calculated, thereby completing the pooling of the detection box B1.

[0094] In some embodiments, the small area R1 may also be divided into two small frames to obtain two sampling points. The center point of the small area R1 may also be directly used as the sampling point.

[0095] In some embodiments, the detection frame B1 may also be pooled into detection frames of other sizes, such as 5*5, 7*7, etc., which is not limited in this application.

[0096] Compared with the traditional ROI pooling algorithm, ROI Align can solve the misalignment problem caused by the quantization of coordinate values ​​and improve the accuracy of recognition results.

[0097] In some embodiments, reference Fig. 9 , Fig. 9 FIG. 2 is a schematic diagram showing a method of generating a detection frame from a feature map in a first feature map group according to an embodiment of the present application. Fig. 9 As shown above Figure 8 Compared with the embodiment in Fig. 9 The embodiment in the embodiment can omit the fusion step of multiple underlying feature maps, and directly map the position information to the third feature layer C3 or other feature layers to extract or generate the detection frame B1. The present application can also further perform pooling processing on the detection frame B1.

[0098] S15: Identify the cleanliness level of the object to be cleaned from the detection frame B1 of the object to be cleaned.

[0099] In some embodiments, the degree of cleanliness can be discretely divided into several levels. For example, the degree of cleanliness can be divided into 11 levels, namely 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90% and 100%. Those skilled in the art should understand that other division methods can also be used, and this application is not limited to this. The degree of cleanliness is also called the degree of cleanliness or the degree of stains. For example, 0% means that the degree of cleanliness is very poor, and 100% means that it is basically clean. Other degrees of cleanliness between 0% and 100% represent different degrees of cleanliness according to their specific numerical values. The larger the value, the cleaner it is.

[0100] For the pooled detection frame B1 obtained in step S14, a global average pooling operation can be further performed to obtain a 1*1*M vector, where M is the number of channels. After that, a 1*1 convolution operation can be performed to convert the number of channels of the vector into the final cleanliness level number, for example, 11 in the above text. The vector can then be input into a softmax network to identify the cleanliness level of the object to be cleaned.

[0101] The degree of cleanliness can be used to control the operation of the cleaning device 100. For example, if the degree of cleanliness does not reach 100%, the control means such as the water output strength, water output volume and water output temperature are dynamically adjusted according to the current degree of cleanliness. If it reaches 100%, the cleaning is terminated. In some embodiments, the drying process of the cleaning device can be started afterwards.

[0102] refer to Fig.12 , Fig.12 FIG. 1 is a flow chart showing a control method of a cleaning device 100 according to an embodiment of the present application. Fig.12 As shown, the control method of the cleaning device 100 includes steps S31 to S34.

[0103] Step S31: controlling the image acquisition device 10 to acquire a target image of the object to be cleaned.

[0104] Specifically, as described above, the cleaning device 100 includes an image acquisition device 10. The present application can control the image acquisition device 10 of the cleaning device 100 to acquire a target image of the object to be cleaned.

[0105] In some embodiments, a trigger may be provided at the door opening mechanism of the cleaning device 100. When the door of the cleaning device 100 is closed, the trigger is triggered and sends a trigger signal. The controller of the cleaning device 100 receives the trigger signal, and in response to the trigger signal, starts to initialize the image acquisition device 10, and controls the image acquisition device 10 to capture images of the dishes in the cleaning device 100.

[0106] In some embodiments, after one stage of cleaning is completed, the image acquisition device 10 may be controlled to acquire a target image of the object to be cleaned.

[0107] In some embodiments, the user can control the image acquisition device 10 to acquire a target image of the object to be cleaned by using a start button.

[0108] Step S32: Identify the cleanliness level of the object to be cleaned according to the cleanliness level identification method.

[0109] Specifically, the above-mentioned cleanliness level recognition method may be performed on the target image to recognize the cleanliness level of the object to be cleaned.

[0110] Step S33: Determine the corresponding cleaning intensity and cleaning agent type based on the cleanliness level of the object to be cleaned.

[0111] Specifically, the present application can dynamically determine the cleaning intensity and cleaning agent type according to the current cleaning degree. The cleaning intensity includes parameters such as water output intensity, cleaning dosage, and cleaning time. The present application can use different cleaning agent types according to different cleaning intensities.

[0112] In some embodiments, if the cleanliness level is very low, such as less than 20%, a strong cleaning agent may be used for cleaning. If the cleanliness level is high, such as between 20% and 80%, a general cleaning agent may be used. If the cleanliness level is very high, such as above 80%, no cleaning agent may be used.

[0113] Step S34: Based on the cleaning intensity and the type of cleaning agent, the cleaning device is controlled to clean the object to be cleaned.

[0114] Specifically, the cleaning device may be pre-set with a plurality of control programs. A specific control program may be selected according to the cleaning intensity and the type of cleaning agent, thereby controlling the cleaning device to clean the object to be cleaned.

[0115] refer to Fig.13 , Fig.13The cleanliness level recognition device 500 for cleaning an object to be cleaned according to an embodiment is shown. Specifically, the cleanliness level recognition device 500 includes a target image acquisition module 510, a feature extraction module 520, a first recognition module 530, a mapping module 540 and a second recognition module 550 which are connected to each other for communication.

[0116] Specifically, the target image acquisition module 510 is configured to acquire a target image including the object to be cleaned.

[0117] Specifically, the feature extraction module 520 is configured to extract a first feature map group and a second feature map group of different scales from the target image. The scale of the first feature map group is greater than the scale of the second feature map group.

[0118] Specifically, the first recognition module 530 is configured to recognize the position information of the object to be cleaned based on the second feature map group.

[0119] Specifically, the mapping module 540 is configured to obtain the detection frame of the object to be cleaned from the first feature map group based on the mapping relationship between the position information and the first feature map group.

[0120] Specifically, the second recognition module 550 is configured to recognize the cleanliness level of the object to be cleaned from the detection frame of the object to be cleaned.

[0121] refer to Fig.14 , Fig.14 FIG. 7 is a schematic diagram showing a structure of a cleaning device 700 according to an embodiment of the present application. The cleaning device 700 may be, for example, a dishwasher, a vegetable washing machine, a dry cleaning machine, or other device for cleaning objects to be cleaned. Fig.14 As shown, the cleaning device 700 further includes a processor 710 and a memory 720. The memory 720 may store a computer program. The processor 710 is used to execute the computer program to implement the above identification method.

[0122] It is understandable that the processor 710 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0123] refer to Fig.15 , Fig.15 A schematic diagram of the structure of a computer-readable storage medium 1000 according to an embodiment of the present application is shown. Specifically, the present application provides a computer-readable storage medium 1000. The computer-readable storage medium 1000 stores a computer program 1010. The computer program 1010 can be executed by a processor of a computer to implement the above-mentioned cleaning degree identification method.

[0124] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0125] The above are only implementation methods of the present application, and are not intended to limit the patent scope of the present application. Equivalent structures or equivalent process changes made by utilizing the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for identifying the cleanliness level of an object to be cleaned in a cleaning device, characterized in that: include: Acquire a target image including the object to be cleaned; Extracting a first feature map group and a second feature map group of different scales from the target image, wherein the scale of the first feature map group is larger than the scale of the second feature map group; Identifying the position information of the object to be cleaned based on the second feature map group; Based on the mapping relationship between the position information and the first feature map group, acquiring the detection frame of the object to be cleaned from the first feature map group; and The cleanliness degree of the object to be cleaned is identified from the detection frame of the object to be cleaned.

2. The method for identifying the degree of cleanliness according to claim 1, characterized in that: The step of acquiring the detection frame of the object to be cleaned from the first feature map group based on the mapping relationship between the position information and the first feature map group includes: The first feature map group includes only one feature map, The position information of the object to be cleaned is mapped to the one feature map to generate a detection frame of the object to be cleaned on the one feature map.

3. The method for identifying the degree of cleanliness according to claim 1, characterized in that: The step of acquiring the detection frame of the object to be cleaned from the first feature map group based on the mapping relationship between the position information and the first feature map group includes: Performing feature fusion on the feature maps in the first feature map group to obtain a fused feature map; The position information of the object to be cleaned is mapped to the fused feature map to generate a detection frame of the object to be cleaned on the fused feature map.

4. The method for identifying the degree of cleanliness according to claim 3, characterized in that: The step of mapping the position information of the object to be cleaned to the fused feature map to generate a detection frame of the object to be cleaned on the fused feature map includes: The position information of the object to be cleaned is mapped to the fused feature map using a ROI align algorithm, and a detection frame of the object to be cleaned is generated on the fused feature map.

5. The method for identifying the degree of cleanliness according to claim 3, characterized in that: The second feature map group is extracted from the first feature map group; The first feature map group includes at least a first feature map and a second feature map; and The second feature map group includes at least a third feature map and a fourth feature map.

6. The method for identifying the degree of cleanliness according to claim 5, characterized in that: The first feature map, the second feature map, the third feature map and the fourth feature map are obtained directly or indirectly from the target image through layer-by-layer convolution.

7. The method for identifying the degree of cleanliness according to claim 5, characterized in that: The third feature map is obtained from the second feature map through a depthwise separable convolution; and The fourth feature map is obtained from the third feature map through depth-wise separable convolution.

8. A control method for a cleaning device, the cleaning device comprising an image acquisition device and a cleaning device, characterized in that: The control method comprises: Controlling the image acquisition device to acquire a target image of the object to be cleaned; The method according to any one of claims 1 to 7, identifying the cleanliness level of the object to be cleaned; Determining the corresponding cleaning intensity and cleaning agent type based on the cleanliness level of the object to be cleaned; and Based on the cleaning intensity and the type of cleaning agent, the cleaning device is controlled to clean the object to be cleaned.

9. A device for identifying the cleanliness level of an object to be cleaned for use in a cleaning device, characterized in that: include: A target image acquisition module, configured to acquire a target image including the object to be cleaned; A feature extraction module, configured to extract a first feature map group and a second feature map group of different scales from the target image, wherein the scale of the first feature map group is larger than the scale of the second feature map group; A first recognition module, configured to recognize the position information of the object to be cleaned based on the second feature map group; a mapping module configured to obtain a detection frame of the object to be cleaned from the first feature map group based on a mapping relationship between the position information and the first feature map group; and The second recognition module is configured to recognize the cleanliness degree of the object to be cleaned from the detection frame of the object to be cleaned.

10. A cleaning device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the method for identifying the degree of cleanliness as claimed in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for identifying the degree of cleanliness according to any one of claims 1 to 7 is implemented.

Images