Dish-washing machine cleanliness detection method adopting T-Faster R-CNN

Abstract

The invention relates to the technical field of intelligent household appliances, in particular to a dishwasher cleanliness detection method adopting T-Faster R-CNN, comprising the following steps: constructing a tableware picture library; de-noising the images of the tableware picture library by using a nonlinear bilateral Wiener filtering method; dividing the denoised tableware picture library into a tableware training set and a tableware testing set; marking the images in the tableware training set; constructing a T-Faster R-CNN (Convolutional Neural Network) model, and performing feature extraction by adopting a T-SR network; the tableware training set is sent to a T-Faster R-CNN model for training, and a trained T-Faster R-CNN model is obtained; a G-Focal cleaning loss function is calculated, and a final T-Faster R-CNN (Convolutional Neural Network) model is obtained; and testing the final T-Faster R-CNN by using the tableware test set, and outputting a detection result to obtain the cleanliness of tableware cleaned by the dish washing machine. By fully utilizing the feature representation capability of the neural network, the dirt in the dish-washing machine can be accurately detected, the dependence on human intervention is reduced, and the robustness and adaptability of the algorithm are improved.

Description

[0001] Technology Neighborhood

[0002] This invention relates to the field of smart home appliance technology, specifically to a dishwasher cleanliness detection method using T-Faster R-CNN. Background Technology

[0003] In recent years, with the continuous improvement of the national economy, the release of consumption upgrade demands, and people's pursuit of quality of life and convenience, the dishwasher industry has flourished. However, in terms of penetration rate, countries such as the United States, France, and Germany have relatively high penetration rates, reaching over 50%, while in my country, the penetration rate is less than 10%. According to surveys, the low penetration rate of dishwashers in my country is because dishwashers entered the Chinese market relatively late, consumers' understanding of dishwashers is still immature, and many consumers have doubts about the cleaning effectiveness of dishwashers. This has resulted in the dishwasher market in my country not yet being fully opened up.

[0004] Therefore, researching methods for testing the cleanliness of dishwashers can help improve dishwasher products, demonstrate their cleaning capabilities to users, enhance their safety, and protect user health and safety. The cleanliness of a dishwasher is primarily assessed by the cleanliness of the dishes it washes. Currently, there are two main types of technologies for testing dish cleanliness: one is the more traditional manual testing method, including visual inspection, chromatography, and biological detection. However, these methods have limitations: visual inspection allows opening the dishwasher and checking the surface of the dishes for residue, grease, or odors, but this method is highly subjective; while chromatography or biological detection can provide quantitative results, they still have some error and are difficult to accurately measure the cleanliness of the dishes washed by the dishwasher. The other method utilizes neural network technology. Current technology uses BP neural networks for cleaning index prediction. BP neural networks have good nonlinear mapping capabilities and flexible network structures, allowing for better calculation of the dishwasher's cleanliness. However, this method is prone to overfitting and is sensitive to initial weights and learning rates, leading to inaccurate cleanliness test results. In conclusion, the current methods for detecting dishwasher cleanliness are still not very accurate, and the test results are not precise enough.

[0005] For dishwasher cleanliness detection, the Faster R-CNN neural network model can also be used. Faster R-CNN is a true end-to-end deep learning detection algorithm that generates candidate boxes based on an anchor mechanism. It integrates feature extraction layers, candidate region layers, region of interest layers, and prediction layers into a single network, effectively improving detection accuracy and efficiency. The Faster R-CNN model can directly extract features from the original image for object detection and classification. Compared to traditional methods, it simplifies the detection process and improves efficiency. Furthermore, Faster R-CNN has multi-class detection capabilities and the ability to automatically learn features, eliminating the need for manual rule design and threshold setting, thus reducing reliance on human intervention and improving the algorithm's robustness and adaptability. However, the Faster R-CNN model suffers from limitations in feature extraction, particularly in low-resolution images, which can negatively impact detection accuracy.

[0006] To address this, a dishwasher cleanliness detection method using T-Faster R-CNN is proposed. Summary of the Invention

[0007] The purpose of this invention is to provide a dishwasher cleanliness detection method using T-Faster R-CNN to solve the above-mentioned technical problems.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A dishwasher cleanliness detection method using T-Faster R-CNN includes:

[0010] S10. Build a tableware image library;

[0011] Preferably, step S10 includes crawling tableware images to obtain the tableware image library.

[0012] S20. Classify and preprocess the tableware image library, the main steps of which include: S21. Denoise the images in the tableware image library; S22. Divide the denoised tableware image library into a tableware training set and a tableware test set.

[0013] Preferably, a nonlinear bilateral Wiener filtering method is used to denoise the images in the tableware image library. The nonlinear bilateral Wiener filtering method includes:

[0014] The local mean μ around each pixel of image C in the tableware image library C and variance Make an estimate:

[0015]

[0016]

[0017] in, Let be the neighborhood of a pixel in the tableware image C, M×S be the size of this neighborhood, and u(n1,n2) be the pixels n1 and n2 in the local neighborhood. The pixel-by-pixel estimation formula for the tableware is as follows:

[0018]

[0019] Where Px represents a pixel, and w(n1,n2) are two-sided weighted coefficients. The variance of the noise is represented by w(n1,n2) and... The formula is as follows:

[0020] w(n1,n2)=w s (n1,n2)w r (n1,n2)

[0021]

[0022]

[0023]

[0024] Where w(n1,n2) is determined by the spatial proximity factor w s (n1,n2) and brightness similarity factor w r The product of (n1, n2) consists of the parameter σ. s and σ r Each controls w s and w r The degree of attenuation, where (x,y) is the center point of the image;

[0025] Since the image noise in the tableware image library has non-linear characteristics, a non-linear function is introduced to process the pixel values ​​non-linearly, as shown in the following formula:

[0026]

[0027] Where G(n1,n2) represents the optimal pixel, K is a dynamic factor that is dynamically adjusted according to the training results, and x is the x-coordinate of the center point of the image.

[0028] Preferably, the tableware images in the tableware image library are divided into a tableware training set and a tableware test set in a 7:3 ratio.

[0029] S30. Label the images in the tableware training set;

[0030] Preferably, the tableware training set includes a clean tableware set and a dirty tableware set. Based on the proportion of the dirty area to the total area of ​​the tableware and the depth of the dirt color, the degree of dirtiness of the dirty tableware set is divided into nine levels: Level 1, Level 2, Level 3, Level 4, Level 5, Level 6, Level 7, Level 8, and Level 9. The higher the level of dirtiness, the higher the degree of dirtiness.

[0031] Preferably, the labeling includes clean tableware, level 1 dirty tableware, level 2 dirty tableware, level 3 dirty tableware, level 4 dirty tableware, level 5 dirty tableware, level 6 dirty tableware, level 7 dirty tableware, level 8 dirty tableware, and level 9 dirty tableware.

[0032] S40. Construct the T-Faster R-CNN model;

[0033] Preferably, the T-Faster R-CNN includes a clean feature extraction layer, a candidate region layer, a region of interest layer, and a prediction layer;

[0034] Preferably, in step S41, the cleaning feature extraction layer containing a T-SR network is used to extract features to obtain a tableware feature map. The T-SR tableware feature map extraction step includes:

[0035] Enter the image of the tableware;

[0036] The tableware image is convolved to obtain the first convolutional feature map;

[0037] The first convolutional feature map is input into the multi-scale feature extraction module to obtain a multi-scale feature map;

[0038] The multi-scale feature map is input into the T-SE module to obtain a resolution-enhanced tableware feature map;

[0039] The tableware feature map is convolved to output a second convolved feature map.

[0040] The first convolutional feature map and the second convolutional feature map are concatenated to generate a tableware feature map.

[0041] Preferably, a T-SE module is added to the SR network to generate a T-SR network, which enhances image detail restoration. Targeted feature reconstruction is achieved by dynamically adjusting the weights, further improving the reconstruction quality of the network and making the generated high-resolution tableware images more realistic and detailed.

[0042] Preferably, the T-SE module includes:

[0043] The T-SE is used to enhance the resolution of the tableware feature map output by the T-SR network. Specific steps include:

[0044] Input the tableware feature map, and aggregate the tableware features in the spatial dimension through a squeezing operation to generate channel information blocks;

[0045] The channel information block is subjected to an incentive mechanism operation;

[0046] In the operation of the incentive mechanism, the incentive mechanism of each channel is controlled by a channel-dependent self-regulation mechanism that specifically activates the tableware samples learned for each channel.

[0047] The feature mapping of the tableware feature map is reweighted to generate a tableware feature map with enhanced resolution.

[0048] Super-resolution (SR) networks can transform low-resolution images into high-resolution images, thereby improving image quality and enhancing details and sharpness. Therefore, SR networks are chosen for feature extraction. However, due to the risk of overfitting and high computational complexity in practical applications, SR networks are not very effective at processing high-resolution images. Therefore, a Transformer-Search-Effect (T-SE) module is introduced. Adding a T-SE module to the SR network helps to increase the importance of channel features and adaptively adjusts their weights, allowing the SR network to better focus on information useful for the super-resolution task and improve its ability to extract and utilize key channel features. Simultaneously, the T-SR network can automatically learn and emphasize the most relevant and useful channel features, thereby reducing the propagation of redundant information. This helps the network better focus on features that contribute more to super-resolution reconstruction, avoiding wasted computational resources and capacity.

[0049] S42. In the candidate region layer, the feature map of the tableware is processed to obtain a predefined number of candidate boxes that may contain the target;

[0050] Preferably, the original image of the tableware is processed by a T-SR network to extract features, resulting in a tableware feature map.

[0051] The tableware feature map is processed through a candidate region layer to extract candidate bounding boxes, specifically as follows:

[0052] Add three 1×1 convolutional layers to the tableware feature map;

[0053] The first 1×1 convolutional layer is used to adjust the number of channels in the feature layer;

[0054] The second 1×1 convolutional layer is used to determine whether the anchor generated by each anchor point is foreground or background;

[0055] The third 1×1 convolutional layer is used for bounding box regression.

[0056] S43. In the region of interest layer, the candidate boxes are processed, the features of the tableware are extracted, the contents of the prior boxes are classified, and the coordinates of the candidate boxes are adjusted.

[0057] Preferably, the tableware region features are aggregated in the region of interest layer. In order to transform the process of tableware region feature aggregation into a continuous process, the position coordinates of the tableware sampling points are calculated and represented by floating-point numbers. Then, the bilinear interpolation method is used to calculate the feature values ​​of the tableware sampling points based on the feature values ​​of the pixels adjacent to the tableware sampling points.

[0058] S50. The tableware training set is fed into the T-Faster R-CNN model for training;

[0059] S60. Calculate the G-Focal cleaning loss function. If it converges, proceed to step S70; otherwise, proceed to step S50.

[0060] Preferably, the G-Focal cleaning loss function formula is as follows:

[0061]

[0062]

[0063]

[0064] in, G-Focal is the baseline classification loss function, while G-Focal is the cleanliness loss function. Let represent the probability of a positive sample of the i-th tableware category. It is the weight coefficient of the i-th tableware category. β is the modulation coefficient, used to address the imbalance between difficult-to-classify and easily-classify samples of tableware cleanliness; β is an adjustable parameter, β≥0, b i It is the number of tableware items in the i-th tableware category;

[0065] There are ten categories of tableware: clean tableware, Level 1 dirty tableware, Level 2 dirty tableware, Level 3 dirty tableware, Level 4 dirty tableware, Level 5 dirty tableware, Level 6 dirty tableware, Level 7 dirty tableware, Level 8 dirty tableware, and Level 9 dirty tableware. The calculation formulas for each parameter are as follows:

[0066]

[0067]

[0068] Where y∈{±1} represents the positive or negative nature of the tableware sample; it is used to resolve the imbalance between positive and negative tableware samples, where positive samples refer to samples whose tableware cleanliness is accurately predicted and classified, and negative samples refer to samples whose tableware cleanliness is inaccurately predicted and classified.

[0069] S70. End training and obtain the trained deep network T-Faster R-CNN model.

[0070] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0071] 1. Denoising images from a tableware image library is performed using a nonlinear bilateral Wiener filtering method. This method considers spatial distance and gray-level differences, exhibiting strong edge-preserving capabilities. It can better preserve edge and detail information in tableware images, resulting in clearer and more detailed images compared to traditional Wiener filtering. This is crucial for the effective identification of tableware cleanliness. Furthermore, since the image noise in the tableware image library has nonlinear characteristics, introducing a nonlinear adjustment weight factor can effectively handle tableware images with complex nonlinear noise. It can also better handle noise crossover situations, enhancing the denoising effect of the tableware image library images.

[0072] 2. When constructing the T-Faster R-CNN model, compared with the traditional Faster R-CNN model, the T-Faster R-CNN model introduces the T-SR network for feature extraction, which has a faster detection speed and higher detection accuracy. Furthermore, the T-SE module is added to the T-SR network for tableware feature extraction, which can enhance the restoration of tableware image details. By dynamically adjusting the weights, targeted tableware feature reconstruction can be achieved, further improving the reconstruction quality of the network. This makes the generated high-resolution tableware images more realistic and delicate, while reducing redundant information, better capturing global and local details of tableware, and avoiding wasting computing resources and capacity.

[0073] 3. The G-Focal cleaning loss function is used to calculate the loss optimization model. By comprehensively optimizing the G-Focal method for calculating the loss optimization model, the model pays more attention to samples that are difficult to classify, thereby improving the overall detection performance. It can also more accurately measure the overlap of tableware image boxes, which helps improve the accuracy and stability of target localization. The G-Focal cleaning loss function can consider both classification and localization needs, performing comprehensive target detection optimization, thereby improving the overall detection performance and accuracy, and enhancing the model's robustness and generalization ability. Attached Figure Description

[0074] Figure 1 This is a schematic diagram of the structure of the present invention;

[0075] Figure 2 This is a flowchart of the T-Faster R-CNN algorithm according to an embodiment of the present invention;

[0076] Figure 3 This is a schematic diagram of the structure of the T-SR network according to an embodiment of the present invention;

[0077] Figure 4 This is an illustration of the effect of tableware cleanliness recognition provided in an embodiment of the present invention;

[0078] Figure 5 A comparison chart of model recognition results provided in an embodiment of the present invention. Detailed Implementation

[0079] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0080] Please see Figures 1 to 5 This invention provides a dishwasher cleanliness detection method using T-Faster R-CNN, the technical solution of which is as follows:

[0081] Specifically, refer to Figure 1 As shown, this invention provides a dishwasher cleanliness detection method using T-Faster R-CNN. This method can be executed by a device, which can be implemented by software and / or hardware. In a specific implementation, it includes steps S10 to S80.

[0082] Specifically, S10. Construct a tableware image library; S20. Denoise the images in the tableware image library using a nonlinear bilateral Wiener filtering method; classify the denoised tableware image library into a tableware training set and a tableware test set; S30. Label the images in the tableware training set; S40. Construct a T-Faster R-CNN model, wherein a T-SR network is used for feature extraction; S50. Feed the tableware training set into the T-Faster R-CNN model for training to obtain a trained T-Faster R-CNN model; S60. Calculate the loss using the G-Focal cleaning loss function to optimize the model and obtain the final T-Faster R-CNN model; S70. Test the final T-Faster R-CNN model using the tableware test set; S80. Output the detection results to obtain the cleanliness of the tableware cleaned by the dishwasher.

[0083] Specifically, in this embodiment of the invention, 8,000 images of tableware are first crawled to train the model. The crawled tableware images are then divided into a test set and a validation set in a 7:3 ratio. After completing the validation on the test set, a classification and recognition model is constructed. By inputting tableware images, the model outputs detection results, which include clean tableware, level 1 dirty tableware, level 2 dirty tableware, level 3 dirty tableware, level 4 dirty tableware, level 5 dirty tableware, level 6 dirty tableware, level 7 dirty tableware, level 8 dirty tableware, or level 9 dirty tableware.

[0084] S10. Build a tableware image library;

[0085] Specifically, S10 includes crawling tableware images to obtain the tableware image library.

[0086] S20. Classify and preprocess the tableware image library, the main steps of which include: S21. Denoise the images in the tableware image library; S22. Divide the denoised tableware image library into a tableware training set and a tableware test set.

[0087] Specifically, S21. Denoising the images in the tableware image library using a nonlinear bilateral Wiener filtering method, wherein the nonlinear bilateral Wiener filtering method includes:

[0088] The local mean μ around each pixel of image C in the tableware image library C and variance Make an estimate:

[0089]

[0090]

[0091] in, Let be the neighborhood of a pixel in the tableware image C, M×S be the size of this neighborhood, and u(n1,n2) be the pixels n1 and n2 in the local neighborhood. The pixel-by-pixel estimation formula for the tableware is as follows:

[0092]

[0093] Where Px represents a pixel, and w(n1,n2) are two-sided weighted coefficients. The variance of the noise is represented by w(n1,n2) and... The formula is as follows:

[0094] w(n1,n2)=w s (n1,n2)w r (n1,n2)

[0095]

[0096]

[0097]

[0098] Where w(n1,n2) is determined by the spatial proximity factor w s (n1,n2) and brightness similarity factor w r The product of (n1, n2) consists of the parameter σ. s and σ r Each controls w s and w r The degree of attenuation, where (x,y) is the center point of the image;

[0099] Since the image noise in the tableware image library has non-linear characteristics, a non-linear function is introduced to process the pixel values ​​non-linearly, as shown in the following formula:

[0100]

[0101] Where G(n1,n2) represents the optimal pixel, K is a dynamic factor that is dynamically adjusted according to the training results, and x is the x-coordinate of the center point of the image.

[0102] Specifically, S22. Divide the tableware images in the tableware image library into a tableware training set and a tableware test set in a 7:3 ratio.

[0103] S30. Label the images in the tableware training set;

[0104] Specifically, the tableware training set includes a clean tableware set and a dirty tableware set. Based on the proportion of the dirty area to the total area of ​​the tableware and the depth of the dirt color, the dirty tableware set is divided into nine levels of dirtiness: Level 1, Level 2, Level 3, Level 4, Level 5, Level 6, Level 7, Level 8, and Level 9. The higher the level of dirtiness, the greater the degree of dirtiness.

[0105] Specifically, the labeling includes clean tableware, level 1 dirty tableware, level 2 dirty tableware, level 3 dirty tableware, level 4 dirty tableware, level 5 dirty tableware, level 6 dirty tableware, level 7 dirty tableware, level 8 dirty tableware, and level 9 dirty tableware.

[0106] S40. Construct the T-Faster R-CNN model;

[0107] Specifically, referring to Figure 2 As shown, the T-Faster R-CNN includes a clean feature extraction layer, a candidate region layer, a region of interest layer, and a prediction layer;

[0108] Specifically, S41. The cleaning feature extraction layer includes feature extraction using a T-SR network to obtain a tableware feature map. The T-SR tableware feature map extraction step includes:

[0109] Enter the image of the tableware;

[0110] The tableware image is convolved to obtain the first convolutional feature map;

[0111] The first convolutional feature map is input into the multi-scale feature extraction module to obtain a multi-scale feature map;

[0112] The multi-scale feature map is input into the T-SE module to obtain a resolution-enhanced tableware feature map;

[0113] The tableware feature map is convolved to output a second convolved feature map.

[0114] The first convolutional feature map and the second convolutional feature map are concatenated to generate a tableware feature map.

[0115] Specifically, a T-SE module is added to the SR network to generate a T-SR network, which enhances image detail restoration. Targeted feature reconstruction is achieved by dynamically adjusting weights, further improving the network's reconstruction quality and making the generated high-resolution tableware images more realistic and detailed.

[0116] Specifically, the T-SE module includes:

[0117] The T-SE is used to enhance the resolution of the tableware feature map output by the T-SR network. Specific steps include:

[0118] Input the tableware feature map, and aggregate the tableware features in the spatial dimension through a squeezing operation to generate channel information blocks;

[0119] The channel information block is subjected to an incentive mechanism operation;

[0120] In the operation of the incentive mechanism, the incentive mechanism of each channel is controlled by a channel-dependent self-regulation mechanism that specifically activates the tableware samples learned for each channel.

[0121] The feature mapping of the tableware feature map is reweighted to generate a tableware feature map with enhanced resolution.

[0122] Specifically, Regression-Survey (SR) networks can transform low-resolution images into high-resolution images, thereby improving image quality and enhancing details and sharpness. Therefore, SR networks are chosen for feature extraction. However, due to the risk of overfitting and high computational complexity in practical applications, SR networks are not very effective at processing high-resolution images. Therefore, a Transformer-Search-Effect (T-SE) module is introduced. Adding a T-SE module to the SR network helps to increase the importance of channel features and adaptively adjusts their weights, allowing the SR network to better focus on information useful for the super-resolution task and improve its ability to extract and utilize key channel features. Simultaneously, the T-SR network can automatically learn and emphasize the most relevant and useful channel features, thereby reducing the propagation of redundant information. This helps the network better focus on features that contribute more to super-resolution reconstruction, avoiding wasted computational resources and capacity.

[0123] S42. In the candidate region layer, the feature map of the tableware is processed to obtain a predefined number of candidate boxes that may contain the target;

[0124] Specifically, the original image of the tableware is processed by a T-SR network to extract features, resulting in a tableware feature map;

[0125] The original image of the tableware is processed by a T-SR network to extract features, resulting in a tableware feature map.

[0126] The tableware feature map is processed through a candidate region layer to extract candidate bounding boxes, specifically as follows:

[0127] Add three 1×1 convolutional layers to the tableware feature map;

[0128] The first 1×1 convolutional layer is used to adjust the number of channels in the feature layer;

[0129] The second 1×1 convolutional layer is used to determine whether the anchor generated by each anchor point is foreground or background;

[0130] The third 1×1 convolutional layer is used for bounding box regression.

[0131] S43. In the region of interest layer, the candidate boxes are processed, the features of the tableware are extracted, the contents of the prior boxes are classified, and the coordinates of the candidate boxes are adjusted.

[0132] Specifically, the tableware region features are aggregated in the region of interest layer. To transform the tableware region feature aggregation process into a continuous process, the position coordinates of the tableware sampling points are calculated and represented by floating-point numbers. Then, the bilinear interpolation method is used to calculate the feature values ​​of the tableware sampling points based on the feature values ​​of the pixels adjacent to the tableware sampling points.

[0133] S50. The tableware training set is fed into the T-Faster R-CNN model for training;

[0134] S60. Calculate the G-Focal cleaning loss function. If it converges, proceed to step S70; otherwise, proceed to step S50.

[0135] Specifically, the G-Focal cleaning loss function formula is as follows:

[0136]

[0137]

[0138]

[0139] in, G-Focal is the baseline classification loss function, while G-Focal is the cleanliness loss function. Let represent the probability of a positive sample of the i-th tableware category. It is the weight coefficient of the i-th tableware category. β is the modulation coefficient, used to address the imbalance between difficult-to-classify and easily-classify samples of tableware cleanliness; β is an adjustable parameter, β≥0, b i It is the number of tableware items in the i-th tableware category;

[0140] There are ten categories of tableware: clean tableware, Level 1 dirty tableware, Level 2 dirty tableware, Level 3 dirty tableware, Level 4 dirty tableware, Level 5 dirty tableware, Level 6 dirty tableware, Level 7 dirty tableware, Level 8 dirty tableware, and Level 9 dirty tableware. The calculation formulas for each parameter are as follows:

[0141]

[0142]

[0143] Where y∈{±1} represents the positive or negative nature of the tableware sample; it is used to resolve the imbalance between positive and negative tableware samples, where positive samples refer to samples whose tableware cleanliness is accurately predicted and classified, and negative samples refer to samples whose tableware cleanliness is inaccurately predicted and classified.

[0144] Calculate the G-Focal cleaning loss function. If it converges, end the training and output the trained model; otherwise, proceed to step S50.

[0145] By using the G-Focal cleaning loss function to calculate the loss optimization model, global loss optimization can be achieved. This can reduce the loss weight of easily classified tableware samples, improve the model's learning ability for difficult-to-classify tableware samples, and effectively improve the accuracy of predicting tableware cleanliness. At the same time, it takes into account the position, scale, and shape information of the tableware target box, and can achieve precise location of dirt spots on dirty tableware.

[0146] S70. End training and obtain the trained deep network T-Faster R-CNN model.

[0147] Specifically, the result refers to the cleanliness of the dishes cleaned by the dishwasher. For example... Figure 4 These are a set of results images used to detect the cleanliness of plates using the T-Faster R-CNN model. The first image shows a plate with a large, darkly colored dirty area; compared to the training image set, this plate is identified as level nine dirty. The second image shows a plate after one wash; the dirty area has decreased, and the color is stronger; compared to the training image set, this plate is identified as level four dirty. The third image shows a plate after three washes; the plate is now clean, and compared to the training image set, this plate is identified as clean.

[0148] This embodiment uses the T-Faster R-CNN model to detect the cleanliness of tableware. Figure 3 This paper proposes a method to improve the accuracy of tableware detection by utilizing a T-SR network in the feature extraction layer. The main function of the SR network is tableware feature extraction. However, due to the risk of overfitting and high computational complexity in practical applications, the SR network is not effective in processing high-resolution tableware images. Therefore, a T-SE module is introduced. This module is added to the SR network, and its weights are adaptively adjusted. The T-SR network automatically learns and emphasizes the most relevant and useful channel features, thereby reducing the propagation of redundant information. To illustrate the feasibility of this approach, [the following text is incomplete and requires further context]. Figure 5 The experiment compares the Faster R-CNN model and the T-Faster R-CNN model on different numbers of tableware images.

[0149] Figure 5The results demonstrate that the T-Faster R-CNN model exhibits excellent performance at different stages. As the number of training tableware images increases, the detection accuracy of the T-Faster R-CNN model also increases. Compared with the Faster R-CNN model, it can be seen that the accuracy of the T-Faster R-CNN model is at least 20% higher than that of Faster R-CNN at the same stage, thus showing that the T-SR network can significantly enhance the performance of the T-Faster R-CNN model in detecting the cleanliness of tableware.

[0150] Working Principle: This paper proposes a dishwasher cleanliness detection method using T-Faster R-CNN, including: constructing a tableware image library; denoising the images in the tableware image library using a nonlinear bilateral Wiener filtering method; classifying the denoised tableware image library into a tableware training set and a tableware test set; labeling the images in the tableware training set; constructing a T-Faster R-CNN model, wherein a T-SR network is used for feature extraction; feeding the tableware training set into the T-Faster R-CNN model for training, obtaining a trained T-Faster R-CNN model; calculating the loss using the G-Focal cleaning loss function to optimize the model, obtaining the final T-Faster R-CNN model; testing the final T-Faster R-CNN model using the tableware test set, and outputting the detection results to obtain the cleanliness of the tableware washed by the dishwasher.

[0151] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for detecting the cleanliness of a dishwasher using T-Faster R-CNN, characterized in that, include: S10. Build a tableware image library; S20. Classify and preprocess the tableware image library; S21. Denoise the images in the tableware image library; The denoising process employs a nonlinear bilateral Wiener filtering method to obtain tableware images with clearer edge and detail information; S22. Divide the denoised tableware image library into a tableware training set and a tableware test set; S30. Label the images in the tableware training set; S40. Construct the T-Faster R-CNN model; The T-Faster R-CNN includes a clean feature extraction layer, a candidate region layer, a region of interest layer, and a prediction layer; S41. The cleaning feature extraction layer containing the T-SR network is used to extract features to obtain a tableware feature map; The T-SR network serves as the feature extraction module of the cleaning feature extraction layer. By adding a T-SE module to the T-SR network, it can adapt to feature extraction of tableware with different resolutions. S42. In the candidate region layer, the tableware feature map is processed to obtain a predefined number of candidate boxes containing the target; S43. In the region of interest layer, process the candidate boxes and extract the features of the tableware; S50. Train the T-Faster R-CNN model using the tableware training set; S60. Calculate the G-Focal cleaning loss function. If it converges, proceed to step S70; otherwise, proceed to step S50. S70. End training and obtain the trained deep network T-Faster R-CNN model.

2. The dishwasher cleanliness detection method using T-Faster R-CNN according to claim 1, characterized in that, S10. Constructing a tableware image library includes: At least 8,000 tableware images were crawled from an online database to form the tableware image library.

3. The dishwasher cleanliness detection method using T-Faster R-CNN according to claim 1, characterized in that, S21. The nonlinear bilateral Wiener filtering method includes: The local mean μ around each pixel of image C in the tableware image library C and variance Make an estimate: in, Let be the neighborhood of a pixel in the tableware image C, M×S be the size of this neighborhood, and u(n1,n2) be the pixels n1 and n2 in the local neighborhood. The pixel-by-pixel estimation formula for the tableware is as follows: Where Px represents a pixel, and w(n1,n2) are two-sided weighted coefficients. The variance of the noise is represented by w(n1,n2) and The formula is as follows: w(n1,n2) = w s (n1,n2)w r (n1,n2) where w(n1, n2) is composed of a spatial proximity factor w s (n1, n2) and a luminance similarity factor w r (n1, n2), and parameters σ s and σ r control the attenuation degree of w s and w r respectively, (x, y) is the center point of the image. Since the image noise in the tableware image library has non-linear characteristics, a non-linear function is introduced to process the pixel values ​​non-linearly, as shown in the following formula: Where G(n1,n2) represents the optimal pixel, K is a dynamic factor that is dynamically adjusted according to the training results, and x is the x-coordinate of the center point of the image.

4. The dishwasher cleanliness detection method using T-Faster R-CNN according to claim 1, characterized in that, Step S22. Dividing the tableware image library into a tableware training set and a tableware test set, including: The tableware images in the tableware image library are divided into a tableware training set and a tableware test set in a 7:3 ratio.

5. The dishwasher cleanliness detection method using T-Faster R-CNN according to claim 1, characterized in that, Step S30. Labeling the images in the tableware training set includes: The tableware training set includes a clean tableware set and a dirty tableware set. Based on the proportion of the dirty area to the total area of ​​the tableware and the depth of the dirt color, the dirty tableware set is divided into nine levels of dirtiness: Level 1, Level 2, Level 3, Level 4, Level 5, Level 6, Level 7, Level 8 and Level 9. The labeling includes clean tableware, level 1 dirty tableware, level 2 dirty tableware, level 3 dirty tableware, level 4 dirty tableware, level 5 dirty tableware, level 6 dirty tableware, level 7 dirty tableware, level 8 dirty tableware, and level 9 dirty tableware.

6. The dishwasher cleanliness detection method using T-Faster R-CNN according to claim 1, characterized in that, The T-SR extraction of tableware feature maps includes: Enter the image of the tableware; The tableware image is convolved to obtain the first convolutional feature map; The first convolutional feature map is input into the multi-scale feature extraction module to obtain a multi-scale feature map; The multi-scale feature map is input into the T-SE module to obtain a resolution-enhanced tableware feature map; The tableware feature map is convolved to output a second convolved feature map. The first convolutional feature map and the second convolutional feature map are concatenated to generate a tableware feature map.

7. The dishwasher cleanliness detection method using T-Faster R-CNN according to claim 1, characterized in that, The T-SE module includes: The T-SE is used to enhance the resolution of the tableware feature map output by the T-SR network. Specific steps include: Input the tableware feature map, and aggregate the tableware features in the spatial dimension through a squeezing operation to generate a channel information block; The channel information block is subjected to an incentive mechanism operation; In the operation of the incentive mechanism, the incentive mechanism of each channel is controlled by a channel-dependent self-regulation mechanism that specifically activates the tableware samples learned for each channel. The feature mapping of the tableware feature map is reweighted to generate a tableware feature map with enhanced resolution.

8. The dishwasher cleanliness detection method using T-Faster R-CNN according to claim 1, characterized in that, Step S42. In the candidate region layer, the feature map of the tableware is processed to obtain a predefined number of candidate boxes containing the target, including: The original image of the tableware is processed by a T-SR network to extract features, resulting in a tableware feature map. The tableware feature map is processed through a candidate region layer to extract candidate bounding boxes, specifically as follows: Add three 1×1 convolutional layers to the tableware feature map; The first 1×1 convolutional layer is used to adjust the number of channels in the feature layer; The second 1×1 convolutional layer is used to determine whether the anchor generated by each anchor point is foreground or background; The third 1×1 convolutional layer is used for bounding box regression.

9. The dishwasher cleanliness detection method using T-Faster R-CNN according to claim 1, characterized in that, S43. In the region of interest layer, the candidate boxes are processed, features of the tableware are extracted, and the Fast R-CNN module is used to classify the content of the prior boxes and adjust the coordinates of the candidate boxes, including: In the region of interest layer, tableware region features are aggregated. To transform the tableware region feature aggregation process into a continuous process, the position coordinates of the tableware sampling points are calculated and represented by floating-point numbers. Then, bilinear interpolation is used to calculate the feature values ​​of the tableware sampling points based on the feature values ​​of the pixels adjacent to the tableware sampling points.

10. The dishwasher cleanliness detection method using T-Faster R-CNN according to claim 1, characterized in that, The formula for the G-Focal cleaning loss function in S60 is as follows: in, G-Focal is the baseline classification loss function, while G-Focal is the cleanliness loss function. Let represent the probability of a positive sample of the i-th tableware category. It is the weight coefficient of the i-th tableware category. β is the modulation coefficient, used to address the imbalance between difficult-to-classify and easily-classify samples of tableware cleanliness; β is an adjustable parameter, β≥0, b i It is the number of tableware items in the i-th tableware category; There are ten categories of tableware: clean tableware, Level 1 dirty tableware, Level 2 dirty tableware, Level 3 dirty tableware, Level 4 dirty tableware, Level 5 dirty tableware, Level 6 dirty tableware, Level 7 dirty tableware, Level 8 dirty tableware, and Level 9 dirty tableware. The calculation formulas for each parameter are as follows: Where y∈{±1} represents the positive or negative nature of the tableware sample; it is used to resolve the imbalance between positive and negative tableware samples, where positive samples refer to samples whose tableware cleanliness is accurately predicted and classified, and negative samples refer to samples whose tableware cleanliness is inaccurately predicted and classified.

Images