Cleaning and testing methods and related devices, electronic equipment and storage media

Abstract

This application discloses a cleaning detection method, related apparatus, electronic device, and storage medium. The cleaning detection method includes: detecting a first image of a target object to obtain a first detection result; the first detection result including at least a first region of the target object; extracting a first target image of the target object from the first image based on the first region; and performing a comparative detection between a reference image and the first target image to obtain a second detection result; the reference image includes a reference object sample corresponding to the target object, which may or may not require cleaning, and the second detection result includes whether the target object needs cleaning. This solution can improve the accuracy of cleaning detection.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a cleaning and detection method and related apparatus, electronic equipment and storage medium. Background Technology

[0002] In many scenarios, cleaning inspection is particularly important for improving cleaning efficiency. For example, in vehicle washing, analyzing whether a vehicle needs cleaning can help improve cleaning efficiency; or in glass exterior wall cleaning, analyzing whether the glass exterior wall needs cleaning can help improve the work efficiency of relevant personnel, and so on.

[0003] Currently, cleaning inspection generally uses the method of acquiring and identifying target images to determine the cleaning result. Taking vehicle washing as an example, cameras are used to identify vehicles entering the cleaning area, and the duration of vehicle stay in the cleaning area is calculated to determine whether the vehicle has been cleaned. However, as the number of vehicles increases, the dwell time of vehicles is affected by external factors, making it difficult to guarantee the accuracy of cleaning inspection. In view of this, how to improve the accuracy of cleaning inspection has become an urgent problem to be solved. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a cleaning and testing method, related apparatus, electronic equipment, and storage medium to improve the accuracy of cleaning and testing.

[0005] To address the aforementioned technical problems, a first aspect of this application provides a cleaning detection method, comprising: performing detection on a first image of a target object to obtain a first detection result; wherein the first detection result includes at least a first region of the target object; extracting a first target image of the target object from the first image based on the first region; and performing comparative detection based on a reference image and the first target image to obtain a second detection result; wherein the reference image includes a reference object sample corresponding to the target object, the reference object sample includes samples that do not require cleaning or samples that require cleaning, and the second detection result includes whether the target object needs cleaning.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a cleaning detection apparatus, comprising a detection module, an extraction module, and a comparison module. A document acquisition module is used to acquire a document set. The detection module performs detection based on a first image captured of a target object to obtain a first detection result, which includes at least a first region of the target object. The extraction module extracts a first target image of the target object from the first image based on the first region. The comparison module performs comparative detection based on a reference image and the first target image to obtain a second detection result. The reference image includes reference object samples corresponding to the target object, which include samples that do not require cleaning or samples that require cleaning. The second detection result includes whether the target object needs cleaning.

[0007] To address the aforementioned technical problems, a third aspect of this application provides an electronic device, including a memory and a processor coupled to each other. The memory stores program instructions, and the processor executes the program instructions to implement the cleaning and detection method described in the first aspect.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the cleaning and detection method described in the first aspect.

[0009] The above scheme detects a first image of the target object to obtain a first detection result, which includes at least a first region of the target object. Based on this first region, a first target image of the target object is extracted from the first image. Then, a comparison detection is performed between a reference image and the first target image to obtain a second detection result. The reference image includes reference object samples, which may be samples requiring cleaning or those requiring cleaning. The second detection result indicates whether the target object requires cleaning. On one hand, by detecting the image of the target object and determining its first region, and then extracting the first target image based on this region, the accuracy of extracting the first target region is improved. On the other hand, after determining the first target region, the comparison detection between the reference image and the first target image, due to the reference image's selectivity, improves the reliability of the detection result and further enhances the applicability of the cleaning detection. Therefore, the accuracy of the cleaning detection is improved. Attached Figure Description

[0010] Figure 1 This is a schematic flowchart of an embodiment of the cleaning and testing method of this application;

[0011] Figure 2 This is a schematic diagram of obtaining the first detection result from the first image detection;

[0012] Figure 3 This is a schematic diagram of the framework of an embodiment of an image detection model;

[0013] Figure 4 This is a schematic diagram of the network structure of an embodiment of the first extraction network;

[0014] Figure 5 This is a schematic diagram of a process of an embodiment of the cleaning and testing method of this application;

[0015] Figure 6 This is a schematic diagram of the framework of an embodiment of the cleaning and testing device of this application;

[0016] Figure 7 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;

[0017] Figure 8 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0018] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0019] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0020] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper means two or more.

[0021] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the cleaning and testing method of this application.

[0022] Specifically, this may include the following steps:

[0023] Step S11: Detect the first image of the target object to obtain the first detection result.

[0024] In an implementation scenario, the target object can be selected based on the specific application. For example, the target object could be a vehicle, the hull of a ship, a glass facade, a kitchen cabinet, etc. The target object can be selected based on the actual situation, and no specific limitations are made here.

[0025] In one implementation scenario, the first image can be an image captured by the imaging device of the target object. When the target object is a vehicle, the first image can be an image captured by the imaging device of the vehicle; when the target object is a glass exterior wall, the first image can be an image captured by the imaging device of the glass exterior wall. The first image can be determined according to the actual situation and is not specifically limited here.

[0026] In this embodiment, the first detection result includes at least a first region of the target object. It should be noted that the first detection result may only include the first region of the target object, or it may include the first region where both cleaning agent and the target object are detected, or it may include a second region where the cleaning personnel are detected in the first image. The first detection result can be determined based on the detection of the first image, and will not be elaborated further here. Furthermore, the first region of the target object may include the complete region of the target object, and the complete region of the target object may be the same size as the image region corresponding to the first image; of course, it may also include a partial region of the target object. For example, please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic diagram illustrating the first detection result obtained from the first image detection, as shown below. Figure 2 As shown, the target object is a vehicle. A first detection result is obtained by detecting the first image. The first detection result includes a first region of the target object, which includes the complete vehicle region 21 and a partial vehicle region 22. The first detection result also includes the vehicle's cleaning agent 23 and a second region 24 representing the cleaning personnel. Therefore, the first region of the target object can be determined according to the actual situation, and no specific limitation is made here.

[0027] In a specific implementation scenario, the cleaning agent used to clean the target object will vary depending on the target object. For example, if the target object is a vehicle, the cleaning agent used to clean the target object is a water jet or water mist; if the target object is a glass exterior wall, the cleaning agent used to clean the target object is a foam cleaner. The cleaning agent used to clean the target object can be selected according to the actual situation, and no specific limitation is made here.

[0028] In one implementation scenario, the first detection result includes a classification detection result. The classification detection result includes whether cleaning agent used to clean the target object is detected in the first image, and it is used to analyze whether the target object has been cleaned. To obtain the first detection result, global image features of the first image can be extracted first. This can be done using the LBP (Local Binary Patterns) algorithm, the HOG (Histogram of Oriented Gradients) algorithm, or a feature extraction network. The extraction method can be chosen based on the actual situation and is not specifically limited here. After obtaining the global image features, target detection can be performed based on these features to obtain a target detection result, which includes at least the first region of the target object. Classification detection can then be performed based on the global image features to obtain a classification detection result. This can be done using a classification detection network; for example, a CNN (convolutional neural network) can be used. The classification detection network can be chosen based on the actual situation and is not specifically limited here. Based on the target detection result and the classification detection result, the first detection result is obtained. The above method obtains classification detection results by detecting the first image. In this process, a multi-task approach is used for detection, which reduces the complexity of the algorithm and the running time, thereby improving the efficiency of cleaning and detection.

[0029] In a specific implementation scenario, ResNet or MobileNet series networks can also be selected to extract global image features. The choice depends on the required accuracy and running speed, and is not limited here. Based on this, a YOLOv3 detection structure can be selected to perform target detection based on global image features. Multiple target detection structures can be used, or only one can be used, to obtain target detection results. The target detection results include the category and location of each target region. The specific location can be defined by the vertex or center position of the bounding rectangle, etc., and the category can be the complete region or part region of the target object. Furthermore, two fully connected layers and one sigmoid activation layer can be used to classify and detect the global image features, obtaining classification detection results. The classification detection results include whether cleaning agent used to clean the target object is detected in the first image.

[0030] Please see Figure 3 , Figure 3This is a schematic diagram of the framework of an embodiment of an image detection model. The image detection model includes a second extraction network, and a target detection network and a classification detection network, both connected to the second extraction network. The first detection result can be obtained by the image detection model, such as... Figure 3 As shown, a first image is input, a second extraction network is used to extract global image features of the first image, an object detection network is used to perform object detection to obtain object detection results, and a classification detection network is used to perform classification detection to obtain classification detection results. This method, by using an image detection model to detect the first image and determine the first detection result, helps to improve the accuracy of the first detection result and further improve the cleaning detection accuracy.

[0031] In a specific implementation scenario, to improve the accuracy of the image detection model, several sample images can be collected in advance. These sample images contain sample objects belonging to the same category as the target object, and are labeled with the target object's first region, the presence of cleaning agents used for cleaning the sample objects, and the presence of cleaning personnel in the sample images' second regions. Based on this, feature extraction can be performed on the sample images using the image detection model, followed by target detection and classification detection to obtain target prediction results and classification prediction results. A first prediction result is obtained based on these two results. The difference between the labeled content on the sample images and the first prediction result is then obtained, and the network parameters of the image detection model are adjusted based on this difference. Specifically, the specific methods for measuring the difference can be found in loss functions such as the intersection-union ratio (IU), and the methods for adjusting the network parameters can be found in optimization methods such as gradient descent, which will not be elaborated upon here.

[0032] Step S12: Based on the first region, extract the first target image of the target object in the first image.

[0033] In one implementation scenario, since the first detection result includes at least the first region of the target object, the first region can be extracted using a deep learning network model to obtain the first target image. The deep learning network model can be RCNN (Region based Convolutional Neural Network). The deep learning network model can be selected according to the actual situation, and no specific limitation is made here.

[0034] In a specific implementation scenario, due to differences in the integrity of the first region of the target object, the integrity of the extracted first target image will also differ. For example, please refer to [link to relevant documentation]. Figure 2When the first region is the complete vehicle region 21, the corresponding extracted first target image is the complete vehicle image within the region; when the first region is a partial vehicle region 22, the corresponding extracted first target image is the partial vehicle image, such as the tires. The completeness of the first target image can be determined based on the completeness of the first region, and is not specifically limited here.

[0035] Step S13: Perform a comparative detection based on the reference image and the first target image to obtain a second detection result.

[0036] In this embodiment of the disclosure, the reference image includes reference object samples belonging to the same category as the target object. For example, when the target object is a vehicle, the category of the reference object samples in the reference image is also vehicle; when the target object is a road barrier, the category of the reference object samples in the reference image is also road barrier. The category of the reference object samples in the reference image can be selected according to the category of the target object, and is not specifically limited here. Furthermore, the reference object samples in the reference image include samples that do not require cleaning or samples that require cleaning; that is, the reference image includes several reference image samples, and the cleaning attribute of the reference image samples is either requiring cleaning or not requiring cleaning. It should be noted that the completeness of the reference object samples in the reference image can be determined based on the completeness of the first target image. For example, if the target object is a vehicle and the first target image is a vehicle tire, then the reference image is also a vehicle tire, and the completeness of the reference image can be determined based on the completeness of the first target image, and is not specifically limited here.

[0037] In one implementation scenario, to obtain the second detection result, the similarity between the reference image and the first target image can be calculated using an average hash algorithm. Specifically, the reference image and the first target image can be scaled up first, that is, the size of the two images is unified, and the scaled images are converted into single-channel grayscale images. The pixel mean is calculated, and then the similarity between the two images is obtained based on the pixel mean fingerprint. Finally, the second detection result is obtained based on the similarity.

[0038] In another implementation scenario, differing from the aforementioned method of obtaining the second detection result, to further improve the efficiency of obtaining the second detection result, first image features of the reference image and second image features of the first target image can be extracted. Specifically, image features can be extracted using a feature extraction model, which can be a CNN (convolutional neural network) or an RNN (recurrent neural network). The feature extraction model can be selected according to the actual situation, without specific limitations here. After obtaining the first and second image features, a first feature distance between the first and second image features can be further obtained. It should be noted that the first feature distance represents the feature vector distance between the first and second image features. Specifically, the first feature distance can be calculated using cosine similarity or Euclidean distance. The method of obtaining the first feature distance can be selected according to the actual situation, without specific limitations here. Finally, based on the first feature distance and whether the reference object sample in the reference image needs cleaning, the second detection result is obtained. The above method, by extracting features from the reference image and the first target image respectively and calculating the distance between the image features, and then determining the second detection result, helps to improve the accuracy of the second detection result, that is, to improve the accuracy of cleaning detection.

[0039] In a specific implementation scenario, in response to the fact that a reference object sample in a reference image does not require cleaning and the first feature distance is greater than a first threshold, the second detection result is determined to include a target object that needs cleaning; conversely, in response to the fact that a reference object sample in a reference image does not require cleaning and the first feature distance is not greater than the first threshold, the second detection result is determined to include a target object sample that does not require cleaning. The first threshold can be calculated from the reference images. For example, two reference images are obtained. It should be noted that the reference images contain reference object samples corresponding to the target object, that is, the target object and the reference object samples have the same cleaning attributes, and the reference object samples in the reference images are samples that do not require cleaning. Reference image features are extracted from the reference images respectively, and the reference distance between the reference images is obtained. The reference distance can be set as the first threshold. Alternatively, multiple reference images can be obtained, and multiple reference distances can be determined. The average of the multiple reference distances is set as the first threshold. The first threshold can be set according to the actual situation and is not specifically limited here.

[0040] In another specific implementation scenario, in response to the requirement that a reference object sample in the reference image needs cleaning and the first feature distance is greater than a second threshold, the second detection result is determined to include the target object that does not need cleaning; conversely, in response to the requirement that a reference object sample in the reference image needs cleaning and the first feature distance is not greater than the second threshold, the second detection result is determined to include the target object that needs cleaning. The second threshold can be calculated from the reference images. For example, two reference images are obtained. It should be noted that the reference images contain reference object samples corresponding to the target object, and one of the reference object samples in the reference images is a sample that does not need cleaning, while the other is a sample that needs cleaning. Reference image features are extracted from the reference images respectively, and the reference distance between the reference images is obtained. The reference distance can be set as the second threshold. Alternatively, multiple reference images can be obtained, including images where the reference object sample does not need cleaning and images where the reference object sample needs cleaning. The reference distance between the features of the image where the reference object sample does not need cleaning and the features of the image where the reference object sample needs cleaning is obtained respectively, thereby determining multiple reference distances. The average of the multiple reference distances is set as the second threshold. The second threshold can be set according to the actual situation and is not specifically limited here.

[0041] In one implementation scenario, the first image features and the second image features are extracted by a first extraction network. Before extracting the first image features and the second image features based on the first extraction network, sample image objects can be obtained first. The sample image pairs contain the first sample image and the second sample image. The first sample object in the first sample image and the second sample object in the second sample image belong to the same category. The first sample image and the second sample image have the same or different cleaning attributes. The cleaning attribute represents whether cleaning is required. For example, if the first sample image and the second sample image have the same cleaning attribute, that is, both sample images have the cleaning attribute of requiring cleaning, or both sample images have the cleaning attribute of not requiring cleaning; if the first sample image and the second sample image have different cleaning attributes, that is, one sample image has the cleaning attribute of requiring cleaning and the other does not require cleaning. Based on this, the first sample image features of the first sample image and the second sample image features of the second sample image are extracted based on the first extraction network. Then, based on the sample distance between the first sample image features and the second sample image features, the network loss is obtained. And when the first sample image and the second sample image have the same cleaning attribute, the network loss is positively correlated with the sample distance. For example, the loss function can be expressed as follows:

[0042]

[0043] Z i Z jThese are a pair of sample images with the same cleaning properties. `sim` represents similarity calculation, and the result can be determined using cosine similarity or Euclidean distance; no specific limitation is made here. Furthermore, `τ` is a temperature parameter that can be used to control the degree of difference. [k≠i] The loss function is a confidence function; that is, the function result is 1 when k is not equal to i, and 0 otherwise. Other calculation methods can also be chosen to determine the loss function, which are not specifically limited here. Conversely, when the first sample object and the second sample image have different cleaning attributes, the network loss is negatively correlated with the sample distance. That is, the optimization direction of the first extraction network is to maximize this distance. Based on this, the network parameters of the first extraction network are adjusted according to the network loss. By training the first extraction network in the above way, the accuracy of the first extraction network can be improved during use, further enhancing the accuracy of cleaning detection.

[0044] Please see Figure 4 , Figure 4 This is a schematic diagram of the network structure of an embodiment of the first extraction network. The first extraction network can be composed of a data augmentation part, convolutional layers, and fully connected layers. Data augmentation is used to expand the feature space of the samples. The methods mainly include two types: one is based on color transformation, brightness, hue, contrast, etc., to expand the samples; the other is based on geometric transformation, such as rotation, cropping, perspective changes, etc., to expand the samples. When training the first extraction network, different data augmentation operations can be performed on the samples. The specific operation can be selected according to the actual situation, and no specific limitation is made here. The structure of the convolutional layer and the fully connected layer is as follows: Figure 3 As shown, when training the first extraction network, two sample images can be input. It should be noted that in the two sample images, the first sample object in the first sample image and the second sample object in the second sample image belong to the same category, and the first sample image and the second sample image have the same or different cleaning attributes.

[0045] In a specific implementation scenario, during the testing phase of the first extraction network, sample images and test images to be tested can be input first. The cleaning attribute of the sample images can be "needs cleaning" or "no cleaning required," and the cleaning attribute of the sample images can be selected according to the actual situation, without specific limitations here. Based on this, the features of the sample images and the features of the test images are obtained separately, and the test distance between the features of the test images and the features of the sample images is obtained. When the cleaning attribute of the sample image is "no cleaning required" and the test distance is greater than a first threshold, the cleaning attribute of the test image is determined to be "needs cleaning"; otherwise, the cleaning attribute of the test image is determined to be "no cleaning required." When the cleaning attribute of the sample image is "needs cleaning" and the test distance is greater than a second threshold, the cleaning attribute of the test image is determined to be "no cleaning required"; otherwise, the cleaning attribute of the test image is determined to be "needs cleaning."

[0046] In one implementation scenario, the first detection result further includes whether a cleaning agent for cleaning the target object is detected in the first image. After obtaining the second detection result, it can also be determined that the target object has not been cleaned if the second detection result includes that the target object needs cleaning and the first detection result includes that no cleaning agent was detected; conversely, it can be determined that the target object has been cleaned if the second detection result includes that the target object needs cleaning and the first detection result includes that cleaning agent was detected. This method, by determining whether the first detection result includes the detection of cleaning agent, and thus determining whether the target object has been cleaned, further improves the accuracy of cleaning detection.

[0047] In one implementation scenario, the first detection result also includes whether a second area of ​​the cleaning personnel is detected in the first image. After determining that the target object has been cleaned, analysis can be performed based on whether a second area of ​​the cleaning personnel is detected in the first image to determine the cleaning method of the target object. This method, by confirming whether a second area of ​​the cleaning personnel is detected in the first image and then determining the cleaning method, can improve the accuracy of cleaning detection, thereby increasing the overall accuracy of cleaning detection.

[0048] In a specific implementation scenario, to determine the cleaning method for the target object, in response to the detection of a second region in the first image, and based on the fact that the distance between the first and second regions meets a preset condition, the cleaning method is determined to be manual cleaning. In this process, the preset condition can be set to a distance between the first and second regions that meets the cleaning distance requirement. It should be noted that the cleaning distance setting varies depending on the target object. For example, when the target object is a vehicle, the cleaning distance is set to no more than 0.8 meters, meaning the preset condition is that the distance between the first and second regions is within 0.8 meters, and the cleaning method is determined to be manual cleaning. When the target object is the hull of a boat and the boat is being cleaned ashore, since a high-pressure water gun is used to clean the hull, the cleaning distance is set to 1 to 2 meters, meaning the preset condition is that the distance between the first and second regions is within 1 to 2 meters, and the cleaning method is determined to be manual cleaning. The preset condition can be determined according to the actual situation and is not specifically limited here. Alternatively, in response to the absence of a second region in the first image, the cleaning method can be determined to be automatic cleaning. The above method, by determining whether the first image includes a second region, and thus determining the cleaning method, helps to improve the accuracy of cleaning detection.

[0049] In one implementation scenario, after obtaining the cleaning method, to further determine whether the target object is clean, in response to the second detection result indicating that the target object needs cleaning, a second image of the target object after cleaning is acquired, and a third region of the target object in the second image is detected. The method for acquiring the third region can refer to the method for acquiring the first region in the aforementioned disclosed embodiments, and will not be repeated here. After acquiring the third region, a second target image of the target object can be extracted from the second image based on the third region. The method for acquiring the second target image can refer to the method for acquiring the first target image in the aforementioned disclosed embodiments, and will not be repeated here. Based on this, a comparison detection is performed based on the reference image and the second target image to obtain a third detection result; and the third detection result includes whether the target object is clean. In this process, the method for acquiring the third detection result can refer to the method for acquiring the second detection result in the aforementioned disclosed embodiments, and will not be repeated here. The above method, by re-detecting the cleaned target object, can further determine whether the target object is clean, thereby improving the applicability of cleaning detection.

[0050] Please see Figure 5 , Figure 5This is a schematic diagram of a process of an embodiment of the cleaning and detection method of this application. First, a first image is captured on the target object, and the first image is input into an image detection model for detection to obtain a first detection result. It should be noted that the first detection result may include the first area of ​​the target object, the cleaning agent used to clean the target object, or the second area of ​​the cleaning personnel. Of course, it may also not include the first area of ​​the target object. Therefore, after obtaining the first detection result, it is necessary to determine whether the first detection result includes the first area of ​​the target object. If it does not include it, the cleaning and detection ends, and a new first image can be re-inputted for detection, or the detection process ends. The specific determination can be made according to the actual situation, and no specific limitation is made here. If the first detection result includes the first region of the target object, the first target image of the target object is extracted from the first image. Then, the first target image and the reference image are input into the first extraction network to obtain the second detection result. The second detection result includes whether the target object needs cleaning. After determining whether the target object needs cleaning, in order to further determine the cleaning method of the target object, it can be confirmed whether the target object needs cleaning. If not, the cleaning detection ends, and a new first image can be re-inputted for detection, or the detection process ends. The specific determination can be made according to the actual situation, and no specific limitation is made here. If it is needed, it is then determined whether the first detection result includes the cleaning agent for cleaning the target object. If not, the target object is not cleaned. If it includes, it is then determined whether the first detection result detects the second region of the cleaning personnel. If not, the cleaning method of the target object is determined to be automatic cleaning. Otherwise, the regional distance between the first region and the second region is obtained, and it is determined whether the regional distance meets the preset conditions. If the preset conditions are met, the cleaning method is determined to be manual cleaning. Otherwise, the cleaning method is determined to be automatic cleaning. After determining the cleaning method, to further determine whether the cleaning is thorough, a second image of the target object after cleaning can be acquired and processed. This image is then passed through a first extraction network to obtain detection results. These results are then confirmed. If the target object is clean, the cleaning detection ends, and a new first image can be input for detection, or the detection process can end. The specific method can be determined based on the actual situation and is not specifically limited here. If the target object is not clean, an alert can be issued, and the target object can be cleaned again. During this process, the cleaning detection can be performed simultaneously with the target object cleaning detection, thereby improving the efficiency of the cleaning detection.

[0051] The above scheme detects a first image of the target object to obtain a first detection result, which includes at least a first region of the target object. Based on this first region, a first target image of the target object is extracted from the first image. Then, a comparison detection is performed between a reference image and the first target image to obtain a second detection result. The reference image contains reference object samples corresponding to the target object, including samples that do not require cleaning and samples that require cleaning. The second detection result indicates whether the target object needs cleaning. On one hand, by detecting the image of the target object and determining its first region, and then extracting the first target image based on this region, the accuracy of extracting the first target region is improved. On the other hand, after determining the first target region, the comparison detection between the reference image and the first target image, due to the autonomous selectivity of the reference image, improves the reliability of the detection result and further enhances the applicability of cleaning detection. Therefore, it can improve the accuracy of cleaning detection.

[0052] Please see Figure 6 , Figure 6 This is a schematic diagram of the framework of an embodiment of the cleaning and inspection apparatus of this application. The cleaning and inspection apparatus 60 includes a detection module 61, an extraction module 62, and a comparison module 63. The detection module 61 is used to perform detection based on a first image captured of a target object to obtain a first detection result; and the first detection result includes at least a first region of the target object. The extraction module 62 is used to extract a first target image of the target object from the first image based on the first region. The comparison module 63 is used to perform comparative detection based on a reference image and the first target image to obtain a second detection result; wherein the reference image includes a reference object sample corresponding to the target object, the reference object sample includes samples that do not require cleaning or samples that require cleaning, and the second detection result includes whether the target object needs cleaning.

[0053] The above scheme, on the one hand, detects the image of the target object to determine its first region, and extracts the first target image based on this first region, which helps improve the accuracy of extracting the first target region. On the other hand, after determining the first target region, it compares and detects the first target image with a reference image. Since the reference image has autonomous selectivity, it can improve the reliability of the detection results and further enhance the applicability of the cleaning detection. Therefore, it can improve the accuracy of cleaning detection.

[0054] In some disclosed embodiments, the comparison module 63 includes a first extraction submodule, which is used to extract a first image feature of a reference image, extract a second image feature of a first target image, and obtain a first feature distance between the first image feature and the second image feature; the comparison module 63 also includes a first determination submodule, which is used to obtain a second detection result based on the first feature distance and whether the reference object sample in the reference image needs to be cleaned.

[0055] Therefore, by extracting features from the reference image and the first target image respectively, and calculating the distance between the image features, the second detection result can be determined, which helps to improve the accuracy of the second detection result, that is, to improve the accuracy of cleaning detection.

[0056] In some disclosed embodiments, the determining submodule includes a first determining unit, configured to determine a second detection result including that the target object needs cleaning in response to a reference object sample in the reference image not needing cleaning and a first feature distance greater than a first threshold; the determining submodule includes a second determining unit, configured to determine a second detection result including that the target object does not need cleaning in response to a reference object sample in the reference image not needing cleaning and a first feature distance not greater than the first threshold; the determining submodule includes a third determining unit, configured to determine a second detection result including that the target object does not need cleaning in response to a reference object sample in the reference image needing cleaning and a first feature distance greater than a second threshold; the determining submodule further includes a fourth determining unit, configured to determine a second detection result including that the target object needs cleaning in response to a reference object sample in the reference image needing cleaning and a first feature distance not greater than the second threshold.

[0057] In some disclosed embodiments, the comparison module 63 includes an image acquisition submodule for acquiring sample image objects; wherein, the sample image pair includes a first sample image and a second sample image, the first sample object in the first sample image and the second sample object in the second sample image belong to the same category, and the first sample object and the second sample image have the same or different cleaning attributes, the cleaning attributes representing whether cleaning is required; the comparison module 63 includes a second extraction submodule for extracting first sample image features of the first sample image and second sample image features of the second sample image based on the first extraction network; the comparison module 63 includes a second determination submodule for obtaining a network loss based on the sample distance between the first sample image features and the second sample image features; wherein, when the first sample object and the second sample image have the same cleaning attributes, the network loss is positively correlated with the sample distance, and when the first sample object and the second sample image have different cleaning attributes, the network loss is negatively correlated with the sample distance; the comparison module 63 also includes a parameter adjustment submodule for adjusting the network parameters of the first extraction network based on the network loss.

[0058] Therefore, by training the first extraction network, the accuracy of the first extraction network can be improved during use, thereby further improving the accuracy of cleaning detection.

[0059] In some disclosed embodiments, the cleaning detection device 60 includes a determining module, which determines that the target object has been cleaned in response to a second detection result including that the target object needs to be cleaned and a first detection result including that cleaning agent is detected.

[0060] Therefore, by determining whether the first detection result includes the detection of cleaning agent, and thus determining whether the target object has been cleaned, the accuracy of cleaning detection can be further improved.

[0061] In some disclosed embodiments, the first detection result further includes: whether a second area of ​​the cleaning personnel is detected in the first image; the determination module includes an analysis submodule, which is used to analyze whether a second area of ​​the cleaning personnel is detected in the first image to obtain the cleaning method of the target object.

[0062] Therefore, by confirming whether a second area of ​​cleaning personnel is detected in the first image, and then determining the cleaning method, the accuracy of cleaning detection can be improved, thereby increasing the accuracy rate of cleaning detection.

[0063] In some disclosed embodiments, the analysis submodule includes a first determining unit, which is configured to determine the cleaning method as manual cleaning based on the fact that the area between the first area and the second area meets a preset condition when a second area is detected in the first image; the analysis submodule also includes a second determining unit, which is configured to determine the cleaning method as automatic cleaning when no second area is detected in the first image.

[0064] Therefore, determining whether the second region is included in the first image, and thus determining the cleaning method, helps to improve the accuracy of cleaning detection.

[0065] In some disclosed embodiments, the first detection result further includes a classification detection result, which includes whether a cleaning agent for cleaning the target object is detected in the first image, and the classification detection result is used to analyze whether the target object has been cleaned; the detection module 61 includes an extraction submodule, which is used to extract global image features of the first image; the detection module 62 includes a detection submodule, which is used to perform target detection based on global image features to obtain a target detection result, and to perform classification detection based on global image features to obtain a classification detection result; and the target detection result includes at least: a first region of the target object; the detection module 62 further includes a determination submodule, which is used to obtain the first detection result based on the target detection result and the classification detection result.

[0066] Therefore, by detecting the first image, classification detection results are obtained. In this process, a multi-task approach is adopted for detection, which reduces the complexity and running time of the algorithm, thereby improving the efficiency of cleaning and detection.

[0067] In some disclosed embodiments, the first detection result is obtained by an image detection model, which includes a second extraction network, a target detection network and a classification detection network, both connected to the second extraction network; and the second extraction network is used to extract global image features, the target detection network is used to perform target detection, and the classification detection network is used to perform classification detection.

[0068] Therefore, detecting the first image using an image detection model and determining the first detection result helps improve the accuracy of the first detection result and further improves the cleaning detection accuracy.

[0069] In some disclosed embodiments, the detection device 60 includes an acquisition module, which is configured to acquire a second image of the target object after cleaning in response to a second detection result that the target object needs to be cleaned, detect a third region of the target object in the second image, and extract a second target image of the target object in the second image based on the third region; the detection device 60 also includes a comparison module, which is configured to perform a comparison detection based on a reference image and a second target image to obtain a third detection result; and the third detection result includes whether the target object is clean.

[0070] Therefore, by re-inspecting the cleaned target object, it can be further determined whether the target object is clean, thereby improving the applicability of cleaning inspection.

[0071] Please see Figure 7 , Figure 7 This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 70 includes a memory 71 and a processor 72 coupled to each other. The memory 71 stores program instructions, and the processor 72 is used to execute the program instructions to implement the steps in any of the above-described cleaning and detection method embodiments. Specifically, the electronic device 70 may include, but is not limited to, desktop computers, laptops, servers, mobile phones, tablets, etc., and is not limited thereto.

[0072] Specifically, processor 72 controls itself and memory 71 to implement the steps in any of the above-described cleaning and detection method embodiments. Processor 72 can also be referred to as a CPU (Central Processing Unit). Processor 72 may be an integrated circuit chip with signal processing capabilities. Processor 72 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 72 can be implemented using integrated circuit chips.

[0073] The above scheme, on the one hand, detects the image of the target object to determine its first region, and extracts the first target image based on this first region, which helps improve the accuracy of extracting the first target region. On the other hand, after determining the first target region, it compares and detects the first target image with a reference image. Since the reference image has autonomous selectivity, it can improve the reliability of the detection results and further enhance the applicability of the cleaning detection. Therefore, it can improve the accuracy of cleaning detection.

[0074] Please see Figure 8 , Figure 8 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 80 stores program instructions 81 that can be executed by a processor. The program instructions 81 are used to implement the steps in any of the above-described cleaning and detection method embodiments.

[0075] The above scheme, on the one hand, detects the image of the target object to determine its first region, and extracts the first target image based on this first region, which helps improve the accuracy of extracting the first target region. On the other hand, after determining the first target region, it compares and detects the first target image with a reference image. Since the reference image has autonomous selectivity, it can improve the reliability of the detection results and further enhance the applicability of the cleaning detection. Therefore, it can improve the accuracy of cleaning detection.

[0076] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0077] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0078] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0079] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0080] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0081] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A cleaning detection method, characterized by, The method comprises: detecting a first image taken by a camera to obtain a first detection result, wherein the first detection result at least includes a first region of the target object and whether a cleaning agent for cleaning the target object is detected in the first image, and the first detection result further includes a classification detection result, wherein the classification detection result includes whether a cleaning agent for cleaning the target object is detected in the first image, and the classification detection result is used to analyze whether the target object is cleaned; extracting a first target image of the target object in the first image based on the first region; performing comparative detection based on a reference image and the first target image to obtain a second detection result, wherein the reference image contains a reference object sample corresponding to the target object, the reference object sample includes a sample that does not need to be cleaned or a sample that needs to be cleaned, and the second detection result includes whether the target object needs to be cleaned; in response to the second detection result representing that the target object needs to be cleaned and the first detection result including that the cleaning agent is not detected, determining that the target object is not cleaned; in response to the second detection result representing that the target object does not need to be cleaned and the first detection result including that the cleaning agent is detected, determining that the target object is cleaned; wherein the detection of the first image taken by the camera to obtain the first detection result comprises: extracting a global image feature of the first image; performing target detection based on the global image feature to obtain a target detection result, and performing classification detection based on the global image feature to obtain the classification detection result; wherein the target detection result at least includes the first region of the target object; and the first detection result is obtained based on the target detection result and the classification detection result.

2. The method of claim 1, wherein, The comparative detection based on the reference image and the first target image to obtain the second detection result comprises: extracting a first image feature of the reference image and a second image feature of the first target image, and obtaining a first feature distance between the first image feature and the second image feature; obtaining the second detection result based on the first feature distance and whether the reference object sample in the reference image needs to be cleaned.

3. The method of claim 2, wherein, The obtaining of the second detection result based on the first feature distance and whether the reference object sample in the reference image needs to be cleaned comprises at least one of: in response to the reference object sample in the reference image not needing to be cleaned and the first feature distance being greater than a first threshold value, determining that the second detection result includes that the target object needs to be cleaned; in response to the reference object sample in the reference image not needing to be cleaned and the first feature distance not being greater than the first threshold value, determining that the second detection result includes that the target object does not need to be cleaned; in response to the reference object sample in the reference image needing to be cleaned and the first feature distance being greater than a second threshold value, determining that the second detection result includes that the target object does not need to be cleaned; In response to the reference object sample in the reference image needing cleaning and the first feature distance being not greater than the second threshold, determining that the second detection result includes the target object needing cleaning.

4. The method of claim 2, wherein, The first image feature and the second image feature are extracted by a first extraction network, and before the first image feature and the second image feature are extracted based on the first extraction network, the method further includes: obtaining a sample image object; wherein the sample image pair includes a first sample image and a second sample image, a first sample object in the first sample image and a second sample object in the second sample image belong to the same category, and the first sample image and the second sample image have the same or different cleaning properties, the cleaning property representing whether cleaning is needed; extracting a first sample image feature of the first sample image and a second sample image feature of the second sample image based on the first extraction network respectively; obtaining a network loss based on a sample distance between the first sample image feature and the second sample image feature; wherein in the case that the first sample object and the second sample image have the same cleaning property, the network loss is positively correlated with the sample distance, and in the case that the first sample object and the second sample image have different cleaning properties, the network loss is negatively correlated with the sample distance; adjusting network parameters of the first extraction network based on the network loss.

5. The method of claim 1, wherein, The first detection result further includes a second area whether cleaning personnel are detected in the first image; the method further includes, after determining that the target object is cleaned: analyzing based on the second area whether cleaning personnel are detected in the first image to obtain a cleaning mode of the target object.

6. The method of claim 5, wherein, The analysis based on the second area whether cleaning personnel are detected in the first image to obtain the cleaning mode of the target object includes at least one of: in response to detecting the second area in the first image, determining that the cleaning mode is manual cleaning based on the area distance between the first area and the second area meeting a preset condition; in response to not detecting the second area in the first image, determining that the cleaning mode is automatic cleaning.

7. The method of claim 1, wherein, The first detection result is detected by an image detection model, and the image detection model includes a second extraction network, and a target detection network and a classification detection network both connected with the second extraction network; wherein the second extraction network is used to extract the global image feature, the target detection network is used to perform the target detection, and the classification detection network is used to perform the classification detection.

8. The method of claim 1, wherein, The method further includes: in response to the second detection result including the target object needing cleaning, obtaining a second image photographed after the target object is cleaned, detecting a third area of the target object in the second image, and extracting a second target image of the target object in the second image based on the third area; performing comparison detection based on the reference image and the second target image to obtain a third detection result; wherein the third detection result comprises whether the target object is cleaned.

9. A cleaning detection device, characterized by The method comprises the following steps: detecting a first image captured by a camera to obtain a first detection result; wherein the first detection result comprises at least a first region of the target object and whether a cleaning agent for cleaning the target object is detected in the first image, and the first detection result further comprises a classification detection result, wherein the classification detection result comprises whether a cleaning agent for cleaning the target object is detected in the first image, and the classification detection result is used to analyze whether the target object is cleaned; extracting a first target image of the target object in the first image based on the first region; performing comparison detection based on a reference image and the first target image to obtain a second detection result; wherein the reference image comprises a reference object sample corresponding to the target object, the reference object sample comprises a sample that does not need to be cleaned or a sample that needs to be cleaned, and the second detection result comprises whether the target object needs to be cleaned; determining that the target object is not cleaned in response to the second detection result representing that the target object needs to be cleaned and the first detection result comprising that the cleaning agent is not detected; and determining that the target object is cleaned in response to the second detection result representing that the target object does not need to be cleaned and the first detection result comprising that the cleaning agent is detected. The detection module is configured to detect a first image captured by a camera to obtain a first detection result, comprising: extracting a global image feature of the first image; performing target detection based on the global image feature to obtain a target detection result, and performing classification detection based on the global image feature to obtain the classification detection result; wherein the target detection result comprises at least a first region of the target object; and obtaining the first detection result based on the target detection result and the classification detection result.

10. An electronic device, comprising: The memory and the processor are coupled to each other, the memory stores program instructions, and the processor is configured to execute the program instructions to implement the cleaning detection method of any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, The memory stores program instructions executable by the processor, and the program instructions are used to implement the cleaning detection method of any one of claims 1 to 8.

Images