A model evaluation method, apparatus, device, program product, and medium

Abstract

The embodiment of the disclosure provides a model evaluation method, device, equipment, program product and medium, relating to the field of deep learning, and the method comprises: acquiring a test image, the test image comprising a foreground region and a background region; inputting the test image into a target classification model to obtain a target classification result, the target classification result comprising at least target feature information and target attribute information; inputting the test image into an attribution analysis model to obtain an attribution graph corresponding to the test image; analyzing a connected domain in the attribution graph based on the test image to determine a type of the connected domain, the type of the connected domain being a foreground type or a background type; determining whether the target classification model is a qualified target classification model according to the type of the connected domain, and determining whether the target classification result is a credible result according to the type of the connected domain.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning, and more specifically, to a model evaluation method, apparatus, device, program product, and medium. Background Technology

[0002] In industrial defect detection, marking the exact location of defects requires significant manpower and time. Therefore, in many cases, only a partial image containing the defect (e.g., 256×256 resolution) is cropped from a large product image (e.g., 1024×1024 resolution). Then, a deep learning algorithm is used to classify the cropped partial image to determine the defect category. Different processing methods are applied based on the defect category (e.g., repair, ignore, or destroy). The defect category is determined by two factors: the process stage from which the defect originates and the shape and location of the defect itself.

[0003] Deep learning-based defect classification algorithms have achieved great success in industrial manufacturing. However, when specific locations in defect scenarios like those mentioned above are not labeled, the lack of interpretability in deep learning makes it difficult to determine whether the target classification model makes decisions based on foreground or background. The foreground determines the shape and location of the defect, while the background determines which process stage the defect originates from. If the target classification model outputs a defect category solely based on either the foreground or background, the model's reliability and generality will be relatively low. Therefore, improving the reliability and generality of target classification models has become a pressing issue in this field. Summary of the Invention

[0004] This disclosure provides a model evaluation method, apparatus, device, program product, and medium, aiming to solve the problem of how to improve the reliability and versatility of target classification models.

[0005] A first aspect of this disclosure provides a model evaluation method, the method comprising: Acquire a test image, which includes a foreground region and a background region; The test image is input into the target classification model to obtain the target classification result, which includes at least target feature information and target attribute information. Input the test image into the attribution analysis model to obtain the attribution map corresponding to the test image; Based on the test image, the connected components in the attribution graph are analyzed to determine the type of the connected components, which is either foreground type or background type. The type of the connected component determines whether the target classification model is a qualified target classification model, and the type of the connected component determines whether the target classification result is a reliable result.

[0006] In some optional implementations, the step of inputting the test image into the attribution analysis model to obtain the attribution map corresponding to the test image includes: The test image is input into the attribution analysis model to obtain an initial attribution map. The gray value of each pixel in the initial attribution map represents the degree of correlation between each pixel and the target feature information. The grayscale value of each pixel in the initial attribution map is classified based on a first threshold. Pixels with grayscale values ​​greater than or equal to the first threshold are designated as first pixels, and pixels with grayscale values ​​less than the first threshold are designated as second pixels. The grayscale value of the first pixel is adjusted to a first grayscale value, and the grayscale value of the second pixel is adjusted to a second grayscale value to obtain the attribution map.

[0007] In some optional implementations, the step of inputting the test image into the attribution analysis model to obtain the attribution map corresponding to the test image includes: The test image is input into the attribution analysis model to obtain an initial attribution map. The gray value of each pixel in the initial attribution map represents the degree of correlation between each pixel and the target feature information. Sort the grayscale values ​​of each pixel in the initial attribution map; The pixels are classified based on the second threshold to obtain third pixels and fourth pixels, wherein the gray value of the third pixel is greater than the gray value of the fourth pixel, and the proportion of the number of third pixels is equal to the second threshold. The grayscale value of the third pixel is adjusted to the first grayscale value, and the grayscale value of the fourth pixel is adjusted to the second grayscale value to obtain the attribution map.

[0008] In some optional implementations, after obtaining the attribution map corresponding to the test image, the method further includes: Obtain an initial pixel and create a connected component corresponding to the initial pixel, wherein the initial pixel is any pixel in the attribution graph; In the attribution graph, all neighboring pixels of the initial pixel are detected one by one, and neighboring pixels with the same gray value as the initial pixel are added to the connected component corresponding to the initial pixel. Repeat the above steps until the initial pixel traverses all pixels of the attribution graph, obtaining at least one connected component corresponding to the attribution graph.

[0009] In some optional implementations, after obtaining at least one connected component corresponding to the attribution graph, the method further includes: Get the number of pixels contained in each connected component; Based on the third threshold, the connected components are filtered, and connected components with a number of pixels less than or equal to the third threshold are deleted.

[0010] In some optional implementations, the step of analyzing the connected components in the attribution graph based on the test image to determine the type of the connected components includes: The test image is overlaid with the attribution map to obtain an overlay image; Based on the overlay map, the overlap between each connected component and the foreground region of the test image is obtained; Based on the overlap, the type of each connected component is determined.

[0011] In some optional implementations, the step of overlaying the test image with the attribution map to obtain an overlay map includes: Identify the pixels at the same location in the attribution map and the test image; The pixel values ​​of the pixels in the test image are added to the gray values ​​of the pixels at the same position in the attribution map to obtain the overlay image; Wherein, the brightness of the pixels corresponding to the connected components in the overlay image and the test image are the same; the brightness of the pixels corresponding to the non-connected components in the overlay image is a preset ratio of the brightness of the pixels corresponding to the non-connected components in the test image.

[0012] In some optional implementations, obtaining the overlap between each connected component and the foreground region of the test image based on the overlay map includes: Based on the overlay map, the pixels of the foreground region of the test image and the pixels of each connected component in the attribution map are obtained. Based on the pixels of the foreground region and the pixels of each connected component, determine the number of intersection pixels and the number of union pixels corresponding to each connected component. The intersection pixels are the pixels common to the foreground region and each connected component, and the union pixels are all the pixels contained in the foreground region and each connected component. The ratio of the number of intersection pixels to the number of union pixels is obtained as the overlap degree between each connected component and the foreground region of the test image.

[0013] In some alternative implementations, determining the type of each connected component based on the overlap includes: Based on the fourth threshold, the overlap corresponding to each of the connected components is filtered; If the overlap is greater than or equal to the fourth threshold, the type of the connected component is determined as the foreground type; If the overlap is less than the fourth threshold, the type of the connected component is determined to be the background type.

[0014] In some optional implementations, the test image comprises a plurality of periodically arranged repeating units, and the analysis of connected components in the attribution graph based on the test image to determine the type of the connected components includes: Based on the test image, the connected components in the attribution graph are analyzed to determine the similarity between each connected component and the corresponding repeating unit in the test image. Based on the similarity, the type of each connected component is determined.

[0015] In some optional implementations, the step of analyzing connected components in the attribution graph based on the test image to determine the similarity between each connected component and its corresponding repeating unit in the test image includes: Based on each of the connected components, the image region in the test image corresponding to each of the connected components is determined as the target image region; In the test image, the target repeating unit corresponding to the target image region is determined; The image region corresponding to the repeating unit of the same type as the target repeating unit is determined as the comparison image region; Determine the similarity between the target image region and the comparison image region.

[0016] In some alternative implementations, determining the type of each connected component based on the similarity includes: Based on the fifth threshold, the similarity corresponding to each of the connected components is filtered; If the similarity is greater than or equal to the fifth threshold, the type of the connected component is determined as the background type; If the similarity is less than the fifth threshold, the type of the connected component is determined as the foreground type.

[0017] In some optional implementations, the step of inputting the test image into the attribution analysis model to obtain an initial attribution map includes: The test image is input into the attribution analysis model to obtain the gradient calculation weights of each pixel in the target channel and target prediction category in the test image. The feature layer of the target channel is linearly weighted and fused with the gradient calculation weights of each pixel under the target channel and the target prediction category, and then normalized by a rectifier function to obtain the initial attribution map corresponding to the test image. The initial attribution map is a feature heatmap corresponding to the target prediction category, and the target prediction category corresponds to the target feature information.

[0018] In some optional implementations, determining whether the target classification model is a qualified target classification model based on the type of the connected components, and determining whether the target classification result is a reliable result based on the type of the connected components, includes: If the type of the connected component satisfies the first condition, the defect classification model is determined to be a qualified defect classification model, and the defect classification result is determined to be a reliable result. The first condition is that the connected component contains at least one connected component of the foreground type and the connected component contains at least one connected component of the background type. If the type of the connected component does not meet the first condition, the target classification model is determined to be an unqualified target classification model, and the target classification result is determined to be an unreliable result.

[0019] A second aspect of this disclosure provides a model evaluation apparatus, the apparatus comprising: An image acquisition module is used to acquire a test image, the test image including a foreground region and a background region; The defect classification module is used to input the test image into the target classification model to obtain the target classification result, wherein the target classification result includes at least target feature information and target attribute information. The attribution graph module is used to input the test image into the attribution analysis model and obtain the attribution graph corresponding to the test image; The connected component analysis module is used to analyze the connected components in the attribution graph based on the test image, and determine the type of the connected component, which is either foreground type or background type. The judgment module is used to determine whether the target classification model is a qualified target classification model based on the type of the connected component, and to determine whether the target classification result is a reliable result based on the type of the connected component.

[0020] In some alternative implementations, the attribution graph module includes: The first initial attribution map submodule is used to input the test image into the attribution analysis model to obtain an initial attribution map, wherein the gray value of each pixel in the initial attribution map represents the degree of correlation between each pixel and the target feature information. The first threshold submodule is used to classify the gray value of each pixel in the initial attribution map based on the first threshold, and to take the pixels with gray values ​​greater than or equal to the first threshold as the first pixel and the pixels with gray values ​​less than the first threshold as the second pixel. The first attribution map submodule is used to adjust the gray value of the first pixel to a first gray value and the gray value of the second pixel to a second gray value to obtain the attribution map.

[0021] In some alternative implementations, the attribution graph module includes: The second initial attribution map submodule is used to input the test image into the attribution analysis model to obtain an initial attribution map, wherein the gray value of each pixel in the initial attribution map represents the degree of correlation between each pixel and the target feature information. The pixel sorting submodule is used to sort the grayscale values ​​of each pixel in the initial attribution map; The second threshold submodule is used to classify the pixels based on the second threshold to obtain the third pixel and the fourth pixel, wherein the gray value of the third pixel is greater than the gray value of the fourth pixel, and the proportion of the number of the third pixel is equal to the second threshold. The second attribution map submodule is used to adjust the gray value of the third pixel to a first gray value and the gray value of the fourth pixel to a second gray value to obtain the attribution map.

[0022] In some optional implementations, the attribution graph module further includes: A connected component creation submodule is used to obtain an initial pixel and create a connected component corresponding to the initial pixel, wherein the initial pixel is any pixel in the attribution graph; The neighborhood detection submodule is used to detect all neighboring pixels of the initial pixel in the attribution graph one by one, and add neighboring pixels with the same gray value as the initial pixel to the connected component corresponding to the initial pixel. The connected component submodule is used to repeat the above steps until the initial pixel point traverses all pixels of the attribution graph to obtain at least one connected component corresponding to the attribution graph.

[0023] In some optional implementations, the attribution graph module further includes: The connected component pixel submodule is used to obtain the number of pixels contained in each connected component; The third threshold submodule is used to filter the connected components based on the third threshold and delete connected components in which the number of pixels is less than or equal to the third threshold.

[0024] In some optional implementations, the connected component analysis module includes: The overlay submodule is used to overlay the test image with the attribution map to obtain an overlay map; The overlap submodule is used to obtain the overlap between each connected component and the foreground region of the test image based on the overlay graph; The first connected component analysis submodule is used to determine the type of each connected component based on the overlap.

[0025] In some optional implementations, the overlay submodule includes: Overlay pixel units are used to determine pixels at the same positions in the attribution map and the test image; The overlay unit is used to add the pixel value of the pixel in the test image to the gray value of the pixel at the same position in the attribution map to obtain the overlay map; Wherein, the brightness of the pixels corresponding to the connected components in the overlay image and the test image are the same; the brightness of the pixels corresponding to the non-connected components in the overlay image is a preset ratio of the brightness of the pixels corresponding to the non-connected components in the test image.

[0026] In some optional implementations, the overlap submodule includes: The pixel determination unit is used to obtain the pixels in the foreground region of the test image and the pixels in each connected component of the attribution map based on the overlay map. The intersection and union pixel unit is used to determine the number of intersection pixels and the number of union pixels corresponding to each connected component based on the pixels of the foreground region and the pixels of each connected component. The intersection pixels are the pixels common to the foreground region and each connected component, and the union pixels are all the pixels contained in the foreground region and each connected component. An overlap unit is used to obtain the ratio of the number of intersection pixels to the number of union pixels, as the overlap between each connected component and the foreground region of the test image.

[0027] In some optional implementations, the first connected component analysis submodule includes: The fourth threshold unit is used to filter the overlap of each connected component based on the fourth threshold. The first foreground type unit is used to determine the type of the connected component as the foreground type when the overlap is greater than or equal to the fourth threshold. A first background type unit is used to determine the type of the connected component as the background type when the overlap is less than the fourth threshold.

[0028] In some optional implementations, the connected component analysis module further includes: The similarity submodule is used to analyze the connected components in the attribution graph based on the test image, and determine the similarity between each connected component and the corresponding repeating unit of the connected component in the test image. The second connected component analysis submodule is used to determine the type of each connected component based on the similarity.

[0029] In some optional implementations, the similarity submodule includes: The target image region unit is used to determine, based on each of the connected components, the image region in the test image corresponding to each of the connected components, as the target image region; A repetition determination unit is used to determine the target repetition unit corresponding to the target image region in the test image; A comparison image region unit is used to determine the image region corresponding to a repeating unit of the same type as the target repeating unit, and use it as the comparison image region. A similarity unit is used to determine the similarity between the target image region and the comparison image region.

[0030] In some optional implementations, the second connected component analysis submodule includes: The fifth threshold unit is used to filter the similarity corresponding to each connected component based on the fifth threshold. The second background type unit is used to determine the type of the connected component as the background type when the similarity is greater than or equal to the fifth threshold. The second foreground type unit is used to determine the type of the connected component as the foreground type when the similarity is less than the fifth threshold.

[0031] In some optional implementations, the first initial attribution graph submodule includes: The gradient calculation weight unit is used to input the test image into the attribution analysis model and obtain the gradient calculation weight of each pixel in the target channel and target prediction category in the test image. The initial attribution map unit is used to linearly weight and fuse the feature layer of the target channel with the gradient calculation weights of each pixel under the target channel and the target prediction category, and then normalize it through a rectification function to obtain the initial attribution map corresponding to the test image. The initial attribution map is a feature heatmap corresponding to the target prediction category, and the target prediction category corresponds to the target feature information.

[0032] In some optional implementations, the determination module includes: The first judgment submodule is used to determine the defect classification model as a qualified defect classification model and the defect classification result as a reliable result when the type of the connected component satisfies the first condition. The first condition is that the connected component contains at least one connected component of the foreground type and the connected component contains at least one connected component of the background type. The second judgment submodule is used to determine the target classification model as an unqualified target classification model and the target classification result as an unreliable result when the type of the connected component does not meet the first condition.

[0033] A third aspect of this disclosure provides a computing processing device, comprising: Memory containing computer-readable code; and One or more processors, when the computer-readable code is executed by the one or more processors, the computing processing device performs the model evaluation method as described in any one of the first aspects.

[0034] A fourth aspect of this disclosure provides a computer program product including computer-readable code that, when run on a computing processing device, causes the computing processing device to perform a model evaluation method according to any one of the first aspects.

[0035] A fifth aspect of this disclosure provides a computer-readable medium storing a computer program product as described in the fourth aspect.

[0036] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, specific embodiments of this disclosure are described below. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart of a model evaluation method for performing the method according to this disclosure is schematically shown; Figure 2 A schematic diagram of a test image for performing the method according to this disclosure is shown; Figure 3 A schematic diagram of an initial attribution graph for performing the method according to this disclosure is shown; Figure 4 A schematic diagram of an attribution graph for performing the method according to this disclosure is shown. Figure 5 A schematic diagram of a model evaluation apparatus for performing the method according to this disclosure is shown. Figure 6 A schematic diagram of the structure of a computing processing apparatus for performing the method according to the present disclosure is shown. Detailed Implementation

[0039] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0040] In industrial defect detection, marking the exact location of defects requires significant manpower and time. Therefore, in many cases, only a partial image containing the defect (e.g., 256×256 resolution) is cropped from a large product image (e.g., 1024×1024 resolution). Then, a deep learning algorithm is used to classify the cropped partial image to determine the defect category. Different processing methods are applied based on the defect category (e.g., repair, ignore, or destroy). The defect category is determined by two factors: the process stage from which the defect originates and the shape and location of the defect itself.

[0041] Deep learning-based defect classification algorithms have achieved great success in the industrial manufacturing field. However, when the specific location of a defect is not labeled, due to the uninterpretability of deep learning, it is difficult to determine whether the target classification model makes a decision based on the foreground or the background. The foreground determines the shape and location of the defect itself, while the background determines which process segment the defect comes from. If the defect category output by the target classification model is obtained solely from the foreground or solely from the background, the reliability and generality of the model will be relatively low.

[0042] In view of this, embodiments of this disclosure propose a model evaluation method. Figure 1 A flowchart illustrating a model evaluation method for performing the method according to this disclosure is shown schematically, such as... Figure 1 As shown, the model evaluation method includes the following steps: S101. Obtain the test image.

[0043] A test image is acquired, which includes a foreground region and a background region.

[0044] S102. Input the test image into the target classification model to obtain the target classification result.

[0045] The test image is input into the target classification model to obtain the target classification result, which includes at least target feature information and target attribute information.

[0046] S103. Input the test image into the attribution analysis model to obtain the attribution map corresponding to the test image.

[0047] S104. Using the test image as a reference, analyze the connected components in the attribution graph to determine the type of the connected components.

[0048] Based on the test image, the connected components in the attribution graph are analyzed to determine the type of the connected components, which is either foreground or background.

[0049] S105. Determine whether the target classification model is a qualified target classification model based on the type of the connected component, and determine whether the target classification result is a reliable result based on the type of the connected component.

[0050] In this embodiment, during the assembly process, the product flows sequentially along the production line via a conveyor device, performing corresponding installation processes at different workstation locations. These workstations can be used for manual assembly of different product components or for automated mechanical assembly. Therefore, as the product flows along the production line, different defects may occur at different workstations due to the different order of flow. In other words, the defects generated are related to target attribute information. In this embodiment, corresponding product images are collected at workstations at different locations on the production line for defect classification, thereby obtaining target classification results. These results characterize the target feature information and target attribute information contained in the product images acquired at each workstation location. For ease of understanding, this embodiment uses a defect detection scenario as an example to illustrate the model evaluation method provided. In this defect detection scenario, the target feature information characterizes the shape and location of defects generated at corresponding workstation locations; the target attribute information characterizes the origin of the defect from a specific workstation or production line process segment. It is easy to understand that the model evaluation method provided in the embodiments of this disclosure is not limited to the defect detection scenario, and any other scenario can be considered, as long as the same technical idea is applied as in the embodiments of this disclosure.

[0051] In this embodiment of the disclosure, the products are arranged at intervals on the production line and transmitted between different workstations. Therefore, the product images collected at different workstations can be the entire product image or images of a partial area of ​​the product. It should be noted that when collecting images from different workstations on the production line, the specific image collection frequency is determined according to the actual situation. It can be a preset fixed collection frequency or a collection frequency determined according to the speed of the production line flow. This embodiment of the disclosure does not limit this.

[0052] The test image is an image of the installation process area corresponding to different workstations in the product image. The relative size relationship between the test image and the product image depends on the size of the area when the component is installed at the corresponding workstation. When the component installed at the workstation occupies a part of the product image, the test image is a part of the product image; when the component installed at the workstation occupies all or most of the area of ​​the product image, the test image is the entirety of the product image. The specific area ratio of the test image relative to the product image can be determined in advance according to the testing requirements and the workstation, and this disclosure does not impose any restrictions on it.

[0053] In this embodiment of the disclosure, the test image includes at least a foreground region and a background region. The foreground region is the defect region in the test image, used to provide image features related to target feature information for the target classification model. Since the test image is used to obtain target classification results for different process segments, the foreground regions of the test images corresponding to different process segments may differ, and the target feature information in the corresponding target classification results will also differ due to the differences in process segments. The background region is the image of other regions in the test image besides the defect region, containing the region information processed by the corresponding process segment or workstation. Therefore, the background region is used to provide image features related to target attribute information for the target classification model.

[0054] In this embodiment, the target classification model is a deep learning model for defect detection based on the test image. The target classification model obtains the target attribute information and target feature information corresponding to the defect image based on image features in the input test image, and combines the target attribute information and target feature information to form the target classification result corresponding to the test image. However, due to the uninterpretable nature of deep learning, without annotation of the test image, it is difficult to determine whether the target classification result is based on the foreground or background region of the test image. As mentioned above, the foreground region has a high correlation with target feature information, while the target attribute information has a high correlation with the background region. This means that if the target classification result is obtained solely based on the foreground or background region, some information in the target classification result will inevitably have poor interpretability, thus reducing customer trust in the target classification model and target classification result.

[0055] Therefore, after obtaining the target classification result based on the target classification model, this embodiment of the present disclosure designs an attribution analysis model based on the correlation between the target feature information and target attribute information and the foreground region and background region. The model performs correlation analysis on the processing basis of the target feature information and target attribute information in the generated target classification result, thereby ensuring the reliability of the target classification model and the generated target classification result, and thereby improving the customer's trust in the target classification model and the generated target classification result.

[0056] To enable those skilled in the art to better understand the technical solutions of this disclosure, the steps and procedures of the model evaluation method provided in this disclosure will be described in detail below: In specific implementation step S101, firstly, based on the production line image, product images collected at workstations or process sections of the production line are acquired. These product images include all or part of the area of ​​the product to be inspected. Specifically, the images collected at workstations or process sections of the production line are subjected to frame line detection corresponding to the product. The positions of the frame line pixels are areas where the pixel values ​​of the product area and the conveyor device area in the production line image show a significant change. The determined set of frame line pixels is connected to form an image area containing the product. This image area containing the product is then cropped to obtain the product image.

[0057] In some optional implementations, since the production line image is an image acquired by the image acquisition device at the actual transmission device, the pixel values ​​of the acquired production line image and the product image generated based on the production line image may be too high or too low. This results in an overall brightness imbalance, with the production line image or product image being too bright, too small, or uneven in brightness. This can lead to a decrease in the detection accuracy of the test image corresponding to the product image during subsequent defect detection due to brightness issues. Therefore, after obtaining the product image, it is preprocessed. The preprocessing includes at least brightness enhancement and color enhancement. Brightness enhancement adjusts the brightness of the product image to a preset brightness range, and color enhancement adjusts the color of the product image to a preset color range. This ensures that the test image corresponding to the product image has a better processing effect during subsequent defect detection and attribution analysis. It should be noted that the brightness enhancement and color enhancement can employ common preprocessing methods to improve the image quality of the product image, such as gamma calibration. This disclosure does not limit the specific preprocessing methods used.

[0058] After obtaining the product image, at each workstation or process segment, based on the installation process of the process segment, the component area corresponding to each process segment in the product image is obtained as the test image. It should be noted that the test image can be randomly selected from the test images corresponding to the product image in the dataset of the target classification model, or it can be manually selected according to actual needs; this disclosure does not impose any restrictions. The test image includes at least a foreground area and a background area. The foreground area is the defect area image in the component area corresponding to the process segment, and the background area is the image of other areas in the component area corresponding to the process segment besides the defect area image. In this embodiment, since the background area is the image of other areas in the component area corresponding to the process segment besides the defect area image, the differences in installation processes of different process segments can be reflected in the corresponding component areas, allowing the background area of ​​the test image to distinguish the target attribute information of each process segment. Figure 2 A schematic diagram of a test image for performing the method according to this disclosure is shown, such as... Figure 2 As shown, the test image includes a foreground region corresponding to the defect area and a background region corresponding to the component area of ​​the process segment.

[0059] In specific implementation step S102, after obtaining the test image, the test image is input into the defect detection model to classify defects in the test image and output the corresponding target classification result. The defect detection model is a pre-trained deep learning model, which can be a classification model, a segmentation model, or other common defect detection models. Specifically, after the test image is input into the defect detection model, the defect detection model extracts a first image feature from the test image, performs defect analysis on the first image feature, and determines the defect morphology and defect location corresponding to the defect region in the test image, as the target feature information; the defect detection model extracts a second image feature from the test image, performs image analysis on the second image feature, and determines the process section or workstation information corresponding to the test image in the production line, as the target attribute information. The target attribute information and the target feature information are combined to form the target classification result. For example, the target classification result includes a defect data code of a first data length (corresponding to target feature information) and a production line data code of a second data length (corresponding to target attribute information), where the first data length and the second data length are the same.

[0060] In this embodiment, the training process of the defect detection model is as follows: A dataset is acquired and split into a training set, a validation set, and a test set according to a preset ratio. An initial defect detection model is constructed; exemplarily, the initial defect detection model can be ResNet, ResNeXt, etc. The data from the training set is input into the initial defect detection model. Based on a deep learning framework, hyperparameters are set to train the initial defect detection model, and the hyperparameters are optimized based on a preset loss function to obtain the defect detection model. When the performance of the defect detection model on the validation set meets preset validation conditions, the defect detection model is output. Optionally, the deep learning framework can be PyTorch, TensorFlow, etc.

[0061] In specific implementation step S103, the test image is input into the attribution analysis model, and the attribution analysis model outputs an attribution map corresponding to the test image. The attribution map is a heatmap representing the degree of association between each pixel and the target feature information (defect category). For example, the higher the association between the pixel and the target feature information, the larger the pixel value of that pixel in the attribution map. Specifically, firstly, the test image is input into the attribution analysis model, and each pixel in the test image is analyzed based on the attribution analysis model to determine the degree of association between each pixel and the target feature information in the target classification result output by the defect detection model. Based on the degree of association, a grayscale value corresponding to each pixel in the test image is generated. Wherein, if the degree of association between the pixel and the target feature information is greater, the corresponding grayscale value is larger; if the degree of association between the pixel and the target feature information is smaller, the corresponding grayscale value is smaller. Based on the grayscale value obtained by the attribution analysis model for each pixel in the test image, a corresponding initial attribution map is generated. Optionally, the attribution analysis model can be an existing attribution analysis model. For example, the attribution analysis model can be selected from any one of GradCam, HiResCAM, ScoreCAM, GradCAM++, AblationCAM, XGradCAM, EigenCAM, FullGrad, etc. The specific attribution analysis model is not limited herein.

[0062] For example, the attribution analysis model is a GradCam model. After inputting the test image into the attribution analysis model, the gradient weights of each pixel in the target channel and target prediction category in the test image are first obtained based on the gradient of backpropagation. The gradient weights are obtained according to the following formula:

[0063] Where c represents the target prediction category; Let be the logits value corresponding to the predicted category c of the target; A is the feature layer output by the last convolutional layer; k is the k-th target channel in feature layer A; i and j are the horizontal and vertical coordinates of the feature layer; This represents the data at coordinates i, j in the kth target channel of feature layer A; Calculate the gradient weights for each pixel in the target channel k and the target prediction category c; Z is the feature layer size.

[0064] Subsequently, the feature layer A of the target channel k is weighted with the gradient of each pixel under the target channel k and the target prediction category c. Linear weighted fusion is performed, and then normalized using the rectified function ReLU, retaining only the regions that have a positive effect on the target prediction category c, to obtain the initial attribution map corresponding to the test image. The initial attribution map is a feature heatmap corresponding to the target prediction category c, and the target prediction category c corresponds to the target feature information. The linear weighted fusion is performed according to the following formula:

[0065] Where c is the target prediction category; k is the k-th target channel in feature layer A; Calculate the weights for the gradient of each pixel under target channel k and target prediction category c; This refers to the data in the k-th target channel of feature layer A; The rectification function is... This is the initial attribution map corresponding to the target prediction category c.

[0066] In this embodiment of the disclosure, each pixel in the initial attribution map is a pixel representing the degree of association between the pixel and the target feature information (target predicted category). Figure 3 A schematic diagram of an initial attribution graph for performing the method according to this disclosure is shown, such as... Figure 3 As shown, the initial attribution map can be a grayscale image. The grayscale value of each pixel in the initial attribution map represents the degree of association with the target feature information corresponding to the test image. When the degree of association with the target feature information is greater, the grayscale value of the pixel is larger (maximum 255, represented as a white pixel in the initial attribution map); when the degree of association with the target feature information is smaller, the grayscale value of the pixel is smaller (minimum 0, represented as a black pixel in the initial attribution map). In an optional implementation, to facilitate subsequent data processing, the grayscale values ​​of each pixel in the initial attribution map can be normalized so that the grayscale value of each pixel is between 0 and 1, where 0 corresponds to the smallest grayscale value (corresponding to black) and 1 corresponds to the largest grayscale value (corresponding to white).

[0067] After obtaining the initial attribution map, it is binarized to obtain another attribution map. In one optional implementation, a fixed threshold method is used to binarize the initial attribution map. Specifically, a first threshold is first determined, which is a fixed pixel grayscale threshold. For example, for normalized pixel grayscale values, the first threshold is a value between 0 and 1 (e.g., the first threshold is 0.5). Subsequently, based on the first threshold, the grayscale values ​​of each pixel in the initial attribution map are classified. Pixels with grayscale values ​​greater than or equal to the first threshold are designated as first pixels, and pixels with grayscale values ​​less than the first threshold are designated as second pixels. The first pixels are those with a high correlation to the target classification result, and their grayscale values ​​are adjusted to the first grayscale value, which is the maximum grayscale value (corresponding to white). The second pixels are those with a low correlation to the target classification result, and their grayscale values ​​are adjusted to the second grayscale value, which is the minimum grayscale value (corresponding to black), thus obtaining the attribution map.

[0068] For example, Figure 4 A schematic diagram of an attribution graph for performing the method according to this disclosure is shown, such as... Figure 3 and Figure 4 As shown, Figure 3 The initial attribution map includes three gray-level regions A1, A2, and A3, where the gray-level values ​​of the pixels in regions A1, A2, and A3 increase sequentially, with the gray-level value of pixels in region A3 being 1. Compared to Figure 3 The initial attribution map, Figure 4 After binarization, pixels with grayscale values ​​less than the first threshold are adjusted to black pixels with a grayscale value of 0; regions A1 and A2 are both adjusted to black pixels. Pixels in region A3 have grayscale values ​​greater than the first threshold, so the entire region of A3 is adjusted to white pixels, resulting in the following: Figure 4 The attribution diagram shown, Figure 4 The attribution map shown is binarized, and only region A3 is retained.

[0069] In some optional implementations, after obtaining the initial attribution map, binarization can be performed using a fixed proportion method to obtain the attribution map. Specifically, firstly, the grayscale values ​​of each pixel in the initial attribution map are sorted. The sorting method can be either sorting from smallest to largest or from largest to smallest; this disclosure does not limit the sorting method. Subsequently, a second threshold is determined. The second threshold is a fixed proportion threshold, for example, for normalized pixel grayscale values, the second threshold is the set of pixels with a fixed proportion among all pixels in the initial attribution map (e.g., the set of pixels with the largest 10% proportion of grayscale values ​​after sorting from largest to smallest). Based on the second threshold, the pixels are classified to obtain the set of pixels in the sorted initial attribution map that corresponds to the proportion of the second threshold. This set is taken as the third pixel, i.e., the proportion of the third pixel is equal to the second threshold. All other pixels besides the third pixel are taken as the fourth pixel, wherein the grayscale value of the third pixel is greater than the grayscale value of the fourth pixel. Finally, the grayscale value of the third pixel is adjusted to the first grayscale value, and the grayscale value of the fourth pixel is adjusted to the second grayscale value to obtain the attribution map.

[0070] It should be noted that other existing binarization methods can also be used for binarization of the initial attribution map. The specific binarization method can be determined according to the actual situation, and this application does not impose any restrictions here.

[0071] In specific implementation step S104, after obtaining the binarized attribution map, when analyzing the attribution map, it is necessary to first obtain the connected components within it. A connected component is a region composed of connectable pixels with the same grayscale value. Specifically, firstly, an initial pixel is obtained, and a connected component corresponding to the initial pixel is created. The initial pixel is any pixel in the attribution map. Subsequently, in the attribution map, all neighboring pixels of the initial pixel are detected one by one. Neighboring pixels refer to other connectable pixels directly adjacent to the initial pixel. Neighboring pixels with the same grayscale value as the initial pixel are added to the connected component corresponding to the initial pixel; that is, all other connectable pixels with the same grayscale value are added to the connected component corresponding to the initial pixel. The above steps are repeated until the initial pixel has traversed all pixels in the attribution map, obtaining at least one connected component corresponding to the attribution map. Figure 4 For example, Figure 4 All pixels in region A3 have the same grayscale value of 1. Therefore, all pixels in region A3 are divided into the same connected component.

[0072] In one optional implementation, after obtaining at least one connected component corresponding to the attribution map, since there is no limit on the number of pixels contained in each connected component, regions containing fewer pixels (such as regions containing only one or a few pixels) are also considered connected components. This causes the regions corresponding to the connected components in the attribution map or the test image to be scattered in various locations, resulting in a decrease in the accuracy of model evaluation based on the connected components. In other words, since the number of pixels in the connected components is not limited, connected components containing fewer pixels due to noise are also considered for evaluation, thus including connected components located in both the foreground and background regions. Consequently, it is impossible to effectively evaluate the target classification model based on the difference in the distribution of the connected components in the foreground and background regions. Therefore, in this embodiment of the disclosure, after obtaining the connected components, it is necessary to filter the connected components based on the number of pixels, removing noisy connected components with a small number of pixels and retaining only the valid connected components. Specifically, firstly, the number of pixels contained in each connected component is obtained; then, a third threshold is determined, which is a fixed threshold representing the area of ​​the connected component (for example, the third threshold is a threshold for the number of pixels contained in the connected component, such as 20 pixels). Based on the third threshold, the connected components are filtered, and connected components with a number of pixels less than or equal to the third threshold are deleted.

[0073] In this embodiment of the disclosure, after determining the connected components in the attribution graph, the target classification model is evaluated by judging the type of the connected components. Since the target classification result generated by the target classification model includes target attribute information and target feature information, and the target attribute information should be obtained by the target classification model based on the features of the background region, and the target feature information should be obtained by the target classification model based on the features of the foreground region, the reliability of the target classification model can be determined based on the type of connected components in the attribution graph corresponding to the test image. If the connected components of the attribution graph corresponding to the test image include both connected components belonging to the foreground type and connected components belonging to the background type, it indicates that the target classification result generated by the target classification model based on the test image is obtained by simultaneously classifying based on the foreground region and the background region. The target feature information contained therein can be considered to be obtained based on the foreground region, and the target attribute information can be considered to be obtained based on the background region, thereby ensuring the reliability of the target classification model and the corresponding target classification result.

[0074] In this embodiment of the disclosure, after obtaining the at least one connected component, the test image is divided according to the position of the connected component, and the regions of the test image corresponding to the connected components are analyzed. In some optional embodiments, since the attribution map and the test image have the same size, each pixel in the test image has a one-to-one corresponding pixel in the attribution map. Based on the position of the pixel in the attribution map and the test image, the pixels at the same position in the attribution map and the test image are determined. The pixel value of the pixel in the test image is added to the gray value of the pixel at the same position in the attribution map, so that the brightness of the pixels corresponding to the pixels in the connected components of the attribution map in the test image remains unchanged, and the brightness of the pixels corresponding to the pixels in the non-connected components of the attribution map is reduced by a preset proportion (e.g., 50%), thus obtaining the overlay image.

[0075] Subsequently, for the overlay map obtained based on the superposition of each connected component, the overlap between each connected component and the foreground region of the test image is obtained. Specifically, based on the overlay map, the pixels of the foreground region of the test image and the pixels of each connected component in the attribution map are obtained; based on the pixels of the foreground region and the pixels of each connected component, the IoU value between each connected component and the foreground region is determined as the overlap. The IoU value is determined as follows: the pixels common to the foreground region and each connected component are determined as intersection pixels, and then the number of intersection pixels corresponding to each connected component is determined; all pixels contained in the foreground region and each connected component are determined as union pixels, and then the number of union pixels is determined; the ratio of the number of intersection pixels to the number of union pixels is obtained as the IoU value between each connected component and the foreground region of the test image, and as the overlap corresponding to each connected component.

[0076] Finally, based on the overlap, the type of each connected component is determined. Specifically, first, a fourth threshold is determined, which is an overlap threshold, for example, 0.5. Based on the fourth threshold, the overlap corresponding to each connected component is filtered. If the overlap is greater than or equal to the fourth threshold, the connected component is considered to have a large overlap with the foreground region of the test image and a small overlap with the background region of the test image, and the type of the connected component is determined to be the foreground type. If the overlap is less than the fourth threshold, the connected component is considered to have a small overlap with the foreground region of the test image and a large overlap with the background region of the test image, and the type of the connected component is determined to be the background type. It should be noted that the type of a connected component can only be either the background type or the foreground type, and cannot be both simultaneously, to prevent a decrease in the accuracy of connected component analysis caused by a single connected component simultaneously satisfying both types.

[0077] In one alternative implementation, such as Figure 2 As shown, when the area of ​​the installation component corresponding to the test image includes multiple repeating units arranged periodically, the test image is composed of repeating units that appear repeatedly in the same pattern. For example, the test image is an array image of subpixels, and the components of the test image include a periodically arranged first subpixel 1, a second subpixel 2, and a third subpixel 3. The type of the connected component can be determined by the similarity between the repeating unit corresponding to the test image and other repeating units of the same type. If the repeating unit corresponding to the connected component has a defect (i.e., contains a foreground region), the similarity between the repeating unit corresponding to the connected component and other repeating units of the same type will decrease due to the presence of the foreground region; conversely, if the repeating unit corresponding to the connected component does not have a defect (i.e., does not contain a foreground region), the similarity between the repeating unit corresponding to the connected component and other repeating units of the same type will be high. Therefore, the embodiments of this disclosure can analyze the connected components in the attribution graph based on the test image, determine the similarity between each connected component and the repeating unit corresponding to the connected component in the test image, and determine the type of the connected component by the difference in similarity between the repeating unit corresponding to the connected component and other repeating units of the same type.

[0078] Specifically, firstly, based on each connected component, an image region corresponding to each connected component in the test image is determined as a target image region; in the test image, a target repeating unit corresponding to the target image region and the periodic arrangement of the target repeating unit in the test image are determined; based on the periodic arrangement, the image region corresponding to the nearest repeating unit of the same type of the target repeating unit is determined as the comparison image region; and the similarity between the target image region and the comparison image region is determined as the similarity corresponding to each connected component. Finally, a fifth threshold is determined, which is the similarity threshold. Based on the fifth threshold, the similarity corresponding to each connected component is filtered: if the similarity is greater than or equal to the fifth threshold, the target image region corresponding to the connected component is considered to have a high similarity with the comparison image region. In this case, the target image region corresponding to the connected component can be considered to have no defective regions, and the type of the connected component is determined to be the background type; if the similarity is less than the fifth threshold, the target image region corresponding to the connected component is considered to have a low similarity with the comparison image region. In this case, the target image region corresponding to the connected component can be considered to have a low similarity due to the presence of defective regions, resulting in the same repeating units having low similarity. In this case, the type of the connected component is determined to be the foreground type.

[0079] by Figure 2 For example, Figure 2 The test image contains three periodically arranged sub-pixels: a first sub-pixel 1, a second sub-pixel 2, and a third sub-pixel 3. The defective region is located within the repeating unit of the first sub-pixel 1. Figure 4 The unique connected component A3 in the corresponding attribution graph can be identified as B1 in the target image region corresponding to the test image. The repeating unit corresponding to B1 is the first sub-pixel 1. Based on the periodic arrangement of the first sub-pixel 1 in the test image, the following can be determined: Figure 2 The comparison image region in A3 is the first sub-pixel 1 of the preset period length to the left of the target image region corresponding to A3. Figure 2 The similarity between the comparison image region B0 and the target image region B1 is calculated. Since the defective region exists in the target image region B1, the similarity between the comparison image region B0 and the target image region B1 is low, thus determining that the type of the connected component A3 is foreground type.

[0080] This embodiment of the disclosure determines whether the target classification result generated when the test image is classified as a defect based on the target classification model is generated based on both foreground and background connected components by determining the type of each connected component in the attribution graph of the corresponding test image. Thus, the reliability of the target classification model and the target classification result can be determined by the number of connected components belonging to the foreground and background types.

[0081] In specific implementation step S105, it is determined whether the target classification model is a qualified target classification model based on the type of the connected component, and whether the target classification result is a reliable result based on the type of the connected component. Since the target feature information has high reliability when obtained based on the foreground region, and the target attribute information has high reliability when obtained based on the background region, this embodiment of the disclosure pre-sets a first condition. The first condition is that the connected component contains at least one connected component of the foreground type and at least one connected component of the background type. When the type of the connected component satisfies the first condition, it can be determined that the target attribute information in the target classification result is obtained based on the background region (connected component corresponding to the background type), and the target feature information in the target classification result is obtained based on the foreground region (connected component corresponding to the foreground type), thereby determining the target classification model as a qualified target classification model and the target classification result as a reliable result.

[0082] In an optional implementation, if the type of the connected component does not meet the first condition, and it is impossible to simultaneously determine that the target attribute information in the target classification result is based on the background region (connected component corresponding to the background type) and the target feature information in the target classification result is based on the foreground region (connected component corresponding to the foreground type), then the target classification model is determined to be an unqualified target classification model, and the target classification result is determined to be an unreliable result.

[0083] It should be noted that the first condition mentioned above is the minimum standard for judging the reliability of the target classification model and the target classification result based on the connected component type. In the actual process of model evaluation based on the connected component type, more stringent judgment conditions can be formulated according to the actual scenario (for example, the connected component contains at least two connected components of the foreground type, and the connected component contains at least one connected component of the background type). This disclosure does not impose any restrictions on this.

[0084] This disclosure provides a model evaluation method, comprising: acquiring a test image, the test image including a foreground region and a background region; inputting the test image into a target classification model to obtain a target classification result, the target classification result including at least target feature information and target attribute information; inputting the test image into an attribution analysis model to obtain an attribution graph corresponding to the test image; using the test image as a reference, analyzing the connected components in the attribution graph to determine the type of the connected components, the type of the connected components being either a foreground type or a background type; determining whether the target classification model is a qualified target classification model based on the type of the connected components; and determining whether the target classification result is a reliable result based on the type of the connected components. This disclosure, by acquiring the attribution graph corresponding to the test image and based on the type of the connected components in the attribution graph, determines whether the data support for the target classification model in defect classification simultaneously includes both foreground and background regions. If both foreground and background regions are simultaneously included, the target classification model and the target classification result are determined to be reliable results, thereby improving the reliability of the model.

[0085] Based on the same inventive concept, this disclosure provides a model evaluation device. Figure 5 A schematic diagram of a model evaluation apparatus for performing the method according to this disclosure is shown, such as... Figure 5 As shown, the model evaluation device includes: An image acquisition module is used to acquire a test image, the test image including a foreground region and a background region; The defect classification module is used to input the test image into the target classification model to obtain the target classification result, wherein the target classification result includes at least target feature information and target attribute information. The attribution graph module is used to input the test image into the attribution analysis model and obtain the attribution graph corresponding to the test image; The connected component analysis module is used to analyze the connected components in the attribution graph based on the test image, and determine the type of the connected component, which is either foreground type or background type. The judgment module is used to determine whether the target classification model is a qualified target classification model based on the type of the connected component, and to determine whether the target classification result is a reliable result based on the type of the connected component.

[0086] In some alternative implementations, the attribution graph module includes: The first initial attribution map submodule is used to input the test image into the attribution analysis model to obtain an initial attribution map, wherein the gray value of each pixel in the initial attribution map represents the degree of correlation between each pixel and the target feature information. The first threshold submodule is used to classify the gray value of each pixel in the initial attribution map based on the first threshold, and to take the pixels with gray values ​​greater than or equal to the first threshold as the first pixel and the pixels with gray values ​​less than the first threshold as the second pixel. The first attribution map submodule is used to adjust the gray value of the first pixel to a first gray value and the gray value of the second pixel to a second gray value to obtain the attribution map.

[0087] In some alternative implementations, the attribution graph module includes: The second initial attribution map submodule is used to input the test image into the attribution analysis model to obtain an initial attribution map, wherein the gray value of each pixel in the initial attribution map represents the degree of correlation between each pixel and the target feature information. The pixel sorting submodule is used to sort the grayscale values ​​of each pixel in the initial attribution map; The second threshold submodule is used to classify the pixels based on the second threshold to obtain the third pixel and the fourth pixel, wherein the gray value of the third pixel is greater than the gray value of the fourth pixel, and the proportion of the number of the third pixel is equal to the second threshold. The second attribution map submodule is used to adjust the gray value of the third pixel to a first gray value and the gray value of the fourth pixel to a second gray value to obtain the attribution map.

[0088] In some optional implementations, the attribution graph module further includes: A connected component creation submodule is used to obtain an initial pixel and create a connected component corresponding to the initial pixel, wherein the initial pixel is any pixel in the attribution graph; The neighborhood detection submodule is used to detect all neighboring pixels of the initial pixel in the attribution graph one by one, and add neighboring pixels with the same gray value as the initial pixel to the connected component corresponding to the initial pixel. The connected component submodule is used to repeat the above steps until the initial pixel point traverses all pixels of the attribution graph to obtain at least one connected component corresponding to the attribution graph.

[0089] In some optional implementations, the attribution graph module further includes: The connected component pixel submodule is used to obtain the number of pixels contained in each connected component; The third threshold submodule is used to filter the connected components based on the third threshold and delete connected components in which the number of pixels is less than or equal to the third threshold.

[0090] In some optional implementations, the connected component analysis module includes: The overlay submodule is used to overlay the test image with the attribution map to obtain an overlay map; The overlap submodule is used to obtain the overlap between each connected component and the foreground region of the test image based on the overlay graph; The first connected component analysis submodule is used to determine the type of each connected component based on the overlap.

[0091] In some optional implementations, the overlay submodule includes: Overlay pixel units are used to determine pixels at the same positions in the attribution map and the test image; The overlay unit is used to add the pixel value of the pixel in the test image to the gray value of the pixel at the same position in the attribution map to obtain the overlay map; Wherein, the brightness of the pixels corresponding to the connected components in the overlay image and the test image are the same; the brightness of the pixels corresponding to the non-connected components in the overlay image is a preset ratio of the brightness of the pixels corresponding to the non-connected components in the test image.

[0092] In some optional implementations, the overlap submodule includes: The pixel determination unit is used to obtain the pixels in the foreground region of the test image and the pixels in each connected component of the attribution map based on the overlay map. The intersection and union pixel unit is used to determine the number of intersection pixels and the number of union pixels corresponding to each connected component based on the pixels of the foreground region and the pixels of each connected component. The intersection pixels are the pixels common to the foreground region and each connected component, and the union pixels are all the pixels contained in the foreground region and each connected component. An overlap unit is used to obtain the ratio of the number of intersection pixels to the number of union pixels, as the overlap between each connected component and the foreground region of the test image.

[0093] In some optional implementations, the first connected component analysis submodule includes: The fourth threshold unit is used to filter the overlap of each connected component based on the fourth threshold. The first foreground type unit is used to determine the type of the connected component as the foreground type when the overlap is greater than or equal to the fourth threshold. A first background type unit is used to determine the type of the connected component as the background type when the overlap is less than the fourth threshold.

[0094] In some optional implementations, the connected component analysis module further includes: The similarity submodule is used to analyze the connected components in the attribution graph based on the test image, and determine the similarity between each connected component and the corresponding repeating unit of the connected component in the test image. The second connected component analysis submodule is used to determine the type of each connected component based on the similarity.

[0095] In some optional implementations, the similarity submodule includes: The target image region unit is used to determine, based on each of the connected components, the image region in the test image corresponding to each of the connected components, as the target image region; A repetition determination unit is used to determine the target repetition unit corresponding to the target image region in the test image; A comparison image region unit is used to determine the image region corresponding to a repeating unit of the same type as the target repeating unit, and use it as the comparison image region. A similarity unit is used to determine the similarity between the target image region and the comparison image region.

[0096] In some optional implementations, the second connected component analysis submodule includes: The fifth threshold unit is used to filter the similarity corresponding to each connected component based on the fifth threshold. The second background type unit is used to determine the type of the connected component as the background type when the similarity is greater than or equal to the fifth threshold. The second foreground type unit is used to determine the type of the connected component as the foreground type when the similarity is less than the fifth threshold.

[0097] In some optional implementations, the first initial attribution graph submodule includes: The gradient calculation weight unit is used to input the test image into the attribution analysis model and obtain the gradient calculation weight of each pixel in the target channel and target prediction category in the test image. The initial attribution map unit is used to linearly weight and fuse the feature layer of the target channel with the gradient calculation weights of each pixel under the target channel and the target prediction category, and then normalize it through a rectification function to obtain the initial attribution map corresponding to the test image. The initial attribution map is a feature heatmap corresponding to the target prediction category, and the target prediction category corresponds to the target feature information.

[0098] In some optional implementations, the determination module includes: The first judgment submodule is used to determine the defect classification model as a qualified defect classification model and the defect classification result as a reliable result when the type of the connected component satisfies the first condition. The first condition is that the connected component contains at least one connected component of the foreground type and the connected component contains at least one connected component of the background type. The second judgment submodule is used to determine the target classification model as an unqualified target classification model and the target classification result as an unreliable result when the type of the connected component does not meet the first condition.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0100] The various component embodiments of this disclosure can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the computing processing device according to embodiments of this disclosure. This disclosure can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing some or all of the methods described herein. Such an implementation of this disclosure can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0101] For example, Figure 6 A schematic diagram of the structure of a computing processing apparatus for performing the method according to this disclosure is shown, such as... Figure 6As shown, the computing processing device 100 conventionally includes a processor 120 and a computer program product or computer-readable medium in the form of a memory 110. The memory 110 can be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. The memory 110 has storage space for program code for performing any of the method steps described above. For example, the storage space for program code may include various program codes for implementing the various steps in the methods described above. This program code can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. Such computer program products are typically portable or fixed storage units. The storage unit may have... Figure 6 The memory 110 in the computing processing device is similarly arranged as storage segments, storage spaces, etc. Program code can be compressed, for example, in an appropriate form. Typically, the storage unit contains computer-readable code, i.e., code that can be read by a processor such as 120, which, when run by the computing processing device, causes the computing processing device to perform the various steps in the methods described above.

[0102] The terms "an embodiment," "embodiment," or "one or more embodiments" as used herein mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this disclosure. Furthermore, please note that the examples of the phrase "in one embodiment" do not necessarily all refer to the same embodiment.

[0103] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this disclosure may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0104] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This disclosure can be implemented by means of hardware comprising a plurality of different elements and by means of a suitably programmed computer. In a unit claim enumerating a plurality of means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A model evaluation method, characterized in that, The method includes: Acquire a test image, which includes a foreground region and a background region; The test image is input into the target classification model to obtain the target classification result, which includes at least target feature information and target attribute information. Input the test image into the attribution analysis model to obtain the attribution map corresponding to the test image; Based on the test image, the connected components in the attribution graph are analyzed to determine the type of the connected components, which is either foreground type or background type. The type of the connected component determines whether the target classification model is a qualified target classification model, and the type of the connected component determines whether the target classification result is a reliable result.

2. The model evaluation method according to claim 1, characterized in that, The step of inputting the test image into the attribution analysis model to obtain the attribution map corresponding to the test image includes: The test image is input into the attribution analysis model to obtain an initial attribution map. The gray value of each pixel in the initial attribution map represents the degree of correlation between each pixel and the target feature information. The grayscale value of each pixel in the initial attribution map is classified based on a first threshold. Pixels with grayscale values ​​greater than or equal to the first threshold are designated as first pixels, and pixels with grayscale values ​​less than the first threshold are designated as second pixels. The grayscale value of the first pixel is adjusted to a first grayscale value, and the grayscale value of the second pixel is adjusted to a second grayscale value to obtain the attribution map.

3. The model evaluation method according to claim 1, characterized in that, The step of inputting the test image into the attribution analysis model to obtain the attribution map corresponding to the test image includes: The test image is input into the attribution analysis model to obtain an initial attribution map. The gray value of each pixel in the initial attribution map represents the degree of correlation between each pixel and the target feature information. Sort the grayscale values ​​of each pixel in the initial attribution map; The pixels are classified based on the second threshold to obtain third pixels and fourth pixels, wherein the gray value of the third pixel is greater than the gray value of the fourth pixel, and the proportion of the number of third pixels is equal to the second threshold. The grayscale value of the third pixel is adjusted to the first grayscale value, and the grayscale value of the fourth pixel is adjusted to the second grayscale value to obtain the attribution map.

4. The model evaluation method according to claim 1, characterized in that, After obtaining the attribution map corresponding to the test image, the method further includes: Obtain an initial pixel and create a connected component corresponding to the initial pixel, wherein the initial pixel is any pixel in the attribution graph; In the attribution graph, all neighboring pixels of the initial pixel are detected one by one, and neighboring pixels with the same gray value as the initial pixel are added to the connected component corresponding to the initial pixel. Repeat the above steps until the initial pixel traverses all pixels of the attribution graph, obtaining at least one connected component corresponding to the attribution graph.

5. The model evaluation method according to claim 1, characterized in that, After obtaining at least one connected component corresponding to the attribution graph, the method further includes: Get the number of pixels contained in each connected component; Based on the third threshold, the connected components are filtered, and connected components with a number of pixels less than or equal to the third threshold are deleted.

6. The model evaluation method according to claim 1, characterized in that, The step of analyzing the connected components in the attribution graph based on the test image to determine the type of the connected components includes: The test image is overlaid with the attribution map to obtain an overlay image; Based on the overlay map, the overlap between each connected component and the foreground region of the test image is obtained; Based on the overlap, the type of each connected component is determined.

7. The model evaluation method according to claim 6, characterized in that, The step of overlaying the test image with the attribution map to obtain an overlay map includes: Identify the pixels at the same location in the attribution map and the test image; The pixel values ​​of the pixels in the test image are added to the gray values ​​of the pixels at the same position in the attribution map to obtain the overlay image; Wherein, the brightness of the pixels corresponding to the connected components in the overlay image and the test image are the same; the brightness of the pixels corresponding to the non-connected components in the overlay image is a preset ratio of the brightness of the pixels corresponding to the non-connected components in the test image.

8. The model evaluation method according to claim 6, characterized in that, The step of obtaining the overlap between each connected component and the foreground region of the test image based on the overlay map includes: Based on the overlay map, the pixels of the foreground region of the test image and the pixels of each connected component in the attribution map are obtained. Based on the pixels of the foreground region and the pixels of each connected component, determine the number of intersection pixels and the number of union pixels corresponding to each connected component. The intersection pixels are the pixels common to the foreground region and each connected component, and the union pixels are all the pixels contained in the foreground region and each connected component. The ratio of the number of intersection pixels to the number of union pixels is obtained as the overlap degree between each connected component and the foreground region of the test image.

9. The model evaluation method according to claim 6, characterized in that, Determining the type of each connected component based on the overlap includes: Based on the fourth threshold, the overlap corresponding to each of the connected components is filtered; If the overlap is greater than or equal to the fourth threshold, the type of the connected component is determined as the foreground type; If the overlap is less than the fourth threshold, the type of the connected component is determined to be the background type.

10. The model evaluation method according to claim 1, characterized in that, The test image comprises multiple repeating units arranged periodically. The analysis of connected components in the attribution graph, based on the test image, to determine the type of the connected components includes: Based on the test image, the connected components in the attribution graph are analyzed to determine the similarity between each connected component and the corresponding repeating unit in the test image. Based on the similarity, the type of each connected component is determined.

11. The model evaluation method according to claim 10, characterized in that, The step of analyzing connected components in the attribution graph based on the test image to determine the similarity between each connected component and its corresponding repeating unit in the test image includes: Based on each of the connected components, the image region in the test image corresponding to each of the connected components is determined as the target image region; In the test image, the target repeating unit corresponding to the target image region is determined; The image region corresponding to the repeating unit of the same type as the target repeating unit is determined as the comparison image region; Determine the similarity between the target image region and the comparison image region.

12. The model evaluation method according to claim 10, characterized in that, Determining the type of each connected component based on the similarity includes: Based on the fifth threshold, the similarity corresponding to each of the connected components is filtered; If the similarity is greater than or equal to the fifth threshold, the type of the connected component is determined as the background type; If the similarity is less than the fifth threshold, the type of the connected component is determined as the foreground type.

13. The model evaluation method according to claim 2, characterized in that, The step of inputting the test image into the attribution analysis model to obtain the initial attribution map includes: The test image is input into the attribution analysis model to obtain the gradient calculation weights of each pixel in the target channel and target prediction category in the test image. The feature layer of the target channel is linearly weighted and fused with the gradient calculation weights of each pixel under the target channel and the target prediction category, and then normalized by a rectifier function to obtain the initial attribution map corresponding to the test image. The initial attribution map is a feature heatmap corresponding to the target prediction category, and the target prediction category corresponds to the target feature information.

14. The model evaluation method according to any one of claims 1 to 13, characterized in that, The step of determining whether the target classification model is a qualified target classification model based on the type of the connected components, and determining whether the target classification result is a reliable result based on the type of the connected components, includes: If the type of the connected component satisfies the first condition, the defect classification model is determined to be a qualified defect classification model, and the defect classification result is determined to be a reliable result. The first condition is that the connected component contains at least one connected component of the foreground type and the connected component contains at least one connected component of the background type. If the type of the connected component does not meet the first condition, the target classification model is determined to be an unqualified target classification model, and the target classification result is determined to be an unreliable result.

15. A model evaluation device, characterized in that, The device includes: An image acquisition module is used to acquire a test image, the test image including a foreground region and a background region; The defect classification module is used to input the test image into the target classification model to obtain the target classification result, wherein the target classification result includes at least target feature information and target attribute information. The attribution graph module is used to input the test image into the attribution analysis model and obtain the attribution graph corresponding to the test image; The connected component analysis module is used to analyze the connected components in the attribution graph based on the test image, and determine the type of the connected component, which is either foreground type or background type. The judgment module is used to determine whether the target classification model is a qualified target classification model based on the type of the connected component, and to determine whether the target classification result is a reliable result based on the type of the connected component.

16. A computing processing device, characterized in that, include: Memory containing computer-readable code; as well as One or more processors, when the computer-readable code is executed by the one or more processors, the computing processing device performs the model evaluation method as described in any one of claims 1-14.

17. A computer program product, characterized in that, Includes computer-readable code, which, when run on a computing processing device, causes the computing processing device to perform the model evaluation method according to any one of claims 1-14.

18. A computer-readable medium, characterized in that, It stores the computer program product as described in claim 17.

Images