Model evaluation method and apparatus, device, program, and medium

Abstract

Embodiments of the present disclosure relate to the field of deep learning, and provide a model evaluation method and apparatus, a device, a program, and a medium. The method comprises: acquiring a test image, the test image comprising a foreground area and a background area; inputting the test image into a target classification model, to obtain a target classification result, the target classification result at least comprising target feature information and target attribute information; inputting the test image into an attribution analysis model, to acquire an attribution graph corresponding to the test image; using the test image as a reference, analyzing a connected component in the attribution graph, and determining a type of the connected component, the type of the connected component being a foreground type or a background type; on the basis of the type of the connected component, determining whether the target classification model is a qualified target classification model, and on the basis of the type of the connected component, determining whether the target classification result is a trusted result.

Description

Model evaluation method, device, equipment, program and medium

[0001] The present disclosure claims priority to the Chinese patent application No. 2024104517665, filed on April 15, 2024, and entitled "A model evaluation method, device, equipment, program and medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] Embodiments of the present disclosure relate to the field of deep learning, in particular, to a model evaluation method, device, equipment, program and medium. BACKGROUND

[0003] In industrial defect detection, annotating the specific position of defects requires a lot of manpower and time, so in many cases, only the local image (such as 256x256 resolution) containing defects is cropped from the large image (such as 1024x1024 resolution) of the product, and then a deep learning algorithm is used to classify the cropped local image to determine the category of the defect, and different ways are taken according to the different categories of defects (such as repair, ignore or destroy). The category of the defect is determined by two factors, one is which process section the defect comes from, and the other is the shape and position of the defect itself.

[0004] The defect classification algorithm based on deep learning has achieved great success in the field of industrial manufacturing, but when the specific position of the above defect scene is not annotated, it is difficult to determine whether the target classification model makes a decision based on the foreground or the background due to the unexplainability of deep learning, where the foreground determines the shape and position of the defect itself, and the background determines which process section the defect comes from. If the defect category output by the target classification model is obtained only by the foreground or only by the background, the reliability and universality of the model will be relatively low. Therefore, how to improve the reliability and universality of the target classification model has become a problem to be solved in the field. SUMMARY

[0005] Embodiments of the present disclosure provide a model evaluation method, device, equipment, program and medium, aiming to solve the problem of how to improve the reliability and universality of the target classification model.

[0006] In a first aspect, a model evaluation method is provided, the method comprising:

[0007] obtaining a test image, the test image comprising a foreground region and a background region;

[0008] inputting the test image into a target classification model to obtain a target classification result, the target classification result comprising at least target feature information and target attribute information;

[0009] inputting the test image into an attribution analysis model to obtain an attribution graph corresponding to the test image;

[0010] analyzing a connected domain in the attribution graph based on the test image to determine a type of the connected domain, the type of the connected domain being a foreground type or a background type;

[0011] determining whether the target classification model is a qualified target classification model according to the type of the connected domain, and determining whether the target classification result is a credible result according to the type of the connected domain.

[0012] In some optional embodiments, the inputting the test image into an attribution analysis model to obtain an attribution graph corresponding to the test image comprises:

[0013] inputting the test image into the attribution analysis model to obtain an initial attribution graph, a gray value of each pixel point in the initial attribution graph representing an association degree of each pixel point with the target feature information;

[0014] classifying the gray value of each pixel point in the initial attribution graph based on a first threshold value, taking a pixel point with a gray value greater than or equal to the first threshold value as a first pixel point, and taking a pixel point with a gray value less than the first threshold value as a second pixel point;

[0015] adjusting the gray value of the first pixel point to a first gray value, and adjusting the gray value of the second pixel point to a second gray value to obtain the attribution graph.

[0016] In some optional embodiments, the inputting the test image into an attribution analysis model to obtain an attribution graph corresponding to the test image comprises:

[0017] inputting the test image into the attribution analysis model to obtain an initial attribution graph, a gray value of each pixel point in the initial attribution graph representing an association degree of each pixel point with the target feature information;

[0018] sorting the gray value of each pixel point in the initial attribution graph;

[0019] classifying the pixel points based on a second threshold value to obtain third pixel points and fourth pixel points, wherein the gray value of the third pixel point is greater than the gray value of the fourth pixel point, and the number ratio of the third pixel points is equal to the second threshold value;

[0020] adjusting the gray value of the third pixel point to a first gray value, and adjusting the gray value of the fourth pixel point to a second gray value to obtain the attribution graph.

[0021] In some optional embodiments, after the attribution graph corresponding to the test image is obtained, the method further comprises:

[0022] An initial pixel point is obtained, and a connected domain corresponding to the initial pixel point is created, the initial pixel point being any one of the pixel points in the attribution graph;

[0023] In the attribution graph, all neighborhood pixel points of the initial pixel point are detected one by one, and the neighborhood pixel points having the same gray value as the initial pixel point are added to the connected domain corresponding to the initial pixel point;

[0024] The above steps are repeated until the initial pixel point traverses all the pixel points in the attribution graph, and at least one connected domain corresponding to the attribution graph is obtained.

[0025] In some optional embodiments, after the at least one connected domain corresponding to the attribution graph is obtained, the method further comprises:

[0026] The number of pixel points contained in each connected domain is obtained;

[0027] Based on a third threshold, the connected domains are screened, and the connected domains having the number of pixel points less than or equal to the third threshold are deleted.

[0028] In some optional embodiments, the analysis of the connected domains in the attribution graph based on the test image to determine the type of the connected domains comprises:

[0029] The test image and the attribution graph are superimposed to obtain a superimposed graph;

[0030] Based on the superimposed graph, the overlap degree of each connected domain and the foreground region of the test image is obtained;

[0031] Based on the overlap degree, the type of each connected domain is determined.

[0032] In some optional embodiments, the superimposition of the test image and the attribution graph to obtain a superimposed graph comprises:

[0033] The pixel points at the same position in the attribution graph and the test image are determined;

[0034] The pixel value of the pixel point in the test image is added to the gray value of the pixel point at the same position in the attribution graph to obtain the superimposed graph;

[0035] The superimposed graph has the same brightness as the pixel point corresponding to the connected domain in the test image; and the brightness of the pixel point corresponding to the non-connected domain in the superimposed graph is a preset proportion of the brightness of the pixel point corresponding to the non-connected domain in the test image.

[0036] In some optional embodiments, the obtaining of the overlap degree of each connected domain and the foreground region of the test image based on the superimposed graph comprises:

[0037] The superimposed graph has the same brightness as the pixel point corresponding to the connected domain in the test image; and the brightness of the pixel point corresponding to the non-connected domain in the superimposed graph is a preset proportion of the brightness of the pixel point corresponding to the non-connected domain in the test image.

[0038] The superimposed graph has the same brightness as the pixel point corresponding to the connected domain in the test image; and the brightness of the pixel point corresponding to the non-connected domain in the superimposed graph is a preset proportion of the brightness of the pixel point corresponding to the non-connected domain in the test image.

[0039] The superimposed graph has the same brightness as the pixel point corresponding to the connected domain in the test image; and the brightness of the pixel point corresponding to the non-connected domain in the superimposed graph is a preset proportion of the brightness of the pixel point corresponding to the non-connected domain in the test image.

[0040] In some optional embodiments, the determining of the type of each connected domain based on the overlap degree comprises:

[0041] The superimposed graph has the same brightness as the pixel point corresponding to the connected domain in the test image; and the brightness of the pixel point corresponding to the non-connected domain in the superimposed graph is a preset proportion of the brightness of the pixel point corresponding to the non-connected domain in the test image.

[0042] In the case where the overlap degree is greater than or equal to the fourth threshold value, the type of the connected domain is determined as the foreground type.

[0043] In the case where the overlap degree is less than the fourth threshold value, the type of the connected domain is determined as the background type.

[0044] In some optional embodiments, the test image comprises a plurality of repeating units arranged periodically, and the analyzing of the connected domains in the attribution graph based on the test image and the determining of the type of each connected domain comprises:

[0045] The superimposed graph has the same brightness as the pixel point corresponding to the connected domain in the test image; and the brightness of the pixel point corresponding to the non-connected domain in the superimposed graph is a preset proportion of the brightness of the pixel point corresponding to the non-connected domain in the test image.

[0046] The superimposed graph has the same brightness as the pixel point corresponding to the connected domain in the test image; and the brightness of the pixel point corresponding to the non-connected domain in the superimposed graph is a preset proportion of the brightness of the pixel point corresponding to the non-connected domain in the test image.

[0047] In some optional embodiments, the analyzing, based on the test image, the connected domains in the attribution graph, determining the similarity between each of the connected domains and the corresponding repeating unit of the connected domain in the test image, comprises:

[0048] Based on each of the connected domains, determining the image region corresponding to each of the connected domains in the test image as a target image region;

[0049] In the test image, determining the target repeating unit corresponding to the target image region;

[0050] Determining the image region corresponding to the repeating unit of the same type as the target repeating unit as a contrast image region;

[0051] Determining the similarity between the target image region and the contrast image region.

[0052] In some optional embodiments, the determining, based on the similarity, the type of each of the connected domains, comprises:

[0053] Based on a fifth threshold, screening the similarity corresponding to each of the connected domains;

[0054] In the case where the similarity is greater than or equal to the fifth threshold, determining the type of the connected domain as the background type;

[0055] In the case where the similarity is less than the fifth threshold, determining the type of the connected domain as the foreground type.

[0056] In some optional embodiments, the inputting the test image into an attribution analysis model to obtain an initial attribution graph, comprises:

[0057] Inputting the test image into the attribution analysis model to obtain the gradient calculation weight of each pixel under the target channel and target prediction category in the test image;

[0058] Linearly weighting and fusing the feature layer of the target channel and the gradient calculation weight of each pixel under the target channel and target prediction category, and regularizing through a rectifier function to obtain the initial attribution graph corresponding to the test image, the initial attribution graph being a feature heat map corresponding to the target prediction category, the target prediction category corresponding to the target feature information.

[0059] In some optional embodiments, the determining, according to the type of the connected domain, whether the target classification model is a qualified target classification model, and determining, according to the type of the connected domain, whether the target classification result is a trusted result, comprises:

[0060] In a case where the type of the connected domain satisfies a first condition, the defect classification model is determined as a qualified defect classification model, and the defect classification result is determined as a reliable result, the first condition being that the connected domain contains at least one connected domain of the foreground type and contains at least one connected domain of the background type.

[0061] In a case where the type of the connected domain does not satisfy the first condition, the target classification model is determined as an unqualified target classification model, and the target classification result is determined as an unreliable result.

[0062] A second aspect of the embodiments of the present disclosure provides a model evaluation device, the device comprising:

[0063] An image acquisition module configured to acquire a test image, the test image comprising a foreground region and a background region;

[0064] A defect classification module configured to input the test image into a target classification model to obtain a target classification result, the target classification result comprising at least target feature information and target attribute information;

[0065] An attribution graph module configured to input the test image into an attribution analysis model to obtain an attribution graph corresponding to the test image;

[0066] A connected domain analysis module configured to analyze a connected domain in the attribution graph based on the test image to determine a type of the connected domain, the type of the connected domain being a foreground type or a background type;

[0067] A judgment module configured to determine whether the target classification model is a qualified target classification model according to the type of the connected domain, and determine whether the target classification result is a reliable result according to the type of the connected domain.

[0068] In some optional embodiments, the attribution graph module comprises:

[0069] A first initial attribution graph submodule configured to input the test image into the attribution analysis model to obtain an initial attribution graph, a gray value of each pixel point in the initial attribution graph representing an association degree of each pixel point with the target feature information;

[0070] A first threshold submodule configured to classify the gray value of each pixel point in the initial attribution graph based on a first threshold, and regard a pixel point with a gray value greater than or equal to the first threshold as a first pixel point, and regard a pixel point with a gray value less than the first threshold as a second pixel point;

[0071] The first attribution map submodule is configured to adjust the gray value of the first pixel point to a first gray value and adjust the gray value of the second pixel point to a second gray value, so as to obtain the attribution map.

[0072] In some optional embodiments, the attribution map module comprises:

[0073] The second initial attribution map submodule is configured to input the test image into an attribution analysis model to obtain an initial attribution map, wherein the gray value of each pixel point in the initial attribution map represents the correlation degree of each pixel point with the target feature information.

[0074] The pixel sorting submodule is configured to sort the gray values of the pixel points in the initial attribution map.

[0075] The second threshold submodule is configured to classify the pixel points based on a second threshold to obtain third pixel points and fourth pixel points, wherein the gray value of the third pixel point is greater than the gray value of the fourth pixel point, and the number ratio of the third pixel points is equal to the second threshold.

[0076] The second attribution map submodule is configured to adjust the gray value of the third pixel point to a first gray value and adjust the gray value of the fourth pixel point to a second gray value, so as to obtain the attribution map.

[0077] In some optional embodiments, the attribution map module further comprises:

[0078] The connected domain creation submodule is configured to obtain an initial pixel point and create a connected domain corresponding to the initial pixel point, wherein the initial pixel point is any pixel point in the attribution map.

[0079] The neighborhood detection submodule is configured to detect all neighborhood pixel points of the initial pixel point in the attribution map one by one, and add the neighborhood pixel points with the same gray value as the initial pixel point to the connected domain corresponding to the initial pixel point.

[0080] The connected domain submodule is configured to repeat the above steps until the initial pixel point traverses all pixel points in the attribution map, so as to obtain at least one connected domain corresponding to the attribution map.

[0081] In some optional embodiments, the attribution map module further comprises:

[0082] The connected domain pixel point submodule is configured to obtain the number of pixel points contained in each connected domain.

[0083] The third threshold submodule is configured to filter the connected domains based on a third threshold and delete the connected domains with the number of pixel points less than or equal to the third threshold.

[0084] In some optional embodiments, the connected domain analysis module comprises:

[0085] a superimposition submodule, configured to superimpose the test image and the attribution map to obtain a superimposition map;

[0086] an overlap degree submodule, configured to obtain, based on the superimposition map, an overlap degree of each of the connected domains and the foreground region of the test image;

[0087] a first connected domain analysis submodule, configured to determine, based on the overlap degree, a type of each of the connected domains.

[0088] In some optional embodiments, the superimposition submodule comprises:

[0089] a superimposition pixel unit, configured to determine pixels at the same position in the attribution map and the test image;

[0090] a superimposition unit, configured to add pixel values of the pixels in the test image and gray scale values of the pixels at the same position in the attribution map to obtain the superimposition map;

[0091] wherein the superimposition map has the same brightness as the pixels corresponding to the connected domains in the test image, and the brightness of the pixels corresponding to the non-connected domains in the superimposition map is a preset proportion of the brightness of the pixels corresponding to the non-connected domains in the test image.

[0092] In some optional embodiments, the overlap degree submodule comprises:

[0093] a pixel to be analyzed determination unit, configured to obtain, based on the superimposition map, pixels of the foreground region of the test image and pixels of each of the connected domains in the attribution map;

[0094] an intersection-union set pixel unit, configured to determine, based on the pixels of the foreground region and the pixels of each of the connected domains, a number of intersection pixels and a number of union pixels corresponding to each of the connected domains, the intersection pixels being pixels common to the foreground region and each of the connected domains, and the union pixels being all pixels contained in the foreground region and each of the connected domains;

[0095] an overlap degree unit, configured to obtain a ratio of the number of the intersection pixels to the number of the union pixels as the overlap degree of each of the connected domains and the foreground region of the test image.

[0096] In some optional embodiments, the first connected domain analysis submodule comprises:

[0097] a fourth threshold unit configured to filter the overlap degree corresponding to each of the connected domains based on a fourth threshold value;

[0098] a first foreground type unit configured to determine the type of the connected domain as the foreground type if the overlap degree is greater than or equal to the fourth threshold value;

[0099] a first background type unit configured to determine the type of the connected domain as the background type if the overlap degree is less than the fourth threshold value.

[0100] In some optional embodiments, the connected domain analysis module further comprises:

[0101] a similarity sub-module configured to analyze the connected domains in the attribution graph based on the test image to determine a similarity between each of the connected domains and a repeated unit corresponding to the connected domain in the test image;

[0102] a second connected domain analysis sub-module configured to determine the type of each of the connected domains based on the similarity.

[0103] In some optional embodiments, the similarity sub-module comprises:

[0104] a target image region unit configured to determine, based on each of the connected domains, an image region corresponding to each of the connected domains in the test image as a target image region;

[0105] a repetition determination unit configured to determine a target repeated unit corresponding to the target image region in the test image;

[0106] a comparison image region unit configured to determine an image region corresponding to a repeated unit of the same type as the target repeated unit as a comparison image region;

[0107] a similarity unit configured to determine a similarity between the target image region and the comparison image region.

[0108] In some optional embodiments, the second connected domain analysis sub-module comprises:

[0109] a fifth threshold unit configured to filter the similarity corresponding to each of the connected domains based on a fifth threshold value;

[0110] a second background type unit configured to determine the type of the connected domain as the background type if the similarity is greater than or equal to the fifth threshold value;

[0111] a second foreground type unit configured to determine the type of the connected domain as the foreground type if the similarity is less than the fifth threshold value.

[0112] In some optional embodiments, the first initial attribution map submodule comprises:

[0113] a gradient calculation weight unit configured to input the test image into the attribution analysis model to obtain gradient calculation weights of each pixel in the target channel and the target prediction category in the test image;

[0114] an initial attribution map unit configured to linearly weight and fuse the feature layer of the target channel and the gradient calculation weights of each pixel in the target channel and the target prediction category, and regularize through a rectifier function to obtain the initial attribution map corresponding to the test image, the initial attribution map being a feature heat map corresponding to the target prediction category, the target prediction category corresponding to the target feature information.

[0115] In some optional embodiments, the judging module comprises:

[0116] a first judging submodule configured to, in a case where the types of the connected domains satisfy a first condition, determine the defect classification model as a qualified defect classification model and determine the defect classification result as a reliable result, the first condition being that the connected domains contain at least one connected domain of the foreground type and at least one connected domain of the background type;

[0117] a second judging submodule configured to, in a case where the types of the connected domains do not satisfy the first condition, determine the target classification model as an unqualified target classification model and determine the target classification result as an unreliable result.

[0118] A third aspect of the embodiments of the present disclosure provides a computing processing device, comprising:

[0119] a memory in which computer readable code is stored; and

[0120] one or more processors, when the computer readable code is executed by the one or more processors, the computing processing device performs the model evaluation method according to any one of the first aspect.

[0121] A fourth aspect of the embodiments of the present disclosure provides a computer program comprising computer readable code, when the computer readable code is run on a computing processing device, causing the computing processing device to perform the model evaluation method according to any one of the first aspect.

[0122] A fifth aspect of the embodiments of the present disclosure provides a computer readable medium in which the computer program according to the fourth aspect is stored.

[0123] The above description is only a summary of the technical solutions of the present application. In order to make the technical means of the present application more clearly understood and implemented according to the content of the description, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application will be described. BRIEF DESCRIPTION OF DRAWINGS

[0124] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0125] Fig. 1 schematically shows a model evaluation method flowchart for performing the method according to the present disclosure;

[0126] Fig. 2 schematically shows a schematic diagram of a test image for performing the method according to the present disclosure;

[0127] Fig. 3 schematically shows a schematic diagram of an initial attribution graph for performing the method according to the present disclosure;

[0128] Fig. 4 schematically shows a schematic diagram of an attribution graph for performing the method according to the present disclosure;

[0129] Fig. 5 schematically shows a model evaluation device structure schematic diagram for performing the method according to the present disclosure;

[0130] Fig. 6 schematically shows a structure schematic diagram of a computing processing device for performing the method according to the present disclosure. DETAILED DESCRIPTION

[0131] In order to make the purposes, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0132] In industrial defect detection, annotating the specific location of defects requires a lot of manpower and time, so in many cases, only the local image (such as 256x256 resolution) containing the defect is cropped from the large image (such as 1024x1024 resolution) of the product, and then a deep learning algorithm is used to classify the cropped local image to determine the category of the defect, and different ways are taken according to the different categories of defects (such as repair, ignore or destroy). The category of the defect is determined by two factors, one is which process section the defect comes from, and the other is the shape and position of the defect itself.

[0133] The defect classification algorithm based on deep learning has achieved great success in the field of industrial manufacturing, but when the specific position of the above defect scene is not annotated, it is difficult to determine whether the target classification model is based on foreground or background to make a decision due to the unexplainability of deep learning, where the foreground determines the shape and position of the defect itself, and the background determines which process section the defect comes from, and if the defect category output by the target classification model is obtained only by the foreground or only by the background, it will lead to relatively low reliability and universality of the model.

[0134] Therefore, the embodiments of the present disclosure propose a model evaluation method, and FIG. 1 schematically shows a model evaluation method flowchart for executing the method according to the present disclosure, as shown in FIG. 1, the model evaluation method comprises the following steps:

[0135] S101, acquiring a test image.

[0136] The test image comprises a foreground region and a background region.

[0137] S102, inputting the test image into a target classification model to obtain a target classification result.

[0138] The test image is input into a target classification model to obtain a target classification result, and the target classification result at least comprises target feature information and target attribute information.

[0139] S103, inputting the test image into an attribution analysis model to obtain an attribution graph corresponding to the test image.

[0140] S104, taking the test image as a reference, analyzing a connected domain in the attribution graph, and determining a type of the connected domain.

[0141] Taking the test image as a reference, analyzing a connected domain in the attribution graph, and determining a type of the connected domain, the type of the connected domain is a foreground type or a background type.

[0142] S105, determine whether the target classification model is a qualified target classification model according to the type of the connected domain, and determine whether the target classification result is a trusted result according to the type of the connected domain.

[0143] In the embodiments of the present disclosure, the product flows in sequence on the production line of the product in the assembly process through the conveying device, so as to perform corresponding installation process flow at the workstations in different areas, wherein the workstations in different areas of the production line can be in the form of manual assembly to install different product components on the product, or can be in the form of assembly of different product components by automatic mechanical device. Therefore, during the flow of the product on the production line, different defects can be generated at different workstations due to different orders of flow at each workstation, in other words, the defects generated by the product are also associated with the target attribute information. In the embodiments of the present disclosure, the corresponding product images collected at the workstations in different positions of the production line are used for related defect classification, so as to obtain a target classification result, and the target feature information and the target attribute information contained in the product image obtained at each workstation position are represented by the target classification result. For ease of understanding, the model evaluation method provided in the embodiments of the present disclosure is exemplarily described in the defect detection scene, wherein in the defect detection scene, the target feature information is used to represent the morphology and position of the defect generated by the product at the corresponding workstation position; and the target attribute information is used to represent that the defect comes from a specific workstation or a production line process section. It is easy to understand that the model evaluation method provided in the embodiments of the present disclosure is not limited to the defect detection scene, and any other scene can be considered, as long as the same technical idea as the embodiments of the present disclosure is applied.

[0144] In the embodiments of the present disclosure, the products are arranged at intervals on the production line and are transmitted between different workstations, so that the product images collected at different workstations can be all images of the product or part area images of the product. It should be noted that when the images of different workstations of the production line are collected, the specific image collection frequency is determined according to the actual situation, which can be a preset fixed collection frequency or a collection frequency determined according to the flow speed of the production line. The embodiments of the present disclosure do not limit this.

[0145] The test image is an image of an installation process area corresponding to different workstations in the product image, and the relative size relationship between the test image and the product image depends on the area size determined when the corresponding workstation is installed. When the components installed at the workstation occupy part of the product image, the test image is part of the product image. When the components installed at the workstation occupy all or most of the area of the product image, the test image is all of the product image. The area ratio of the specific test image to the product image can be determined in advance according to the detection requirements and the workstation, and the present disclosure does not limit this.

[0146] In the embodiments of the present disclosure, the test image at least includes a foreground region and a background region. The foreground region is a defect region in the test image, and is used to provide image features related to target feature information for the target classification model. Since the test image is used to obtain target classification results of different process sections, the foreground regions of the test images corresponding to different process sections may be different, and the target feature information in the target classification results corresponding thereto is also different due to the difference between the process sections. The background region is an image of other regions outside the defect region in the test image, and contains region information processed by the corresponding process section or workstation. Therefore, the background region is used to provide image features related to target attribute information for the target classification model.

[0147] In the embodiments of the present application, the target classification model is a deep learning model for defect detection based on the test image. The target classification model obtains the target attribute information and the target feature information corresponding to the defect image based on the image features in the input test image, and forms the target classification result corresponding to the test image by combining the target attribute information and the target feature information. However, due to the unexplainability of the target classification model with deep learning, under the premise that the test image is not labeled, it is difficult to determine whether the target classification result is obtained based on the foreground region or the background region of the test image. As described above, the foreground region has a higher correlation with the target feature information, and the target attribute information has a higher correlation with the background region. This makes it necessary to explain the information in the target classification result if the target classification result is obtained based on the foreground region or the background region alone. Therefore, the customer reliance on the target classification model and the target classification result is reduced.

[0148] Therefore, after obtaining the target classification result based on the target classification model, the disclosure embodiment designs an attribution analysis model to perform correlation analysis on the processing basis of the target feature information and the target attribute information in the generated target classification result based on the correlation between the target feature information and the target attribute information and the foreground region and the background region, thereby ensuring the reliability of the target classification model and the generated target classification result, and thereby improving the customer's reliance on the target classification model and the generated target classification result.

[0149] In order for those skilled in the art to better understand the technical solutions of the disclosure, the steps of the model evaluation method provided by the disclosure are described in detail as follows:

[0150] In the specific implementation step S101, first, based on the production line image, the product image collected by the production line at the workstation or process section is obtained, and the product image includes all or part of the region of the product to be detected. Specifically, the frame line detection of the corresponding product is performed on the image collected by the production line at the workstation or process section, the position of the frame line pixel point is the region in which the pixel value between the product region and the conveying device region in the production line image changes obviously, the set of determined frame line pixel points is connected to form an image region containing the product, and the image region containing the product is cropped to obtain the product image.

[0151] In some optional embodiments, since the production line image is an image collected by the image acquisition device at the actual conveying device, the pixel value of the collected production line image and the product image generated based on the production line image may be high or low, and the overall performance is that the brightness of the production line image or the product image is too large, too small or uneven, which causes the test image corresponding to the product image to have a problem of decreased detection accuracy due to brightness problems in subsequent defect detection. Therefore, after obtaining the product image, the product image is preprocessed, and the preprocessing at least includes brightness enhancement and color enhancement. The brightness enhancement is used to adjust the brightness of the product image to a preset brightness range, and the color enhancement is used to adjust the color of the product image to a preset color interval, so that the test image corresponding to the product image has a good processing effect in the subsequent defect detection and attribution analysis process. It should be noted that the brightness enhancement and color enhancement can use common preprocessing methods to improve the image quality of the product image, such as gamma calibration, and the specific preprocessing method is not limited in the disclosure embodiment.

[0152] After obtaining the product image, at each workstation or process section, a part area corresponding to each process section in the product image is obtained as the test image based on the installation process of the process section. It should be noted that the test image can be randomly selected from the test image corresponding to the product image in the data set of the target classification model, or can be manually selected according to actual needs, and the present disclosure does not limit this. The test image at least includes a foreground area and a background area, the foreground area is a defect area image in the part area corresponding to the process section, and the background area is an image of other areas in the part area corresponding to the process section except the defect area image. In the embodiment of the present disclosure, since the background area is an image of other areas in the part area corresponding to the process section except the defect area image, different process sections can reflect the difference in the corresponding part area due to the difference in the installation process, so that the background area of the test image can distinguish the target attribute information of each process section. FIG. 2 schematically shows a schematic diagram of a test image for performing a method according to the present disclosure, as shown in FIG. 2, the test image includes a foreground area corresponding to a defect area, and a background area corresponding to a part area of the process section.

[0153] In the implementation step S102, after obtaining the test image, the test image is input into the defect detection model, the test image is classified for defects, and the corresponding target classification result is output. The defect detection model is a pre-trained deep learning model, which can be a classification model, a segmentation model or other common defect detection model. Specifically, after the test image is input into the defect detection model, the defect detection model extracts first image features in the test image, analyzes the first image features for defects, determines the defect morphology and defect position corresponding to the defect area in the test image as the target feature information; the defect detection model extracts second image features in the test image, analyzes the second image features for images, determines the process section or workstation information corresponding to the test image in the production line as the target attribute information. The target attribute information and the target feature information are combined as the target classification result. For example, the target classification result includes a defect data code (corresponding to the target feature information) of a first data length and a production line data code (corresponding to the target attribute information) of a second data length, and the first data length is the same as the second data length.

[0154] In the embodiments of the present disclosure, the training process of the defect detection model is as follows: a data set is obtained, the data set is split according to a preset proportion to obtain a training set, a validation set and a test set. An initial defect detection model is constructed, for example, the initial defect detection model can be ResNet, ResNeXt, etc. The data of the training set is input into the initial defect detection model, the initial defect detection model is trained based on a deep learning framework, hyperparameters are set, the hyperparameters are optimized based on a preset loss function, and the defect detection model is obtained. When the performance of the defect detection model on the validation set meets a preset validation condition, the defect detection model is output. Optionally, the deep learning framework can be pytorch, TensorFlow, etc.

[0155] In the step S103, the test image is input into the attribution analysis model, and an attribution map corresponding to the test image is output based on the attribution analysis model. The attribution map is a heat map representing the correlation degree of each pixel point and the target feature information (defect category). For example, the higher the correlation degree of the pixel point and the target feature information, the larger the pixel value of the pixel point in the attribution map. Specifically, first, the test image is input into the attribution analysis model, and each pixel point in the test image is analyzed based on the attribution analysis model to determine the correlation degree between each pixel point and the target feature information in the target classification result output by the defect detection model. Based on the correlation degree, the gray value corresponding to each pixel point in the test image is generated. If the correlation degree between the pixel point and the target feature information is larger, the corresponding gray value is larger; if the correlation degree between the pixel point and the target feature information is smaller, the corresponding gray value is smaller. Based on the gray value of each pixel point in the test image obtained based on the attribution analysis model, the corresponding initial attribution map is generated. Optionally, the attribution analysis model can be an existing attribution analysis model, for example, the attribution analysis model can be selected from any one of GradCam, HiResCAM, ScoreCAM, GradCAM++, AblationCAM, XGradCAM, EigenCAM, FullGrad, etc. The specific attribution analysis model is not limited in the present disclosure.

[0156] For example, the attribution analysis model is a GradCam model. After the test image is input into the attribution analysis model, first, the gradient calculation weight of each pixel in the target channel and the target prediction category in the test image is obtained based on the gradient of back propagation, and the gradient calculation weight is obtained according to the following formula:

[0157] wherein c is the target prediction category; y cis the logit value corresponding to the target prediction category c; A is the feature layer output by the last convolutional layer; k is the kth target channel in the feature layer A; i, j are the horizontal and vertical coordinates of the feature layer; is the data of the feature layer A at the kth target channel, at the coordinates i, j; is the gradient calculation weight of each pixel under the target channel k and the target prediction category c; Z is the feature layer size.

[0158] Subsequently, the feature layer A of the target channel k is linearly weighted and fused with the gradient calculation weight of each pixel under the target channel k and the target prediction category c to obtain the initial attribution graph corresponding to the test image, the initial attribution graph being a feature heat map corresponding to the target prediction category, the target prediction category c corresponding to the target feature information, the linearly weighted and fused being performed according to the following formula:

[0159] wherein c is the target prediction category; k is the kth target channel in the feature layer A; is the gradient calculation weight of each pixel under the target channel k and the target prediction category c; A k is the data of the feature layer A at the kth target channel; RELU is the rectifier function; is the initial attribution graph corresponding to the target prediction category c.

[0160] In the embodiments of the present disclosure, each pixel point in the initial attribution graph represents the degree of association between the target feature information (target prediction category), and FIG. 3 schematically shows a schematic diagram of an initial attribution graph for performing the method according to the present disclosure. As shown in FIG. 3, the initial attribution graph can be a gray-scale image, and the gray-scale value of each pixel point in the initial attribution graph represents the degree of association of the target feature information corresponding to the test image. The greater the degree of association with the target feature information, the greater the gray-scale value of the pixel point (the maximum being 255, which is represented by a white pixel point in the initial attribution graph); the smaller the degree of association with the target feature information, the smaller the gray-scale value of the pixel point (the minimum being 0, which is represented by a black pixel point in the initial attribution graph). In an alternative embodiment, in order to facilitate subsequent data processing, the gray-scale values of the pixel points in the initial attribution graph can be normalized so that the gray-scale value of each pixel point is between 0 and 1, 0 corresponding to the minimum gray-scale value (corresponding to black), and 1 corresponding to the maximum gray-scale value (corresponding to white).

[0161] After the initial attribution graph is obtained, the initial attribution graph is binarized to obtain an attribution graph. In an optional embodiment, the initial attribution graph is binarized by using a fixed threshold method. Specifically, a first threshold is first determined, the first threshold being a fixed pixel grayscale threshold, for example, for a normalized pixel grayscale value, the first threshold being a value between 0 and 1 (for example, the first threshold is 0.5); then, based on the first threshold, the grayscale value of each pixel point in the initial attribution graph is classified, and the pixel point with a grayscale value greater than or equal to the first threshold is regarded as a first pixel point, and the pixel point with a grayscale value less than the first threshold is regarded as a second pixel point. The first pixel point has a greater degree of association with the target classification result, and the grayscale value of the first pixel point is adjusted to a first grayscale value, the first grayscale value being a maximum grayscale value (corresponding to the grayscale value of white); the second pixel point has a smaller degree of association with the target classification result, and the grayscale value of the second pixel point is adjusted to a second grayscale value, the second grayscale value being a minimum grayscale value (corresponding to the grayscale value of black), thereby obtaining the attribution graph.

[0162] Exemplarily, FIG. 4 schematically shows a schematic diagram of an attribution graph for performing a method according to the present disclosure. As shown in FIGS. 3 and 4, the initial attribution graph of FIG. 3 includes three grayscale regions A1, A2 and A3, wherein the grayscale values of the corresponding grayscale regions of A1, A2 and A3 increase in turn, and the grayscale value of the pixel point of the A3 region is 1. Compared with the initial attribution graph of FIG. 3, the pixel points with a grayscale value less than the first threshold are adjusted to black pixel points with a grayscale value of 0 after binarization in FIG. 4, and the A1 region and the A2 region are both adjusted to black pixel points; the pixel points of the A3 region have a grayscale value greater than the first threshold, and the A3 region is adjusted to white pixel points, thereby obtaining the attribution graph shown in FIG. 4. The attribution graph shown in FIG. 4 only retains the A3 region after binarization.

[0163] In some optional embodiments, after obtaining the initial attribution map, the binary processing can be performed by a fixed proportion method to obtain the attribution map. Specifically, first, the gray values of each pixel point in the initial attribution map are sorted. The sorting manner can be from small to large or from large to small, which is not limited in the present disclosure. Then, a second threshold is determined. The second threshold is a fixed proportion threshold. For example, for the normalized pixel gray values, the second threshold is a fixed proportion of all pixel points in the initial attribution map (for example, the top 10% of pixel points with the largest gray values after sorting from large to small). Based on the second threshold, the pixel points are classified to obtain a set of pixel points corresponding to the proportion of the second threshold in the sorted pixel points in the initial attribution map, which is taken as the third pixel point, i.e., the proportion of the number of the third pixel point is equal to the second threshold. The pixel points other than the third pixel point are taken as the fourth pixel point, wherein the gray value of the third pixel point is greater than that of the fourth pixel point. Finally, the gray value of the third pixel point is adjusted to the first gray value, and the gray value of the fourth pixel point is adjusted to the second gray value to obtain the attribution map.

[0164] It should be noted that the binary processing of the initial attribution map can also use other existing binary processing methods. The specific binary processing method can be determined according to the actual situation, which is not limited in the present disclosure.

[0165] In the implementation step S104, after obtaining the binary attribution map, the connected domain in the attribution map is obtained before analyzing the attribution map. The connected domain is a region composed of connectable pixel points with the same gray value. Specifically, first, an initial pixel point is obtained, and a connected domain corresponding to the initial pixel point is created. The initial pixel point is any pixel point in the attribution map. Then, all neighborhood pixel points of the initial pixel point are detected in the attribution map. The neighborhood pixel point refers to other connectable pixel points directly adjacent to the initial pixel point. The neighborhood pixel points with the same gray value as the initial pixel point are added to the connected domain corresponding to the initial pixel point, i.e., all other connectable pixel points with the same gray value as the initial pixel point are added to the connected domain corresponding to the initial pixel point. The above steps are repeated until the initial pixel point traverses all pixel points of the attribution map to obtain at least one connected domain corresponding to the attribution map. Taking FIG. 4 as an example, all pixel points in the A3 region in FIG. 4 have the same gray value, which is 1. At this time, all pixel points in the A3 region are divided into the same connected domain.

[0166] In an alternative embodiment, after obtaining the at least one connected domain corresponding to the attribution graph, since there is no limitation on the number of pixels contained in each connected domain, the regions containing fewer pixels (such as regions containing only one or a few pixels) in all the obtained connected domains are also recognized as connected domains, which will make the corresponding regions of the connected domains in the attribution graph or the test image scattered in various positions, resulting in a decrease in the accuracy of model evaluation based on the connected domains. In other words, since the number of pixels of the connected domains is not limited, the connected domains containing fewer pixels formed by noise are also considered as connected domains for evaluation, so that the connected domains contain both connected domains located in the foreground region and connected domains located in the background region, thereby failing to effectively evaluate the target classification model based on the difference in the distribution of the connected domains in the foreground region and the background region. Therefore, in the embodiments of the present disclosure, after obtaining the connected domains, the connected domains need to be screened based on the number of pixels to remove noise-like connected domains containing a small number of pixels and only keep the effective connected domains. Specifically, first, the number of pixels contained in each connected domain is obtained; then a third threshold value is determined, the third threshold value being a fixed threshold value representing the area of the connected domain (for example, the third threshold value is a pixel number threshold value contained in the connected domain, such as the third threshold value being 20 pixels), and the connected domains are screened based on the third threshold value to delete the connected domains with a number of pixels less than or equal to the third threshold value.

[0167] In the embodiments of the present disclosure, after determining the connected domains in the attribution graph, the target classification model is evaluated by judging the types of the connected domains. Since the target classification result generated by the target classification model contains target attribute information and target feature information, the target attribute information should be obtained by the target classification model based on the features of the background region, and the target feature information should be obtained by the target classification model based on the features of the foreground region, so the reliability of the target classification model can be determined based on the types of the connected domains in the attribution graph corresponding to the test image. If the connected domains of the attribution graph corresponding to the test image contain both connected domains belonging to the foreground type and connected domains belonging to the background type, it means that the target classification result generated by the target classification model based on the test image is obtained by simultaneously classifying the foreground region and the background region, and the target feature information contained therein can be considered to be obtained based on the foreground region, and the target attribute information can be considered to be obtained based on the background region, thereby ensuring the reliability of the target classification model and the corresponding target classification result.

[0168] In the embodiments of the present disclosure, after the at least one connected domain is obtained, the test image is divided according to the positions of the connected domains, and the regions of the test image corresponding to the connected domains are analyzed. In some optional embodiments, since the attribution graph has the same size as the test image, each pixel point in the test image has a one-to-one corresponding pixel point in the attribution graph. Based on the positions of the pixel points in the attribution graph and the test image, the pixel points in the test image corresponding to the pixel points in the attribution graph are determined, and the pixel values of the pixel points in the test image are added to the gray values of the pixel points in the attribution graph corresponding to the pixel points in the test image, so that the brightness of the pixel points in the test image corresponding to the pixel points in the attribution graph remains unchanged, and the brightness of the pixel points in the test image corresponding to the pixel points in the attribution graph decreases by a preset proportion (such as 50%), to obtain the superimposed graph.

[0169] Subsequently, for the superimposed graph obtained based on each connected domain, the overlap degree of each connected domain and the foreground region of the test image is obtained. Specifically, based on the superimposed graph, the pixel points of the foreground region of the test image and the pixel points of each connected domain in the attribution graph are obtained; based on the pixel points of the foreground region and the pixel points of each connected domain, the IoU value of each connected domain and the foreground region is determined as the overlap degree. The IoU value is determined in the following manner: the pixel points common to the foreground region and each connected domain are determined as intersection pixel points, and then the number of intersection pixel points corresponding to each connected domain is determined; all pixel points contained in the foreground region and each connected domain are determined as union pixel points, and then the number of union pixel points is determined; the ratio of the number of intersection pixel points to the number of union pixel points is obtained as the IoU value of each connected domain and the foreground region of the test image, as the overlap degree corresponding to each connected domain.

[0170] Finally, based on the overlap degree, the type of each connected domain is determined. Specifically, a fourth threshold is first determined, which is an overlap degree threshold, for example, the fourth threshold is 0.5; based on the fourth threshold, the overlap degree corresponding to each connected domain is screened, in the case that the overlap degree is greater than or equal to the fourth threshold, it is considered that the connected domain has a larger overlap degree with the foreground region of the test image and a smaller overlap degree with the background region of the test image, and the type of the connected domain is determined as the foreground type; in the case that the overlap degree is less than the fourth threshold, it is considered that the connected domain has a smaller overlap degree with the foreground region of the test image and a larger overlap degree with the background region of the test image, and the type of the connected domain is determined as the background type. It should be noted that the type of the connected domain can only be one of the background type and the foreground type, and cannot be both the foreground type and the background type at the same time, so as to prevent the accuracy of connected domain analysis from being reduced due to the fact that a single connected domain meets two types at the same time.

[0171] In an optional embodiment, as shown in FIG. 2, in the case that the installation component region corresponding to the test image includes a plurality of repeating units arranged periodically, the test image is composed of repeating units arranged in the same mode, for example, the test image is an array image of sub-pixels, and the component of the test image includes a first sub-pixel 1, a second sub-pixel 2 and a third sub-pixel 3 arranged periodically. The type of the connected domain can be determined by the similarity between the connected domain and other repeating units of the same type. If the repeating unit corresponding to the connected domain has a defect (i.e., contains a foreground region), the similarity between the repeating unit corresponding to the connected domain and other repeating units of the same type will decrease due to the existence of the foreground region; on the contrary, if the repeating unit corresponding to the connected domain has no defect (i.e., does not contain a foreground region), the similarity between the repeating unit corresponding to the connected domain and other repeating units of the same type is higher. Therefore, the embodiment of the present disclosure can analyze the connected domains in the attribution graph based on the test image, determine the similarity between each connected domain and the repeating unit corresponding to the connected domain in the test image, and determine the type of the connected domain by the difference in similarity between the repeating unit corresponding to the connected domain and other repeating units of the same type.

[0172] Specifically, first, based on each of the connected domains, an image region corresponding to each of the connected domains in the test image is determined as a target image region; in the test image, a target repeating unit corresponding to the target image region and a periodic arrangement manner of the target repeating unit in the test image are determined; based on the periodic arrangement manner, an image region corresponding to a repeating unit of the same type closest to the target repeating unit is determined as the comparison image region, and a similarity between the target image region and the comparison image region is determined as a similarity corresponding to each of the connected domains. Finally, a fifth threshold value is determined, the fifth threshold value is the similarity threshold value, and based on the fifth threshold value, the similarity corresponding to each of the connected domains is screened: in the case where the similarity is greater than or equal to the fifth threshold value, it is considered that the similarity of the target image region and the comparison image region corresponding to the connected domain is high, at this time, it is considered that there is no defect region in the target image region corresponding to the connected domain, and the type of the connected domain is determined as the background type; in the case where the similarity is less than the fifth threshold value, it is considered that the similarity of the target image region and the comparison image region corresponding to the connected domain is low, at this time, it is considered that the same repeating unit appears lower similarity because of the existence of the defect region in the target image region corresponding to the connected domain, and then the type of the connected domain is determined as the foreground type.

[0173] Taking FIG. 2 as an example, the test image in FIG. 2 contains three periodically arranged first sub-pixels 1, second sub-pixels 2 and third sub-pixels 3, and the defect region is located in a repeating unit of the first sub-pixel 1. Based on the only one connected domain A3 in the attribution graph corresponding to FIG. 4, it can be determined that the target image region corresponding to the connected domain A3 in the test image is B1, and the repeating unit corresponding to B1 is the first sub-pixel 1. At this time, based on the periodic arrangement manner of the first sub-pixel 1 in the test image, the comparison image region corresponding to the target image region A3 in FIG. 2 is the first sub-pixel 1 (B0 in FIG. 2) of a preset periodic length on the left side of the target image region, and the similarity between the comparison image region B0 and the target image region B1 is calculated. Since the target image region B1 contains the defect region, the similarity between the comparison image region B0 and the target image region B1 is low, and thus the type of the connected domain A3 is determined as the foreground type.

[0174] By determining the type of each of the connected domains in the attribution graph corresponding to the test image, it can be determined whether the target classification result generated by the test image based on the target classification model is generated based on the connected domains of the foreground type and the background type at the same time, so as to determine the reliability of the target classification model and the target classification result by the number of the connected domains belonging to the foreground type and the background type.

[0175] In the step S105, it is determined whether the target classification model is a qualified target classification model according to the type of the connected domain, and it is determined whether the target classification result is a reliable result according to the type of the connected domain. Since the target feature information has a high reliability when obtained based on the foreground region, and the target attribute information has a high reliability when obtained based on the background region, the first condition is set in advance, that is, the connected domain contains at least one connected domain of the foreground type, and the connected domain contains at least one connected domain of the background type. When the type of the connected domain satisfies the first condition, it can be determined that the target attribute information in the target classification result is obtained based on the background region (the connected domain corresponding to the background type), the target feature information in the target classification result is obtained based on the foreground region (the connected domain corresponding to the foreground type), thereby determining that the target classification model is a qualified target classification model, and the target classification result is a reliable result.

[0176] In an optional embodiment, when the type of the connected domain does not satisfy the first condition, it is not determined that the target attribute information in the target classification result is obtained based on the background region (the connected domain corresponding to the background type), and the target feature information in the target classification result is obtained based on the foreground region (the connected domain corresponding to the foreground type). At this time, the target classification model is determined to be an unqualified target classification model, and the target classification result is determined to be an unreliable result.

[0177] It should be noted that the first condition is the lowest standard for determining the reliability of the target classification model and the target classification result based on the type of the connected domain. In the actual model evaluation process based on the type of the connected domain, a more stringent judgment condition (for example, the connected domain contains at least two connected domains of the foreground type, and the connected domain contains at least one connected domain of the background type) can be formulated according to the actual scene, and the present disclosure does not limit this.

[0178] The embodiment of the present disclosure provides a model evaluation method, which comprises the following steps: acquiring a test image, wherein the test image comprises a foreground region and a background region; inputting the test image into a target classification model to obtain a target classification result, wherein the target classification result at least comprises target feature information and target attribute information; inputting the test image into an attribution analysis model to acquire an attribution graph corresponding to the test image; analyzing a connected domain in the attribution graph based on the test image to determine a type of the connected domain, wherein the type of the connected domain is a foreground type or a background type; determining whether the target classification model is a qualified target classification model according to the type of the connected domain, and determining whether the target classification result is a credible result according to the type of the connected domain. According to the embodiment of the present disclosure, the attribution graph corresponding to the test image is acquired, and whether the data support of the target classification model in defect classification simultaneously comprises the foreground region and the background region is determined based on the type of the connected domain in the attribution graph. On the premise that the foreground region and the background region are simultaneously included, the target classification model and the target classification result are determined as the credible result, so that the credibility of the model is improved.

[0179] Based on the same inventive concept, the embodiment of the present disclosure discloses a model evaluation device. FIG. 5 schematically shows a structure diagram of a model evaluation device for performing the method according to the present disclosure. As shown in FIG. 5, the model evaluation device comprises:

[0180] An image acquisition module is configured to acquire a test image, wherein the test image comprises a foreground region and a background region.

[0181] A defect classification module is configured to input the test image into a target classification model to obtain a target classification result, wherein the target classification result at least comprises target feature information and target attribute information.

[0182] An attribution graph module is configured to input the test image into an attribution analysis model to acquire an attribution graph corresponding to the test image.

[0183] A connected domain analysis module is configured to analyze a connected domain in the attribution graph based on the test image to determine a type of the connected domain, wherein the type of the connected domain is a foreground type or a background type.

[0184] A judgment module is configured to determine whether the target classification model is a qualified target classification model according to the type of the connected domain, and determine whether the target classification result is a credible result according to the type of the connected domain.

[0185] In some optional embodiments, the attribution graph module comprises:

[0186] The first initial attribution map submodule is configured to input the test image into the attribution analysis model to obtain an initial attribution map, wherein a gray value of each pixel point in the initial attribution map represents an association degree between each pixel point and the target feature information.

[0187] The first threshold submodule is configured to classify the gray values of the pixel points in the initial attribution map based on a first threshold, and regard a pixel point with a gray value greater than or equal to the first threshold as a first pixel point, and regard a pixel point with a gray value less than the first threshold as a second pixel point.

[0188] The first attribution map submodule is configured to adjust the gray value of the first pixel point to a first gray value, and adjust the gray value of the second pixel point to a second gray value, to obtain the attribution map.

[0189] In some optional embodiments, the attribution map module comprises:

[0190] The second initial attribution map submodule is configured to input the test image into the attribution analysis model to obtain an initial attribution map, wherein a gray value of each pixel point in the initial attribution map represents an association degree between each pixel point and the target feature information.

[0191] The pixel sorting submodule is configured to sort the gray values of the pixel points in the initial attribution map.

[0192] The second threshold submodule is configured to classify the pixel points based on a second threshold to obtain third pixel points and fourth pixel points, wherein a gray value of the third pixel point is greater than a gray value of the fourth pixel point, and a quantity ratio of the third pixel points is equal to the second threshold.

[0193] The second attribution map submodule is configured to adjust the gray value of the third pixel point to a first gray value, and adjust the gray value of the fourth pixel point to a second gray value, to obtain the attribution map.

[0194] In some optional embodiments, the attribution map module further comprises:

[0195] The connected domain creation submodule is configured to obtain an initial pixel point, and create a connected domain corresponding to the initial pixel point, wherein the initial pixel point is any pixel point in the attribution map.

[0196] The neighborhood detection submodule is configured to detect all neighborhood pixel points of the initial pixel point in the attribution map one by one, and add a neighborhood pixel point with a same gray value as the initial pixel point to the connected domain corresponding to the initial pixel point.

[0197] The connected domain submodule is configured to repeat the above steps until the initial pixel point traverses all pixel points of the attribution graph, and at least one connected domain corresponding to the attribution graph is obtained.

[0198] In some optional embodiments, the attribution graph module further includes:

[0199] The connected domain pixel point submodule is configured to obtain the number of pixel points contained in each connected domain.

[0200] The third threshold submodule is configured to filter the connected domains based on a third threshold, and delete a connected domain whose number of pixel points is less than or equal to the third threshold.

[0201] In some optional embodiments, the connected domain analysis module includes:

[0202] The superposition submodule is configured to superimpose the test image and the attribution graph to obtain a superposition graph.

[0203] The overlap degree submodule is configured to obtain, based on the superposition graph, an overlap degree of each connected domain and a foreground region of the test image.

[0204] The first connected domain analysis submodule is configured to determine the type of each connected domain based on the overlap degree.

[0205] In some optional embodiments, the superposition submodule includes:

[0206] The superposition pixel unit is configured to determine pixel points at the same position in the attribution graph and the test image.

[0207] The superposition unit is configured to add pixel values of pixel points in the test image and gray values of pixel points at the same position in the attribution graph to obtain the superposition graph.

[0208] In the superposition graph, the brightness of a pixel point corresponding to a non-connected domain is a preset proportion of the brightness of a pixel point in the test image corresponding to the non-connected domain.

[0209] In some optional embodiments, the overlap degree submodule includes:

[0210] The pixel to be analyzed determination unit is configured to obtain, based on the superposition graph, pixel points of the foreground region of the test image and pixel points of each connected domain in the attribution graph.

[0211] An intersection pixel unit is configured to determine a number of intersection pixels and a number of union pixels corresponding to each of the connected domains based on the pixels of the foreground region and the pixels of each of the connected domains, wherein the intersection pixels are common pixels of the foreground region and each of the connected domains, and the union pixels are all pixels contained in the foreground region and each of the connected domains;

[0212] An overlap degree unit is configured to obtain a ratio of the number of the intersection pixels to the number of the union pixels as an overlap degree of each of the connected domains and the foreground region of the test image.

[0213] In some optional embodiments, the first connected domain analysis submodule includes:

[0214] A fourth threshold unit is configured to filter the overlap degree corresponding to each of the connected domains based on a fourth threshold;

[0215] A first foreground type unit is configured to determine a type of the connected domain as the foreground type in a case where the overlap degree is greater than or equal to the fourth threshold;

[0216] A first background type unit is configured to determine the type of the connected domain as the background type in a case where the overlap degree is less than the fourth threshold.

[0217] In some optional embodiments, the connected domain analysis module further includes:

[0218] A similarity submodule is configured to analyze the connected domains in the attribution graph based on the test image to determine a similarity between each of the connected domains and a corresponding repeated unit of the connected domain in the test image;

[0219] A second connected domain analysis submodule is configured to determine a type of each of the connected domains based on the similarity.

[0220] In some optional embodiments, the similarity submodule includes:

[0221] A target image region unit is configured to determine, based on each of the connected domains, an image region corresponding to each of the connected domains in the test image as a target image region;

[0222] A repeated determination unit is configured to determine a target repeated unit corresponding to the target image region in the test image;

[0223] A comparison image region unit is configured to determine an image region corresponding to a repeated unit of the same type as the target repeated unit as a comparison image region;

[0224] A similarity unit is configured to determine a similarity between the target image region and the contrast image region.

[0225] In some optional embodiments, the second connected component analysis submodule comprises:

[0226] A fifth threshold unit is configured to filter the similarity corresponding to each connected component based on a fifth threshold.

[0227] A second background type unit is configured to determine the type of the connected component as the background type when the similarity is greater than or equal to the fifth threshold.

[0228] A second foreground type unit is configured to determine the type of the connected component as the foreground type when the similarity is less than the fifth threshold.

[0229] In some optional embodiments, the first initial attribution map submodule comprises:

[0230] A gradient calculation weight unit is configured to input the test image into the attribution analysis model to obtain a gradient calculation weight of each pixel in the test image under a target channel and a target prediction category.

[0231] An initial attribution map unit is configured to linearly weight and fuse a feature layer of the target channel and the gradient calculation weight of each pixel under the target channel and the target prediction category, and regularize through a rectifier function to obtain the initial attribution map corresponding to the test image, the initial attribution map being a feature heat map corresponding to the target prediction category, the target prediction category corresponding to the target feature information.

[0232] In some optional embodiments, the judging module comprises:

[0233] A first judging submodule is configured to determine the defect classification model as a qualified defect classification model and determine the defect classification result as a reliable result when the type of the connected component satisfies a first condition, the first condition being that the connected component contains at least one connected component of the foreground type and contains at least one connected component of the background type.

[0234] A second judging submodule is configured to determine the target classification model as an unqualified target classification model and determine the target classification result as an unreliable result when the type of the connected component does not satisfy the first condition.

[0235] The apparatus embodiments described above are only schematic and many variations can be made to the described embodiments, elements and details. In the embodiment, each unit illustrated in the various embodiments can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. Those skilled in the art can understand and implement without creative labor.

[0236] Various component embodiments of the present disclosure can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the computing processing device according to the embodiments of the present disclosure. The present disclosure can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for performing part or all of the methods described herein. Such a program implementing the present disclosure can be stored on a computer readable medium or can have the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or in any other form.

[0237] For example, Figure 6 schematically shows a structural diagram of a computing processing device for performing the methods according to the present disclosure, as shown in Figure 6, the computing processing device 100 traditionally comprises a processor 120 and a computer program product or computer readable medium in the form of a memory 110. The memory 110 can be an electronic memory such as a flash memory, EEPROM (electrically erasable programmable read-only memory), EPROM, hard disk or ROM. The memory 110 has storage space for program code for performing any of the method steps in the above methods. For example, the storage space for program code can include individual program codes for implementing various steps in the above methods, respectively. These program codes can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, compact discs (CDs), memory cards or floppy disks. Such computer program products are usually portable or fixed storage units. The storage unit can have storage segments, storage spaces, etc. arranged similarly to the memory 110 in the computing processing device of Figure 6. The program code can be compressed, for example, in an appropriate form. In general, the storage unit includes computer readable code, i.e., code that can be read by a processor such as 120, which, when run by the computing processing device, causes the computing processing device to perform the various steps in the methods described above.

[0238] As used in this description, the terms "one embodiment", "an embodiment” or "one or more embodiments” mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. The appearances of the phrase "in one embodiment” in various places in this description are not necessarily all referring to the same embodiment.

[0239] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the disclosure can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0240] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising” does not exclude the presence of elements or steps other than those listed in a claim. The word "a” or "an” preceding an element does not exclude the presence of a plurality of such elements. The disclosure can be implemented by means of both hardware and software, and any combinations of these. In a unitary or integrated claim, items such as multiple parts, clauses and the like are implemented by one and the same item or by one item carried out by one or more physical devices. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. The sole use of the word "comprise” does not exclude the presence of other elements or steps than those recited in a claim. The word "example” does not mean or represent that the following description is an example only, a number of examples are described herein, by way of example only, any such number of examples are therefore optional.

[0241] It should also be noted that the above-mentioned embodiments illustrate rather than limit the application, since the application is amenable to many changes without departing from the scope of the ensuing claims.

Claims

1. A model evaluation method, wherein, The method comprises: acquiring a test image, the test image comprising a foreground region and a background region; inputting the test image into a target classification model to obtain a target classification result, the target classification result comprising at least target feature information and target attribute information; inputting the test image into an attribution analysis model to obtain an attribution graph corresponding to the test image; analyzing a connected domain in the attribution graph based on the test image to determine a type of the connected domain, the type of the connected domain being a foreground type or a background type; determining whether the target classification model is a qualified target classification model according to the type of the connected domain, and determining whether the target classification result is a credible result according to the type of the connected domain.

2. The model evaluation method of claim 1, wherein, The step of inputting the test image into the attribution analysis model to obtain an attribution graph corresponding to the test image comprises: inputting the test image into the attribution analysis model to obtain an initial attribution graph, a gray value of each pixel point in the initial attribution graph representing an association degree of each pixel point with the target feature information; classifying the gray value of each pixel point in the initial attribution graph based on a first threshold, taking a pixel point with a gray value greater than or equal to the first threshold as a first pixel point, and taking a pixel point with a gray value less than the first threshold as a second pixel point; adjusting the gray value of the first pixel point to a first gray value and adjusting the gray value of the second pixel point to a second gray value to obtain the attribution graph.

3. The model evaluation method of claim 1, wherein, The step of inputting the test image into the attribution analysis model to obtain an attribution graph corresponding to the test image comprises: inputting the test image into the attribution analysis model to obtain an initial attribution graph, a gray value of each pixel point in the initial attribution graph representing an association degree of each pixel point with the target feature information; sorting the gray value of each pixel point in the initial attribution graph; classifying the pixel points based on a second threshold to obtain third pixel points and fourth pixel points, wherein the gray value of the third pixel points is greater than the gray value of the fourth pixel points, and the number ratio of the third pixel points is equal to the second threshold; adjusting the gray value of the third pixel points to a first gray value and adjusting the gray value of the fourth pixel points to a second gray value to obtain the attribution graph.

4. The model evaluation method of claim 1, wherein, After obtaining the attribution graph corresponding to the test image, the method further comprises: acquiring an initial pixel point and creating a connected domain corresponding to the initial pixel point, the initial pixel point being any one pixel point in the attribution graph; detecting all neighborhood pixel points of the initial pixel point one by one in the attribution graph, and adding a neighborhood pixel point with the same gray value as the initial pixel point to the connected domain corresponding to the initial pixel point; repeating the above steps until the initial pixel point traverses all pixel points of the attribution graph to obtain at least one connected domain corresponding to the attribution graph.

5. The model evaluation method of claim 1, wherein, After obtaining the at least one connected domain corresponding to the attribution graph, the method further comprises: acquiring a number of pixel points contained in each connected domain; The connected domains are screened based on a third threshold value, and connected domains in which the number of pixels is less than or equal to the third threshold value are deleted.

6. The model evaluation method of claim 1, wherein, The step of analyzing the connected domain in the attribution graph based on the test image to determine the type of the connected domain includes: Overlaying the test image with the attribution map to obtain an overlay map; Based on the overlay map, obtaining the degree of overlap between each of the connected domains and the foreground area of ​​the test image; Based on the overlap degree, the type of each connected component is determined.

7. The model evaluation method of claim 6, wherein, The step of superimposing the test image and the attribution map to obtain a superimposed map includes: Determining pixels at the same position in the attribution map and the test image; Adding the pixel value of a pixel point in the test image to the grayscale value of a pixel point at the same position in the attribution map to obtain the overlay map; The brightness of the pixels corresponding to the connected domain in the overlay image is the same as that in the test image; the brightness of the pixels corresponding to the non-connected domain in the overlay image is a preset ratio of the brightness of the pixels corresponding to the non-connected domain in the test image.

8. The model evaluation method of claim 6, wherein, The obtaining, based on the overlay map, a degree of overlap between each of the connected domains and the foreground area of ​​the test image includes: Based on the overlay map, obtaining pixel points of the foreground area of ​​the test image and pixel points of each connected domain in the attribution map; Based on the pixels of the foreground area and the pixels of each connected domain, determining the number of intersection pixels and the number of union pixels corresponding to each connected domain, wherein the intersection pixels are the pixels common to the foreground area and each connected domain, and the union pixels are all the pixels contained in the foreground area and each connected domain; A ratio of the number of the intersection pixels to the number of the union pixels is obtained as the degree of overlap between each of the connected domains and the foreground area of ​​the test image.

9. The model evaluation method of claim 6, wherein, The determining the type of each connected domain based on the overlap degree includes: Based on a fourth threshold, screening the overlap degree corresponding to each of the connected domains; When the degree of overlap is greater than or equal to the fourth threshold, determining the type of the connected domain as the foreground type; When the degree of overlap is less than the fourth threshold, the type of the connected domain is determined to be the background type.

10. The model evaluation method of claim 1, wherein, The test image includes a plurality of periodically arranged repeating units. The connected domain in the attribution graph is analyzed based on the test image to determine the type of the connected domain, including: Based on the test image, analyzing the connected domains in the attribution graph to determine the similarity between each of the connected domains and a corresponding repeating unit in the test image; Based on the similarity, the type of each connected component is determined.

11. The model evaluation method of claim 10, wherein, The step of analyzing the connected domains in the attribution graph based on the test image to determine the similarity between each connected domain and a corresponding repeating unit of the connected domain in the test image includes: Based on each of the connected domains, determining an image region in the test image corresponding to each of the connected domains as a target image region; In the test image, a target repeating unit corresponding to the target image region is determined; An image region corresponding to a repeating unit of the same type as the target repeating unit is determined as a contrast image region; A similarity between the target image region and the contrast image region is determined.

12. The model evaluation method of claim 10, wherein, The type of each connected domain is determined based on the similarity, including: The similarity corresponding to each connected domain is filtered based on a fifth threshold value; In a case where the similarity is greater than or equal to the fifth threshold value, the type of the connected domain is determined as the background type; In a case where the similarity is less than the fifth threshold value, the type of the connected domain is determined as the foreground type.

13. The model evaluation method of claim 2, wherein, The test image is input into an attribution analysis model to obtain an initial attribution graph, including: The test image is input into the attribution analysis model to obtain gradient calculation weights of each pixel under a target channel and a target prediction category; A feature layer of the target channel is linearly weighted and fused with the gradient calculation weights of each pixel under the target channel and the target prediction category, and is regularized through a rectifier function to obtain the initial attribution graph corresponding to the test image, the initial attribution graph being a feature heat map corresponding to the target prediction category, the target prediction category corresponding to the target feature information.

14. The model evaluation method according to any one of claims 1 to 13, wherein, Whether the target classification model is a qualified target classification model and whether the target classification result is a trusted result are determined according to the type of the connected domain, including: In a case where the type of the connected domain meets a first condition, the defect classification model is determined as a qualified defect classification model, and the defect classification result is determined as a trusted result, the first condition being that the connected domain contains at least one foreground type connected domain and at least one background type connected domain; In a case where the type of the connected domain does not meet the first condition, the target classification model is determined as an unqualified target classification model, and the target classification result is determined as an untrusted result.

15. A model evaluation apparatus, wherein, The device includes: An image acquisition module configured to acquire a test image, the test image including a foreground region and a background region; A defect classification module configured to input the test image into a target classification model to obtain a target classification result, the target classification result including at least target feature information and target attribute information; An attribution graph module configured to input the test image into an attribution analysis model to obtain an attribution graph corresponding to the test image; A connected domain analysis module configured to analyze a connected domain in the attribution graph based on the test image to determine a type of the connected domain, the type of the connected domain being a foreground type or a background type; A judgment module configured to determine whether the target classification model is a qualified target classification model and whether the target classification result is a trusted result according to the type of the connected domain.

16. A computing processing device, wherein, including: a memory having computer readable code stored therein; and and one or more processors, the computing processing device, when the computer readable code is executed by the one or more processors, performs the model evaluation method as claimed in any of claims 1-14.

17. A computer program, wherein, including computer readable code which, when run on a computing processing device, causes the computing processing device to perform the model evaluation method according to any of claims 1-14.

18. A computer readable medium, wherein, a computer program as claimed in claim 17. a computer program as claimed in claim 17.

Images