Image defect detection model prediction result manual review method, device and equipment

Abstract

The application relates to the technical field of defect review, and discloses a method, device and equipment for manually reviewing prediction results of an image defect detection model, which comprises the following steps: acquiring a defect frame output by an image defect detection model and a feature vector and a confidence degree corresponding to the defect frame; adjusting the confidence degree of a defect frame which has not been manually reviewed according to the similarity of the feature vector of a defect frame which has been manually reviewed and the feature vector of a defect frame which has not been manually reviewed; determining the review order of the defect frame which has not been manually reviewed according to the adjusted confidence degree; and adjusting the confidence degree of the defect frame which has not been manually reviewed in real time according to the completed manual review result by calculating the similarity of the feature vector of the defect frame, so that the review order is dynamically adjusted, and the workload required for manual review is reduced under the premise of ensuring the defect detection rate.

Description

Technical Field

[0001] This invention relates to the field of defect review technology, specifically to a method, apparatus, and equipment for manual review of image defect detection model prediction results. Background Technology

[0002] Currently, artificial intelligence models are widely used in industrial inspection. The industry uses drones, surveillance devices, or inspection vehicles to take standardized photos of power transmission towers, railway overhead contact lines, station equipment, and passageway environments. Image defect detection models are then used to screen for potential problems in the photos (including equipment defects, scene hazards, and personnel violations, hereinafter referred to as defects). However, due to the complexity of real-world scenarios and the varied forms of defects, image defect detection models are not entirely accurate. After identifying potential defects using these models, manual review or re-examination is necessary. Different review strategies yield significantly different defect detection rates and require varying amounts of manual review work.

[0003] In the existing review strategy, the image defect detection model prediction results can be reviewed from high to low confidence. The differences between different defect categories in the model training set and the actual application scenario can be considered to adjust the posterior probability of different categories with true prediction results. This reduces the workload of review while ensuring the defect detection rate.

[0004] However, in practical applications, the false positives and false negatives of defects predicted by the model in the images obtained from a single inspection are similar. Existing technical solutions only consider the confidence deviation of different categories in actual application scenarios and cannot dynamically adjust the review order based on the review results, resulting in poor review effectiveness. Summary of the Invention

[0005] In view of this, the present invention provides a method, apparatus, equipment and medium for manual review of image defect detection model prediction results, in order to solve the problem of poor review results caused by the inability to dynamically adjust the review order in the prior art.

[0006] In a first aspect, the present invention provides a method for manual review of prediction results of an image defect detection model, comprising: acquiring defect boxes output by the image defect detection model and the feature vectors and confidence scores corresponding to the defect boxes; adjusting the confidence scores of defect boxes that have not been manually reviewed based on the similarity between the feature vectors of manually reviewed defect boxes and feature vectors of unreviewed defect boxes; and determining the review order of unreviewed defect boxes based on the adjusted confidence scores.

[0007] Optionally, obtaining the defect box output by the image defect detection model and the corresponding feature vector and confidence level of the defect box includes: obtaining the defect box and the defect category, confidence level and defect features corresponding to the defect box through the image defect detection model; and reducing and calibrating the defect features to obtain a feature vector of a preset dimension.

[0008] Optionally, the defect features are obtained by extracting features of the location of the defect box from the feature extraction network of the image defect detection model.

[0009] Optionally, adjusting the confidence level of defective boxes that have not undergone manual review based on the similarity between the feature vectors of defective boxes that have completed manual review and those that have not undergone manual review includes: constructing a confidence level set for defective boxes of each defect category, wherein the confidence level set includes several defective boxes with the highest confidence level in the same defect category or several defective boxes corresponding to feature vectors that are closest to the feature vectors of defective boxes that have completed manual review in the same defect category; determining a confidence level adjustment value based on the review results of defective boxes that have completed manual review; searching for several target defective boxes that have not undergone manual review and are located in different scenes from the defective boxes that have completed manual review in the confidence level set corresponding to the defect category to which the defective boxes that have completed manual review belong; and adjusting the confidence level of the target defective boxes based on the similarity between the feature vectors of the defective boxes that have completed manual review and the feature vectors of the target defective boxes, as well as the confidence level adjustment value.

[0010] Optionally, the method for adjusting the confidence level of the target defect box based on the similarity between the feature vector of the defect box that has undergone manual review and the feature vector of the target defect box, as well as the confidence adjustment value, is as follows:

[0011] S jh =S j +aA×exp(-w×D(vi, vj) 2 / 2)

[0012] In the formula, S jh S is the adjusted confidence level. j The confidence level before adjustment is given, where a and w are preset coefficients, and D(v) i ,v j ) represents the similarity between the feature vector of the defect box that has undergone manual review and the feature vector of the target defect box, v i v is the feature vector of the defect box that has undergone manual review. j Let A be the feature vector of the target defect box, and let A be the confidence adjustment value.

[0013] Optionally, after constructing a confidence set for each defect box of each defect category, the process includes: after manually reviewing the defect boxes selected according to the review order, removing the selected defect boxes from the confidence set; and reconstructing the confidence set when the number of defect boxes in the confidence set is less than a preset number.

[0014] Optionally, the review order of defect boxes that have not been manually reviewed is determined based on the adjusted confidence level, including: calculating the review priority of defect boxes that have not been manually reviewed based on the adjusted confidence level, using the following formula:

[0015] K = S j hf(k1,k2)

[0016] Where K represents the review priority, S j h is the adjusted confidence level, f is a non-decreasing function of k1 and k2, where k1 and k2 are the number of reviewed confirmed or denied defect boxes that have been manually confirmed or denied in the same scenario, respectively; the review order is determined according to the review priority.

[0017] Optionally, determining the review order based on the review priority includes: constructing a priority queue, wherein the queue elements of the priority queue are defect boxes that have not been manually reviewed, and the priority of the queue element is the review priority when the queue element is placed into the priority queue; extracting the defect box and its priority from the priority queue, and recalculating its review priority; if the recalculated review priority is lower than the extracted priority, then the defect box is put back into the priority queue with the recalculated review priority, otherwise it is added to the queue to be displayed.

[0018] Optionally, determining the review order based on the review priority includes: selecting defective boxes from the priority queue for manual review based on the review priority; if the selected defective box belongs to the blacklist, discarding the selected defective box and reselecting a defective box; if other defective boxes belonging to the same scene as the selected defective box are in the queue awaiting review, reducing the review priority of the selected defective box and returning it to the priority queue; if the selected defective box does not belong to the blacklist and there are no other defective boxes belonging to the same scene in the queue awaiting review, adding the selected defective box to the queue awaiting display.

[0019] Secondly, the present invention provides a device for manual review of prediction results of an image defect detection model, comprising: an information acquisition module for acquiring defect boxes output by the image defect detection model and the feature vectors and confidence scores corresponding to the defect boxes; a confidence score adjustment module for adjusting the confidence scores of defect boxes that have not been manually reviewed based on the similarity between the feature vectors of manually reviewed defect boxes and the feature vectors of unreviewed defect boxes; and an order determination module for determining the review order of unreviewed defect boxes based on the adjusted confidence scores.

[0020] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the image defect detection model prediction result manual review method described in the first aspect or any corresponding embodiment.

[0021] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the image defect detection model prediction result manual review method described in the first aspect or any corresponding embodiment.

[0022] As can be seen from the above technical solutions, the embodiments of the present invention have the following advantages:

[0023] This invention provides a method, apparatus, device, and medium for manual review of image defect detection model prediction results. It acquires the defect bounding boxes, corresponding feature vectors, and confidence scores output by the image defect detection model. The confidence scores of unreviewed defect boxes are adjusted based on the similarity between the feature vectors of manually reviewed defect boxes and those of unreviewed defect boxes. The review order of unreviewed defect boxes is determined based on the adjusted confidence scores. By calculating the similarity of the feature vectors of defect boxes and adjusting the confidence scores of unreviewed defect boxes in real time based on the completed manual review results, the review order is dynamically adjusted, reducing the workload of manual review while ensuring the defect detection rate. Attached Figure Description

[0024] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating a method for manually reviewing the prediction results of an image defect detection model according to an embodiment of the present invention.

[0026] Figure 2 This is a flowchart illustrating another method for manually reviewing the prediction results of an image defect detection model according to an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram illustrating the working principle of the image defect detection model according to an embodiment of the present invention;

[0028] Figure 4This is a structural block diagram of the image defect detection model prediction result manual review device according to an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] This invention provides a method for manually reviewing the prediction results of an image defect detection model, applicable to the manual review of the prediction results of an image defect detection model for inspected images. Figure 1 and Figure 2 As shown, the method includes:

[0032] Step S101: Obtain the defect box output by the image defect detection model, as well as the corresponding feature vector and confidence level.

[0033] Specifically, an image defect detection model is trained based on an image target detection algorithm. This involves collecting inspection / monitoring images and identifying defective images from them, including equipment defects, potential hazards in the scene, and personnel violations. Defect bounding boxes are then labeled into the images to create labeled datasets for different defect categories, which are then used to train the image defect detection model.

[0034] The image defect detection model comprises three parts: feature extraction (backbone), feature fusion (neck), and target detection head (head). In this embodiment, based on the characteristics of the dataset, the feature extraction network for target detection uses ViT, Swin, or CSPDarkNet; the feature fusion network uses SimpleFPN; and the target detection head uses YOLO v5. Existing image defect detection models can detect defect boxes and output corresponding confidence scores. Building upon this, this application further extracts features from the defect box location within the feature extraction network of the image defect detection model to obtain feature defects, and further processes these feature defects to obtain feature vectors.

[0035] Step S102: Adjust the confidence level of the defective boxes that have not been manually reviewed based on the similarity between the feature vectors of the defective boxes that have been manually reviewed and the feature vectors of the defective boxes that have not been manually reviewed.

[0036] Specifically, by calculating the similarity between the feature vectors of defective boxes that have undergone manual review and those that have not, if the review result of the defective boxes that have undergone manual review is confirmed as defects, the confidence level is increased, and the greater the similarity, the greater the increase in confidence level. If the defective boxes are confirmed as not defects, the confidence level is decreased, and the greater the similarity, the greater the decrease in confidence level.

[0037] Step S103: Determine the review order of defect frames that have not been manually reviewed based on the adjusted confidence level.

[0038] Specifically, the higher the adjusted confidence level, the earlier the review order. The review stops when the confidence level of the reviewed defect box is equal to the preset score or when the number of reviews reaches the preset number.

[0039] This invention discloses a method for manually reviewing the prediction results of an image defect detection model. The method acquires the defect boxes output by the image defect detection model, along with their corresponding feature vectors and confidence levels. It adjusts the confidence levels of unreviewed defect boxes based on the similarity between the feature vectors of manually reviewed defect boxes and those of unreviewed defect boxes. The method then determines the review order of these unreviewed defect boxes based on the adjusted confidence levels. By calculating the similarity of the feature vectors of the defect boxes and adjusting the confidence levels of unreviewed defect boxes in real time based on the completed manual review results, the method dynamically adjusts the review order, reducing the workload of manual review while ensuring the defect detection rate.

[0040] In one specific embodiment, step S101, obtaining the defect bounding box, the corresponding feature vector, and the confidence score output by the image defect detection model, includes:

[0041] Step S1011: Obtain the defect box and the corresponding defect category, confidence level and defect features through the image defect detection model.

[0042] Specifically, such as Figure 3 As shown, the principle of the image defect detection model is as follows:

[0043] For each image, it is scaled to multiple resolutions and input into the image defect detection model for defect detection. The image defect detection model extracts feature maps of the input images at different resolutions through a feature extraction network and identifies defect boxes, obtaining the corresponding defect category and confidence level. All defect boxes predicted by the image defect detection model are selected and sorted by confidence level. The first 8192 boxes for each image (a pre-specified value, generally N times the final output box size, where 1 / N is the lower limit of the proportion of target boxes retained by the estimated non-maximum suppression method, in this example, 1 / N = 1 / 32) are sent to the non-maximum suppression algorithm for target box deduplication. Here, defect boxes of different categories at the same location are deduplicated. For each defect box, it is mapped onto the input images at different resolutions, and the logarithmic size of the mapped defect box (log(sqrt(width × height))) is calculated. The input image with the result closest to the defect size with the highest model recognition accuracy is marked as the feature extraction image corresponding to the defect box. For example, the defect size with the highest model recognition accuracy is log(144). For each input image resolution, the output layer features of the feature extraction network at resolution 16 are extracted, and the defect boxes that mark the image as the feature extraction image are extracted. The output layer features and defect boxes are processed by ROI-Align to obtain the D×M×M defect features corresponding to these defect boxes (M is pre-specified, with a value between 4 and 8, and D is the dimension of the feature extraction network). The defect features obtained from all resolution images are reordered, and the defect boxes and their corresponding defect categories, confidence scores, and defect features are used as the model output.

[0044] Step S1012: Dimensionality reduction and calibration of the defect features are performed to obtain feature vectors of a preset dimension.

[0045] Specifically, for all training images, the defect boxes, as well as the corresponding defect categories, confidence levels, and defect features, are extracted using the image defect detection model.

[0046] A feature analysis model is constructed. This model includes a feature mapping module, a feature processing module, and a feature prediction module. The feature mapping module maps D×M×M defect features to N×M×M defect features, where N is the feature dimension output by the image defect detection model, typically 64, 128, or 256; this embodiment uses 128. The feature processing module employs a Transformer architecture; this embodiment uses a 6-layer, 128-dimensional, 4-head Transformer to process the dimensionality-reduced defect features. The feature prediction module employs a CNN architecture; this embodiment uses a 4-layer, 128-dimensional 3×3 CNN and a 1-layer, 128×256 linear layer to predict the defect features. The function of the feature processing and feature prediction modules is to calibrate the dimensionality-reduced defect features to better represent the corresponding defect boxes.

[0047] The feature analysis model is trained. First, a sample selector is constructed to determine the probability of selection for each defect box based on its confidence level, and randomly selects defect boxes and their corresponding defect features. The M×M feature points of each sample are randomly divided into X parts (X=4 in this embodiment), with each part forming a subsample. All subsamples and the original sample are fed into the feature analysis model. Between the feature processing module and the feature prediction module, subsamples with only M×M / X points (dimension N×(M×M / X)) are restored to N×M×M by padding with mask tokens. Then, a loss function is constructed, with two learning objectives: one is the standardized value of the mean of the standardized features output by each layer of the original sample's FFN (dimension N×M×M), and the other is the standardized value of the mean ... The final loss function is obtained by weighted summing the average of the squared losses for objective one at the mask token filling positions in the subsamples with the average of the squared losses for objective two. Finally, the AdamW optimizer is used to train the feature analysis model and the samples.

[0048] The trained image defect detection model and feature analysis model are concatenated. The defect features output by the image defect detection model are fed into the feature analysis model for processing. The 129th to 256th features of the last layer of the feature analysis model are extracted. The mean of the last two dimensions is calculated to obtain the N-dimensional feature vector of each defect box. Finally, the defect box and its corresponding defect category, confidence score and feature vector are output.

[0049] In one specific implementation, the defect features are obtained by extracting features of the defect box location from the feature extraction network of the image defect detection model.

[0050] Specifically, by extracting features from the location of the defect box from the feature extraction network, the defect features corresponding to the defect box can be obtained. The defect features can be output based on the image defect detection model, simplifying the feature extraction process.

[0051] In one specific implementation, step S102, adjusting the confidence level of defective boxes that have not been manually reviewed based on the similarity between the feature vectors of defective boxes that have undergone manual review and those that have not, includes:

[0052] Step S1021: Construct a confidence set for each defect box of each defect category. The confidence set includes several defect boxes with the highest confidence in the same defect category or several defect boxes corresponding to feature vectors that are closest to the feature vectors of defect boxes that have completed manual review in the same defect category.

[0053] Specifically, after constructing a confidence set for the defect box for each defect category, the process includes:

[0054] After manually reviewing the defect boxes selected according to the review order, the selected defect boxes are removed from the confidence set; when the number of defect boxes in the confidence set is less than the preset number, the confidence set is reconstructed.

[0055] For example, before the review begins, a confidence set is constructed for each defect category, containing k defect boxes corresponding to the feature vectors with the highest confidence or the feature vectors closest to the feature vectors of defect boxes that have already undergone manual review. Specifically, K is set to 100. During the review, each time a defect box is selected for review, it is removed from the confidence set. When the number of defect boxes in the confidence set is less than k / 2, the confidence set is reconstructed. Furthermore, a KNN tree index is constructed for the feature vectors corresponding to all defect boxes. Specifically, hsnwlib and cosine distance are used to construct the KNN tree index. The purpose of constructing the KNN tree index is to facilitate the search for defect boxes.

[0056] Step S1022: Determine the confidence adjustment value based on the review results of the defect frames that have been manually reviewed.

[0057] Specifically, if the review result of the defect box that has completed manual review is confirmed as a defect, the confidence level adjustment value is A = Max - Si. If the review result is confirmed as not a defect, the confidence level adjustment value is A = -Max - Si, where Max is the maximum confidence level and Si is the confidence level of the defect box that has completed manual review.

[0058] Step S1023: Search for several target defect boxes that have not been manually reviewed and are located in different scenarios from the confidence set corresponding to the defect category to which the defect boxes that have completed manual review belong.

[0059] Specifically, the results of manual review have a significant impact on the confidence level of defect boxes of the same category. Therefore, in this embodiment, defect boxes of the same defect category are selected and their confidence levels are adjusted. The confidence level deviation of different forms of the same defect category is taken into account. The confidence level is adjusted in real time based on the review results, which can make the confidence level closer to the true value.

[0060] For defect boxes in the same scene, it is highly likely that they are duplicate images taken multiple times for the same defect. Therefore, in order to avoid a large number of images of the same scene appearing at the same time, a priority penalty item is set instead of adjusting based on the review results. Therefore, it is necessary to remove them in this step.

[0061] Step S1024: Adjust the confidence of the target defect box based on the similarity between the feature vector of the defect box that has been manually reviewed and the feature vector of the target defect box, as well as the confidence adjustment value.

[0062] In one specific implementation, the confidence level adjustment formula is as follows:

[0063] S jh =Sj+aA×exp(-w×D(vi,vj)) 2 / 2)

[0064] In the formula, S jh S represents the confidence level adjusted for the target defect bounding box. j The confidence level of the target defect box before adjustment, where a and w are preset coefficients, and D(v) i ,v j ) represents the similarity between the feature vectors of the defect bounding boxes that have undergone manual review and the feature vectors of the target defect bounding box, v i v is the feature vector of the defect box that has undergone manual review. j Let A be the feature vector of the target defect bounding box, and let A be the confidence adjustment value.

[0065] Specifically, for each manually confirmed or denied defect box i, if the review result of the defect box that has completed manual review confirms it as a defect, the confidence adjustment value is A = Max - Si; if the review result confirms it as not a defect, the confidence adjustment value is A = -Max - Si, where Max is the maximum confidence value and Si is the confidence value of the defect box that has completed manual review. Otherwise, the confidence adjustment value is 0. Max is the maximum confidence value, which is taken as 8 in this embodiment. a and w are preset coefficients that can be set independently, for example, 0.3 and 50 respectively. Similarity is calculated using the cosine similarity distance between two vectors, i.e.:

[0066]

[0067] in, After adjusting the confidence level of the target defect boxes, the review order of the target defect boxes is adjusted according to the adjusted confidence level.

[0068] In one specific implementation, step S103, determining the review order of defect frames that have not been manually reviewed based on the adjusted confidence level, includes:

[0069] Step S1031: Calculate the review priority of defect frames that have not been manually reviewed based on the adjusted confidence level. The calculation formula is as follows:

[0070] K = S jh -f(k1,k2)

[0071] Where K represents the review priority, Sjh The adjusted confidence level is f, which is a non-decreasing function of k1 and k2, where k1 and k2 are the number of reviewed and confirmed or denied defect boxes that have been manually confirmed or denied in the same scenario, respectively.

[0072] Specifically, f(k1,k2) is:

[0073] if k1+0.3k2<=0:f=0; elif k1+0.3k2<=3:f=2; elifk1+0.3k2<=10:f=4; else:f=8.

[0074] In this embodiment of the invention, the same scene is defined as follows: during power transmission line inspections, photos of the same tower belong to the same scene; in surveillance images, images from the same camera belong to the same scene; during railway inspections, photos taken every 500 meters can be considered as the same scene.

[0075] Step S1032: Determine the review order based on review priority.

[0076] Specifically, a priority queue is constructed based on the review priority. The review priority and the set defect box number of each defect box are added to the queue, and the priority queue is constructed in a heap manner. The review order is determined by the priority queue, that is, a review priority is set for each defect box. The higher the review priority, the earlier it is reviewed.

[0077] In one specific implementation, step S1032, determining the review order based on review priority, includes:

[0078] Step b1: Construct a priority queue, wherein the queue elements of the priority queue are defect boxes that have not been manually reviewed, and the priority of the queue element is the review priority when the queue element is placed into the priority queue.

[0079] Step b2: Extract the defect box and its priority from the priority queue, and recalculate its review priority. If the recalculated review priority is lower than the extracted priority, put the defect box back into the priority queue with the recalculated review priority; otherwise, add it to the queue to be displayed.

[0080] Specifically, a priority queue is constructed based on the review priority. The review priority and the set defect box number of each defect box are added to the queue, and the priority queue is constructed in a heap manner. The review order is determined by the priority queue, that is, a review priority is set for each defect box. The higher the review priority, the earlier it is reviewed.

[0081] Extract defect boxes and their priorities from the priority queue, and recalculate the real-time review priority of the extracted defect boxes. If the recalculated review priority is lower than the priority at the time of extraction (i.e., the review priority before adjustment), then put the defect box back into the priority queue with the recalculated review priority; otherwise, add it to the display queue and determine the review order through the display queue.

[0082] Furthermore, step S1032, determining the review order based on review priority, also includes:

[0083] Step a1: Select defect boxes from the priority queue for manual review based on review priority.

[0084] Specifically, defect bounding boxes, their corresponding defect categories, confidence levels, and defect vectors are obtained by predicting all images in a specified folder. Review priorities are calculated based on the confidence levels, and each defect bounding box's review priority and a set defect bounding box number are added to a queue. A priority queue is constructed using a heap. The defect bounding box with the highest review priority is selected from the priority queue for manual review.

[0085] Step a2: If the selected defect box belongs to the blacklist, discard the selected defect box and select a new defect box.

[0086] Specifically, reviewers can set up category blacklists during the review process to block problem categories containing specific keywords. If the selected defect box belongs to a category on the blacklist, the selected defect box will be discarded to improve review efficiency.

[0087] Step a3: If other defect boxes belonging to the same scene as the selected defect box are in the queue to be reviewed, then the review priority of the selected defect box is reduced and it is put back into the priority queue.

[0088] Specifically, if the selected defect box belongs to other defect boxes in the same scene that exist in the queue to be reviewed, its review priority is reduced by T. In this embodiment of the invention, T is 1, and it is put back into the priority queue with the reduced review priority.

[0089] Step a4: If the selected defect box does not belong to the blacklist and there are no other defect boxes belonging to the same scenario in the queue to be reviewed, then add the selected defect box to the queue to be displayed.

[0090] By setting up a blacklist and prioritizing penalties for defect boxes belonging to the same scenario, we can prevent images from appearing on the blacklist and avoid the frequent appearance of images from the same scenario, thereby improving the efficiency of the review process.

[0091] In one embodiment, the defect review interface displays the defect boxes, confidence levels, defect categories, and defect images in the review queue to the reviewers, who then confirm or deny whether the defect is a defect. During this process, users can view adjacent images or defect boxes by clicking on images, fill in relevant defect information in the editing area, set a defect category blacklist through the menu bar, or save defect orders and defect review progress, etc.

[0092] This invention also provides a device for manual review of image defect detection model prediction results, such as... Figure 4 As shown, it includes:

[0093] The information acquisition module 401 is used to acquire the defect box output by the image defect detection model and the feature vector and confidence level corresponding to the defect box;

[0094] The confidence adjustment module 402 is used to adjust the confidence of defective boxes that have not been manually reviewed based on the similarity between the feature vectors of defective boxes that have completed manual review and the feature vectors of defective boxes that have not been manually reviewed.

[0095] The sequence determination module 403 is used to determine the review order of defect boxes that have not been manually reviewed based on the adjusted confidence level.

[0096] In one embodiment, the information acquisition module 401 includes:

[0097] The first acquisition module is used to acquire defect boxes and the corresponding defect categories, confidence levels and defect features through an image defect detection model.

[0098] The second acquisition module is used to reduce the dimensionality of the defect features and calibrate them to obtain feature vectors of a preset dimension.

[0099] In one embodiment, the defect features are obtained by extracting features of the defect box location from the feature extraction network of the image defect detection model.

[0100] In one embodiment, the confidence adjustment module 402 includes:

[0101] The set construction module is used to construct a confidence set for each defect box of each defect category. The confidence set includes several defect boxes with the highest confidence in the same defect category or defect boxes corresponding to several feature vectors that are closest to the feature vectors of defect boxes that have completed manual review in the same defect category.

[0102] The adjustment value determination module is used to determine the confidence adjustment value based on the review results of the defect boxes that have completed manual review;

[0103] The defect box search module is used to search for several target defect boxes that have not been manually reviewed and are located in different scenarios from the defect boxes that have been manually reviewed, from the confidence set corresponding to the defect category to which the defect boxes that have been manually reviewed belong.

[0104] The confidence module is used to adjust the confidence of the target defect box based on the similarity between the feature vectors of the defect boxes that have undergone manual review and the feature vectors of the target defect box, as well as the confidence adjustment value.

[0105] In one embodiment, the method for adjusting the confidence level of the target defect box based on the similarity between the feature vector of the defect box that has undergone manual review and the feature vector of the target defect box, as well as the confidence adjustment value, is as follows:

[0106] S jh =S j +aA×exp(-w×D(vi, vj) 2 In equation / 2), S jh S is the adjusted confidence level. j The confidence level before adjustment is given, where a and w are preset coefficients, and D(v) i ,v j ) represents the similarity between the feature vectors of the defect bounding boxes that have undergone manual review and the feature vectors of the target defect bounding box, v i v is the feature vector of the defect box that has undergone manual review. j Let A be the feature vector of the target defect bounding box, and let A be the confidence adjustment value.

[0107] In one embodiment, the adjustment value determination module includes:

[0108] The first adjustment module is used to adjust the confidence level to A = Max - Si if the review result of the defect box that has been manually reviewed is confirmed to be a defect.

[0109] The second adjustment module is used to adjust the confidence level to A = -Max - Si if the review result confirms that it is not a defect, where Max is the maximum confidence level and Si is the confidence level of the defect box that has been manually reviewed.

[0110] In one embodiment, the confidence adjustment module 402 further includes:

[0111] The removal module is used to remove the selected defect boxes from the confidence set after manual review based on the review order.

[0112] The reconstruction module is used to reconstruct the confidence set when the number of defect boxes in the confidence set is less than a preset number.

[0113] In one embodiment, the sequence determination module 403 includes:

[0114] The review priority module is used to calculate the review priority of defect frames that have not been manually reviewed based on the adjusted confidence level. The calculation formula is as follows:

[0115] K = S jh -f(k1,k2)

[0116] Where K represents the review priority, S jh For the adjusted confidence level, f is a non-decreasing function of k1 and k2, where k1 and k2 are the number of reviewed and confirmed or denied defect boxes that have been manually confirmed or denied in the same scenario, respectively.

[0117] The review order module is used to determine the review order based on review priority.

[0118] In one embodiment, the review sequence module includes:

[0119] The selection module is used to select defect boxes from the priority queue for manual review based on the review priority.

[0120] The discard module is used to discard the selected defect box and reselect a defect box if the category of the selected defect box belongs to the blacklist.

[0121] The penalty module is used to reduce the review priority of the selected defect box and put it back into the priority queue if other defect boxes belonging to the same scene as the selected defect box are in the queue to be reviewed.

[0122] The "Add" module is used to add the selected defect box to the "Display" queue if the category of the selected defect box does not belong to the blacklist and there are no other defect boxes belonging to the same scenario in the queue awaiting review.

[0123] This invention provides a device for manually reviewing the prediction results of an image defect detection model. It acquires the defect boxes output by the image defect detection model, along with their corresponding feature vectors and confidence levels. The device adjusts the confidence levels of unreviewed defect boxes based on the similarity between the feature vectors of manually reviewed defect boxes and those of unreviewed defect boxes. It then determines the review order of these unreviewed defect boxes based on the adjusted confidence levels. By calculating the similarity of the feature vectors of the defect boxes and adjusting the confidence levels of unreviewed defect boxes in real time based on the completed manual review results, the device dynamically adjusts the review order, reducing the workload of manual review while ensuring the defect detection rate.

[0124] This invention also provides a schematic diagram of the structure of a computer device, such as... Figure 5As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 5 Take a processor 10 as an example.

[0125] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0126] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0127] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0128] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0129] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0130] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0131] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0132] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method for manual review of prediction results from an image defect detection model, characterized in that, include: Obtain the defect bounding box output by the image defect detection model, as well as the feature vector and confidence level corresponding to the defect bounding box; The confidence level of defective boxes that have not been manually reviewed is adjusted based on the similarity between the feature vectors of defective boxes that have undergone manual review and those that have not. The review order of defect frames that have not been manually reviewed is determined based on the adjusted confidence level; The confidence level of defective boxes that have not undergone manual review is adjusted based on the similarity between the feature vectors of defective boxes that have completed manual review and those that have not. This includes: A confidence set is constructed for each defect box of each defect category. The confidence set includes several defect boxes with the highest confidence in the same defect category or defect boxes corresponding to several feature vectors that are closest to the feature vectors of defect boxes that have completed manual review in the same defect category. The confidence level adjustment value is determined based on the review results of the defect frames that have undergone manual review; From the confidence set corresponding to the defect category to which the defect box that has completed manual review belongs, find several target defect boxes that have not been manually reviewed and are located in different scenarios from the defect boxes that have completed manual review. The confidence level of the target defect box is adjusted based on the similarity between the feature vectors of the defect boxes that have undergone manual review and the feature vectors of the target defect box, as well as the confidence adjustment value. The adjustment method is as follows: In the formula, The adjusted confidence level. The confidence level before adjustment. and For preset coefficients, The similarity between the feature vector of the defect box that has undergone manual review and the feature vector of the target defect box is calculated. The feature vector of the defect box that has completed manual review. The feature vector of the target defect box. This is the confidence level adjustment value.

2. The method according to claim 1, characterized in that, Obtaining the defect bounding box output by the image defect detection model, along with the corresponding feature vector and confidence score, includes: The defect bounding box, along with the corresponding defect category, confidence level, and defect features, are obtained through an image defect detection model. The defect features are reduced in dimensionality and calibrated to obtain a feature vector of a preset dimension.

3. The method according to claim 2, characterized in that, The defect features are obtained by extracting features at the location of the defect box from the feature extraction network of the image defect detection model.

4. The method according to claim 1, characterized in that, After constructing a confidence set for the defect box for each defect category, it includes: After manually reviewing the defect boxes selected according to the review order, the selected defect boxes are removed from the confidence set. The confidence set is reconstructed when the number of defect boxes in the confidence set is less than a preset number.

5. The method according to claim 1, characterized in that, The review order of defect frames that have not been manually reviewed is determined based on the adjusted confidence level, including: The review priority of defect frames that have not been manually reviewed is calculated based on the adjusted confidence level, using the following formula: in, To prioritize the review, For the adjusted confidence level, f is a non-decreasing function of k1 and k2, where k1 and k2 are the number of reviewed and confirmed or denied defect boxes that have been manually confirmed or denied in the same scenario, respectively. The review order is determined based on the aforementioned review priority.

6. The method according to claim 5, characterized in that, The review order is determined according to the aforementioned review priority, including: Construct a priority queue, wherein the queue elements of the priority queue are defect boxes that have not been manually reviewed, and the priority of the queue element is the review priority when the queue element is placed into the priority queue; Extract the defect box and its priority from the priority queue, and recalculate its review priority. If the recalculated review priority is lower than the extracted priority, put the defect box back into the priority queue with the recalculated review priority; otherwise, add it to the queue to be displayed.

7. The method according to claim 6, characterized in that, The review order is determined according to the aforementioned review priority, including: Defect boxes are selected from the priority queue for manual review based on their review priority. If the selected defect box belongs to the blacklist, the selected defect box will be discarded and a new defect box will be selected. If other defect boxes belonging to the same scenario as the selected defect box are in the queue awaiting review, then the review priority of the selected defect box will be reduced and it will be put back into the priority queue. If the selected defect box does not belong to the blacklist and there are no other defect boxes belonging to the same scenario in the queue awaiting review, then the selected defect box will be added to the queue awaiting display.

8. A device for manual review of prediction results of an image defect detection model, characterized in that, include: The information acquisition module is used to acquire the defect box output by the image defect detection model, as well as the feature vector and confidence level corresponding to the defect box; The confidence adjustment module is used to adjust the confidence of defective boxes that have not been manually reviewed based on the similarity between the feature vectors of defective boxes that have undergone manual review and those that have not. The sequence determination module is used to determine the review order of defect boxes that have not been manually reviewed based on the adjusted confidence level; The confidence adjustment module includes: The set construction module is used to construct a confidence set for each defect box of each defect category. The confidence set includes several defect boxes with the highest confidence in the same defect category or defect boxes corresponding to several feature vectors that are closest to the feature vectors of defect boxes that have completed manual review in the same defect category. The adjustment value determination module is used to determine the confidence adjustment value based on the review results of the defect boxes that have completed manual review; The defect box search module is used to search for several target defect boxes that have not been manually reviewed and are located in different scenarios from the defect boxes that have been manually reviewed, from the confidence set corresponding to the defect category to which the defect boxes that have been manually reviewed belong. The confidence module is used to adjust the confidence of the target defect box based on the similarity between the feature vector of the defect box that has completed manual review and the feature vector of the target defect box, as well as the confidence adjustment value. The method for adjusting the confidence level of the target defect box based on the similarity between the feature vectors of the defect boxes that have undergone manual review and the feature vectors of the target defect box, as well as the confidence adjustment value, is as follows: In the formula, The adjusted confidence level. The confidence level before adjustment. and For preset coefficients, The similarity between the feature vectors of the defect bounding boxes that have undergone manual review and the feature vectors of the target defect bounding box. The feature vector of the defect box that has completed manual review. Let be the feature vector of the target defect bounding box. This is the confidence level adjustment value.

9. A computer device, characterized in that, include: A memory and a processor are interconnected, the memory stores computer instructions, and the processor executes the computer instructions to perform the manual review method for the prediction results of the image defect detection model according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the manual review method for the prediction results of the image defect detection model according to any one of claims 1 to 7.

Images