Defect detection model training method and device, computer device, readable storage medium and program product

Abstract

The application relates to a defect detection model training method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a feature map of a first sample image; performing segmentation mask on the feature map through a segmentation head in an initial defect detection model, simultaneously performing defect prediction classification through a classification head connected in parallel with the segmentation head, obtaining a target segmentation mask set of defect instances output by the segmentation head and a target query vector group, and a defect prediction classification set output by the classification head; obtaining a joint loss value according to the target segmentation mask set, the target query vector group and the defect prediction classification set, adjusting parameters in the initial defect detection model according to the joint loss value, and obtaining a target defect detection model. The method can inhibit the influence of segmentation errors on classification results and improve the accuracy of defect detection.

Description

Technical Field

[0001] This application relates to the field of industrial vision inspection technology, and in particular to a defect detection model training method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of industrial vision inspection technology, industrial vision inspection is a core link in quality control of intelligent manufacturing, and defect detection methods based on instance segmentation are gradually becoming the mainstream.

[0003] In traditional techniques, a sequential pipeline of "segmentation first, classification later" is usually adopted: first, the segmentation model generates a mask region for defect instances, and then the mask region is sent to an independent classifier for category determination.

[0004] However, the current segmentation-then-classification method has obvious drawbacks: the segmentation and classification tasks are disconnected from each other, and the positioning error in the segmentation stage is directly transmitted to the classification stage and cannot be corrected in subsequent stages, resulting in low defect detection accuracy. Summary of the Invention

[0005] Therefore, it is necessary to provide a defect detection model training method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of defect detection in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a method for training a defect detection model, the method comprising:

[0007] Obtain the feature map of the first sample image;

[0008] The segmentation mask is applied to the feature map by the segmentation head in the initial defect detection model. Simultaneously, defect prediction classification is performed by the classification head connected in parallel with the segmentation head. This yields the target segmentation mask set and target query vector set of defect instances output by the segmentation head, as well as the defect prediction classification set output by the classification head.

[0009] Based on the target segmentation mask set, target query vector set, and defect prediction classification set, a joint loss value is obtained. The parameters in the initial defect detection model are adjusted according to the joint loss value to obtain the target defect detection model.

[0010] In one embodiment, a segmentation mask is applied to the feature map using a segmentation head in the initial defect detection model. Simultaneously, a defect prediction classification is performed using a classification head connected in parallel with the segmentation head. This yields a target segmentation mask set and a target query vector set for defect instances output by the segmentation head, as well as a defect prediction classification set output by the classification head, including:

[0011] The current layer decoder of the segmentation head performs cross-attention interaction with the feature map based on the historical query vector group output by the previous layer decoder of the current layer to obtain a new query vector group for the current layer.

[0012] The current layer's defect prediction classification set is obtained by using the classification head connected in parallel with the current layer decoder based on the corresponding new query vector set; wherein, while the next layer decoder of the current layer executes the new query vector set output by the current layer decoder as the historical query vector set, the classification head connected in parallel with the current layer decoder executes the corresponding new query vector set.

[0013] When the current layer is the last layer, the new query vector group of the last layer is used as the target query vector group; the target segmentation mask set is obtained by the mask processing module of the segmentation head based on the target query vector group.

[0014] In one embodiment, obtaining the target segmentation mask set based on the target query vector group includes:

[0015] Based on the target query vector group, an initial segmentation mask set is obtained;

[0016] The initial segmentation mask set is iteratively refined to obtain the target segmentation mask set.

[0017] In one embodiment, the iterative mask refinement of the initial segmentation mask set to obtain the target segmentation mask set includes:

[0018] Based on the initial segmentation mask set and the feature map, the feature region corresponding to each mask in the initial segmentation mask set is obtained;

[0019] For each of the aforementioned feature regions, at least one of boundary refinement, hole filling, and topology preservation processing is performed to obtain at least one of the boundary refinement mask, hole filling mask, and topology preservation mask, respectively.

[0020] Based on at least one of the boundary refinement mask, the hole filling mask, and the topology preservation mask, the refinement mask update amount is obtained;

[0021] Based on the refined mask update amount and the initial segmentation mask set, the target segmentation mask set is determined.

[0022] In one embodiment, the method further includes:

[0023] If the classification confidence of the defect prediction classification corresponding to any defect instance in the defect prediction classification set is lower than a preset classification confidence threshold, the initial segmentation mask corresponding to the defect instance is determined from the initial segmentation mask set.

[0024] Based on the initial segmentation mask and the feature map, a new feature map is obtained;

[0025] The new feature map is processed by the target quantity layer decoder to obtain a new query vector for the corresponding layer;

[0026] Based on the new query vector, the classification head connected in parallel with the target number layer decoder obtains a new defect prediction classification for the corresponding layer.

[0027] The new query vector is processed by the mask processing module in the initial defect detection model to obtain a new segmentation mask.

[0028] In one embodiment, obtaining the joint loss value based on the target segmentation mask set, the target query vector set, and the defect prediction classification set includes:

[0029] Based on the aforementioned defect prediction classification set, determine the auxiliary loss function value;

[0030] The main loss function value is determined based on the segmentation mask set, the target query vector set, and the defect prediction classification set;

[0031] The joint loss value is determined based on the auxiliary loss function and the main loss function.

[0032] In one embodiment, determining the main loss function value based on the segmentation mask set, the target query vector set, and the defect prediction classification set includes:

[0033] Based on the initial segmentation mask set and the true segmentation mask set of the first sample image, the segmentation loss value is determined;

[0034] Based on the defect prediction classification set, the true category set of the first sample image, and the target query vector group, the classification loss value is determined;

[0035] Based on the initial segmentation mask set and the target segmentation mask set, the consistency loss value is determined;

[0036] The main loss value is determined based on the consistency loss value, the segmentation loss value, and the classification loss value.

[0037] In one embodiment, determining the segmentation loss value based on the initial segmentation mask set and the true segmentation mask set of the first sample image includes:

[0038] Based on the initial segmentation mask set and the real segmentation mask set of the first sample image, the mask segmentation loss value is determined;

[0039] The pixel segmentation loss value is determined based on the pixel binary classification of the initial segmentation mask set and the pixel binary classification of the real segmentation mask set.

[0040] The segmentation loss value is determined based on the mask segmentation loss value and the pixel segmentation loss value.

[0041] In one embodiment, determining the classification loss value based on the defect prediction classification set, the true category set of the first sample image, and the target query vector set includes:

[0042] Based on the defect prediction classification set and the true category set of the first sample image, the cross-entropy loss value is determined;

[0043] Based on the target query vector set, determine the triplet loss value;

[0044] The classification loss value is determined based on the cross-entropy loss value and the triplet loss value.

[0045] In one embodiment, obtaining the feature map of the first sample image includes:

[0046] Acquire the original sample image and preprocess the original sample image to obtain the first sample image;

[0047] Based on the first sample image, determine the multi-scale image;

[0048] Based on the multi-scale image, the feature map of the first sample image is determined.

[0049] In one embodiment, the method further includes:

[0050] When a second sample image corresponding to a new defect type is obtained, the second sample image is input into the target defect detection model, and the sample query vector corresponding to the second sample image output by the target defect detection model is received.

[0051] Based on the aforementioned sample query vectors, a new sample query vector is determined;

[0052] The new sample query vector is added to the category library; wherein the category library stores query vectors of known defect types;

[0053] The target defect detection model is fine-tuned using knowledge distillation with the updated category library.

[0054] Secondly, this application provides a defect detection method, the method comprising:

[0055] Acquire the target image to be detected;

[0056] The target image to be detected is input into the target defect detection model as described in any of the above embodiments, and the instance segmentation mask, instance prediction category, and classification confidence corresponding to the instance prediction category are obtained for each defect instance output by the target defect detection model.

[0057] Thirdly, this application provides a defect detection model training device, the device comprising:

[0058] The first acquisition module is used to acquire the feature map of the first sample image;

[0059] A segmentation head is used to perform segmentation masking on the feature map to obtain a target segmentation mask set and a target query vector set for defect instances;

[0060] A classification head connected in parallel with the segmentation head is used to perform defect prediction classification to obtain a defect prediction classification set of defect instances;

[0061] The fine-tuning module is used to obtain a joint loss value based on the target segmentation mask set, the target query vector set, and the defect prediction classification set, and to adjust the parameters in the initial defect detection model based on the joint loss value to obtain the target defect detection model.

[0062] Fourthly, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0063] Fifthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0064] Sixthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0065] The aforementioned defect detection model training method, apparatus, computer equipment, computer-readable storage medium, and computer program product firstly perform cross-scale processing on the first sample image to obtain a feature map. This feature map combines low-scale global structural information with high-scale fine texture information, taking into account both details and the overall picture. Secondly, during the segmentation masking process on the feature map using the segmentation head in the initial defect detection model, a classification head connected in parallel with the segmentation head synchronously performs defect prediction classification. This yields the target segmentation mask set and target query vector set of defect instances output by the segmentation head, and the defect prediction classification set output by the classification head. The segmentation head and classification head share the same set of query vectors, allowing classification information to supervise the segmentation process in reverse, enabling the segmentation head to learn features helpful for classification during training, thereby reducing the impact of segmentation errors on classification. Finally, based on the target segmentation mask set, target query vector set, and defect prediction classification set, a joint loss value is obtained. The parameters in the initial defect detection model are adjusted based on this joint loss value to obtain the target defect detection model. The joint loss function optimizes both classification and segmentation results, forcing the model to learn features that are beneficial for both segmentation and classification. Since segmentation errors do not propagate to classification results, the model's accuracy in defect detection is improved. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a flowchart illustrating a defect detection model training method in one embodiment;

[0068] Figure 2 This is a schematic diagram illustrating the process of a classification head synchronously performing defect prediction and classification during the segmentation mask generation process of the segmentation head on the feature map in one embodiment.

[0069] Figure 3 This is a schematic diagram illustrating the process of iteratively refining the initial segmentation mask set in one embodiment;

[0070] Figure 4 This is a flowchart illustrating the reverse correction of the segmentation mask based on the classification result in one embodiment.

[0071] Figure 5 This is a schematic diagram of the process for determining the joint loss value in one embodiment;

[0072] Figure 6This is a flowchart illustrating the process of determining the main loss function value in one embodiment;

[0073] Figure 7 This is a schematic diagram of the process for determining the segmentation loss value in one embodiment;

[0074] Figure 8 This is a schematic diagram of the process for determining the classification loss value in one embodiment;

[0075] Figure 9 This is a schematic diagram illustrating the process of incremental learning and target defect detection model update in one embodiment;

[0076] Figure 10 This is a structural block diagram of a defect detection model training device in one embodiment;

[0077] Figure 11 This is a structural block diagram of a defect detection device in one embodiment;

[0078] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0080] In one embodiment, such as Figure 1 As shown, a defect detection model training method is provided. This embodiment illustrates the method applied to a terminal, but it is understood that the method can also be applied to a server, and to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes steps S102 to S106. Wherein:

[0081] Step S102: Obtain the feature map of the first sample image.

[0082] The first sample image is an image of an industrial product, such as an image of the surface of a PCB board.

[0083] Optionally, the terminal acquires a first sample image through an image acquisition module. The first sample image can be an original image of the industrial product or a pre-processed image. The image acquisition module sends the first sample image to the initial defect detection model, and the first acquisition module in the initial defect detection model performs cross-scale processing on the first sample image to obtain a feature map.

[0084] Step S104: Perform segmentation masking on the feature map using the segmentation head in the initial defect detection model, and simultaneously perform defect prediction classification using the classification head connected in parallel with the segmentation head, to obtain the target segmentation mask set and target query vector group of defect instances output by the segmentation head, and the defect prediction classification set output by the classification head.

[0085] The initial defect detection model includes a first acquisition module, a segmentation head, and a classification head, with the first acquisition module connected to the classification head and the segmentation head. Each target segmentation mask in the target segmentation mask set corresponds to a defect instance; each query vector in the target query vector group corresponds to a defect instance; similarly, each defect prediction classification in the defect prediction classification set corresponds to a defect instance.

[0086] Optionally, during the process of performing segmentation masking on the feature map by the segmentation head in the initial defect detection model, the terminal also performs defect prediction classification synchronously by the classification head connected in parallel with the segmentation head, to obtain the target segmentation mask set and target query vector group of multiple defect instances output by the segmentation head, as well as the defect prediction classification set output by the classification head.

[0087] Step S106: Based on the target segmentation mask set, the target query vector set, and the defect prediction classification set, obtain the joint loss value. Adjust the parameters in the initial defect detection model based on the joint loss value to obtain the target defect detection model.

[0088] Optionally, the terminal uses a fine-tuning module to obtain a joint loss value based on the target segmentation mask set, the target query vector set, the defect prediction classification set, and the joint loss function. The joint loss function is used to simultaneously optimize the classification results (defect prediction classification set) and the segmentation results (segmentation mask set and query vector set). The terminal then adjusts the parameters in the initial defect detection model based on the joint loss value until the joint loss value converges, resulting in a trained target defect detection model.

[0089] In the above defect detection model training method, firstly, the first sample image undergoes cross-scale processing to obtain a feature map. This feature map combines low-scale global structural information with high-scale fine texture information, taking into account both details and the overall picture. Secondly, during the segmentation masking process on the feature map using the segmentation head in the initial defect detection model, a classification head connected in parallel with the segmentation head synchronously performs defect prediction classification. This yields the target segmentation mask set and target query vector set output by the segmentation head, and the defect prediction classification set output by the classification head. The segmentation head and classification head share the same set of query vectors, allowing classification information to supervise the segmentation process in reverse. This enables the segmentation head to learn features helpful for classification during training, thus reducing the impact of segmentation errors on classification. Finally, a joint loss value is obtained based on the target segmentation mask set, target query vector set, and defect prediction classification set. The parameters in the initial defect detection model are adjusted based on this joint loss value to obtain the target defect detection model. The joint loss function simultaneously optimizes both classification and segmentation results, forcing the model to learn features that are beneficial for both segmentation and classification. Furthermore, since segmentation errors do not propagate to classification results, the model's accuracy in defect detection is improved.

[0090] In one exemplary embodiment, such as Figure 2 As shown, the initial defect detection model performs segmentation masking on the feature map using the segmentation head, and simultaneously performs defect prediction classification using the classification head connected in parallel with the segmentation head. This yields the target segmentation mask set and target query vector set of the defect instances output by the segmentation head, and the defect prediction classification set output by the classification head, including the following steps S202 to S206. Wherein:

[0091] Step S202: The current layer decoder of the segmentation head performs cross-attention interaction with the feature map based on the historical query vector group output by the previous layer decoder, to obtain a new query vector group for the current layer.

[0092] The segmentation head consists of multiple layers, each with a corresponding decoder. For example, the segmentation head may use a standard architecture of a 6-layer encoder and a 6-layer decoder.

[0093] Optionally, the terminal uses the current layer decoder of the segmentation head to perform cross-attention interaction with the feature map based on the historical query vector group output by the previous layer decoder, to obtain a new query vector group for the current layer. If the current layer is the first layer, then the historical query vector group is the initialized query vector group. If the current layer is the second layer, then the historical query vector group is the query vector group output by the first layer decoder. Finally, a new query vector group updated for each layer can be obtained.

[0094] Step S204: Based on the corresponding new query vector group, the classification head connected in parallel with the current layer decoder is used to obtain the defect prediction classification set of the current layer.

[0095] In this process, while the next layer decoder of the current layer executes the new query vector group output by the current layer decoder as the historical query vector group, the classification head connected in parallel with the current layer decoder executes the corresponding new query vector group.

[0096] Optionally, after each layer decoder of the segmentation head, a lightweight classification head is connected in parallel. The classification head takes the new instance query vector set output by the current layer decoder as input and outputs the defect prediction classification of each defect instance in the current layer. All defect prediction classifications form a defect prediction classification set.

[0097] Furthermore, for example, if the current layer is layer i, where i is a natural number greater than 1, the current layer decoder of the segmentation head performs cross-attention interaction with the feature map based on the historical query vector group output by the decoder of the previous layer (i-1) of current layer i, to obtain a new query vector group i for the current layer i. This new query vector group i is then passed to the decoder of layer (i+1), while the classification head i, connected in parallel with the decoder of current layer i, also uses the new query vector group i as input. That is, while the decoder of layer (i+1) is executing the new query vector group i, the classification head i is also executing the new query vector group i. In other words, the segmentation head and the classification head share the new query vector group i.

[0098] Step S206: When the current layer is the last layer, the new query vector group of the last layer is taken as the target query vector group; the target segmentation mask set is obtained by the mask processing module of the segmentation head according to the target query vector group.

[0099] Optionally, until the current layer is the last layer, the terminal uses the new query vector group of the last layer as the target query vector group. For example, the last layer is L. The mask processing module is connected to the last layer decoder. It should be noted that the mask processing module is only connected to the last layer decoder; all layers before layer L are not connected to the mask processing module. The principle is that mask generation requires the highest quality features, and only the query vector group of the last layer after multiple layers of cross-attention can provide sufficiently accurate spatial details. If intermediate layers output masks, it will increase the computational load and the quality will be low, which has no practical value.

[0100] Furthermore, the terminal sends the new query vector group L output by the last layer decoder of the segmentation head to the mask processing module; the terminal processes the target query vector group through the mask processing module of the segmentation head to obtain the target segmentation mask set.

[0101] In this embodiment, a lightweight classification head is connected in parallel after each layer of the segmentation head's decoder, sharing a new set of query vectors. This allows the classification results and segmentation results to naturally align with the same defective instances, avoiding mismatches between the segmented regions and the input of the classification head, and reducing localization errors in the segmentation stage (which cannot be corrected in subsequent stages).

[0102] In an exemplary embodiment, obtaining a target segmentation mask set based on a target query vector group includes: obtaining an initial segmentation mask set based on the target query vector group; and performing iterative mask refinement on the initial segmentation mask set to obtain the target segmentation mask set.

[0103] Optionally, the terminal processes the target query vector group through the mask processing module of the segmentation header to obtain an initial segmentation mask set; the terminal then performs iterative mask refinement on the initial segmentation mask set through the mask processing module of the segmentation header to obtain the target segmentation mask set.

[0104] Optionally, the terminal can also skip iterative mask refinement of the initial segmentation mask set through the mask processing module of the segmentation head, and directly use the initial segmentation mask set as the target segmentation mask set.

[0105] In this embodiment, by refining the initial segmentation mask set, a reliable input can be provided for subsequent defect quantification analysis.

[0106] In one exemplary embodiment, such as Figure 3 As shown, the initial segmentation mask set is iteratively refined to obtain the target segmentation mask set, including steps S302 to S308. Wherein:

[0107] Step S302: Based on the initial segmentation mask set and feature map, obtain the feature region corresponding to each mask in the initial segmentation mask set.

[0108] Optionally, the iterative mask refinement can adopt a T-step iterative refinement strategy, where T is a preset hyperparameter and T≥2. The refinement process in step t includes: the terminal takes the initial segmentation mask set M_{t-1} from step t-1 and the feature map F_co as input, and extracts the feature region corresponding to each mask.

[0109] Step S304: Perform at least one of boundary refinement, hole filling and topology preservation processing on each feature region to obtain at least one of boundary refinement mask, hole filling mask and topology preservation mask respectively.

[0110] Optionally, the terminal performs boundary refinement, hole filling, and topology preservation processing in parallel for each feature region through a mask processing module, obtaining a boundary refinement mask, a hole filling mask, and a topology preservation mask, respectively. For example, branch A (boundary refinement branch): uses a boundary-aware convolutional network to enhance the features of the mask boundary region, generating a boundary refinement mask. M_bd. Branch B (Hole-filling branch): Employs a morphological hole detection and filling algorithm to identify and fill the hole regions inside the mask, generating a hole-filling mask. M_hole. Branch C (Topology Preservation Branch): Employing connected component analysis and skeleton extraction algorithms, it identifies broken regions in the mask and connects them appropriately, while removing overly small isolated regions to generate a topology-preserving mask. M_top.

[0111] Step S306: Based on at least one of the boundary refinement mask, hole-filling mask, and topology-preserving mask, obtain the refinement mask update amount.

[0112] Optionally, the terminal fuses the boundary refinement mask, hole-filling mask, and topology-preserving mask output from the three branches through a mask processing module to obtain the refined mask update amount at step t. M_t, as shown in formula (1).

[0113] M_t = w_bd· M_bd + w_hole· M_hole + w_top· M_top formula (1)

[0114] In the formula, w_bd, w_hole, and w_top are learnable fusion weight coefficients.

[0115] Step S308: Determine the target segmentation mask set based on the refined mask update amount and the initial segmentation mask set.

[0116] Optionally, the terminal updates the fine-tuning mask based on formula (2) through the mask processing module. M_t updates the initial segmentation mask set and performs boundary smoothing on the updated initial segmentation mask set M_t.

[0117] M_t=M_{t-1}+ M_t formula (2)

[0118] Furthermore, the terminal calculates the refinement quality evaluation index Q_t, which includes the comprehensive score of three sub-indicators: boundary alignment, region integrity, and mask smoothness. The iteration terminates early when Q_t ≥ Q_threshold; otherwise, it continues for t+1 iterations. After T iterations or early termination, the refined target segmentation mask set M_ref and the corresponding refined confidence score set C_ref are output.

[0119] In this embodiment, a T-step iterative refinement strategy is proposed. Each refinement step enhances the mask from different dimensions through three parallel branches (boundary refinement, hole filling, and topology preservation), and the contribution of each branch is dynamically adjusted using learnable weight coefficients. A refinement quality evaluation index Q_t is introduced, allowing the refinement process to terminate early once the mask quality reaches the expected level, avoiding unnecessary computational overhead. This mechanism enables the segmentation mask quality to increase with the number of iterations, achieving a dynamic balance between computational accuracy and inference efficiency.

[0120] In one exemplary embodiment, such as Figure 4 As shown, the inverse correction of the segmentation mask by the classification result includes steps S402 to S408. Wherein:

[0121] Step S402: If the classification confidence of the defect prediction classification corresponding to any defect instance in the defect prediction classification set is lower than the preset classification confidence threshold, determine the initial segmentation mask corresponding to the defect instance from the initial segmentation mask set.

[0122] Optionally, the classification confidence of the defect prediction classification corresponding to any defect instance in the defect prediction classification set is lower than a preset classification confidence threshold. In the case of _c, such as The value of _c ranges from 0.6 to 0.8. If the classification confidence of the defect prediction classification (initially classified as "scratch") of a certain defect instance is 0.52, which is lower than 0.6, the terminal determines that the segmentation mask quality of the instance is poor and triggers the reverse correction mechanism.

[0123] Step S404: Based on the initial segmentation mask and feature map, a new feature map is obtained.

[0124] Optionally, the reverse correction mechanism includes: the terminal determining the initial segmentation mask corresponding to the defect instance from the initial segmentation mask set; extracting local features of the region corresponding to the initial segmentation mask from the feature map F_co (e.g., expanding the boundary of the region of interest (ROI) by 20%) to obtain a new feature map.

[0125] Step S406: The new feature map is processed by the target quantity layer decoder to obtain a new query vector for the corresponding layer; the new defect prediction classification for the corresponding layer is obtained by the classification head connected in parallel with the target quantity layer decoder based on the new query vector.

[0126] The target number is, for example, the last N layers, where N is set according to the actual situation, such as 2.

[0127] Optionally, the terminal inputs the newly extracted feature map into the last two layers of the segmentation head decoder for secondary inference to obtain new query vectors such as Q(n-1) and Q(n) for each layer.

[0128] Furthermore, by using the classification head connected in parallel with the target number layer decoder based on the new query vector, a new defect prediction classification for the corresponding layer is obtained. The new defect prediction classification of the last layer is used as the result of secondary inference. For example, if the instance is actually a "metallic foreign object" rather than a "scratch", the confidence level is increased to 0.89.

[0129] Step S408: The new query vector is processed by the mask processing module in the initial defect detection model to obtain a new segmentation mask.

[0130] Optionally, while obtaining a new defect prediction classification for the last layer based on the new query vector through the classification head connected in parallel with the last layer decoder, the terminal processes the new query vector through the mask processing module in the initial defect detection model to obtain a new segmentation mask, such as a more compact circular region (consistent with the true shape of the metal foreign object).

[0131] In this embodiment, when the classification confidence is too low, the segmentation quality is automatically determined to be poor and local feature re-extraction and secondary segmentation inference are triggered, thereby realizing bidirectional information flow between segmentation and classification.

[0132] In one exemplary embodiment, such as Figure 5 As shown, the joint loss value is obtained based on the target segmentation mask set, the target query vector set, and the defect prediction classification set, including steps S502 to S506. Wherein:

[0133] Step S502: Determine the auxiliary loss function value based on the defect prediction classification set.

[0134] The defect prediction classification set includes the defect prediction classification set of the last layer, as well as the defect prediction classification sets of all other layers.

[0135] Optionally, for each layer, the terminal calculates the loss value between the defect prediction classification set (excluding the last layer) and the true category set of the first sample image using binary cross-entropy loss, thus obtaining the auxiliary loss function value for each layer except the last layer.

[0136] Step S504: Determine the value of the main loss function based on the segmentation mask set, the target query vector set, and the defect prediction classification set.

[0137] Optionally, the terminal determines the main loss function value of the last layer based on the segmentation mask set, the target query vector set, and the defect prediction classification set of the last layer.

[0138] Step S506: Determine the joint loss value based on the auxiliary loss function value and the main loss function value.

[0139] Optionally, the terminal performs a weighted summation of the auxiliary loss function values ​​of each layer and the main loss function values ​​of the last layer to obtain the joint loss value.

[0140] In this embodiment, by calculating the auxiliary loss function value of the intermediate layer, a direct classification supervision signal is provided to the intermediate layer decoder. When the gradient of the deep layer is backpropagated to the shallow layer, it is easy to decay. The auxiliary loss enables the shallow layer to obtain an effective updated gradient, which accelerates the convergence. At the same time, multi-layer supervision is equivalent to applying additional constraints to the detection model to prevent deep overfitting.

[0141] In one exemplary embodiment, such as Figure 6 As shown, the main loss function value is determined based on the segmentation mask set, the target query vector set, and the defect prediction classification set, including steps S602 to S608. Wherein:

[0142] Step S602: Determine the segmentation loss value based on the initial segmentation mask set and the real segmentation mask set of the first sample image.

[0143] The segmentation loss value is used to monitor the quality of the initial segmentation mask set.

[0144] Optionally, the terminal substitutes the initial segmentation mask set generated by the initial detection model and the real segmentation mask set of the first sample image into the segmentation loss function L_seg to determine the segmentation loss value.

[0145] Step S604: Determine the classification loss value based on the defect prediction classification set, the true category set of the first sample image, and the target query vector group.

[0146] The classification loss value is used to supervise the category prediction of each defective instance and to make the features of similar instances closer to each other and dissimilar instances farther apart.

[0147] Optionally, the terminal substitutes the defect prediction classification set, the true category set of the first sample image, and the target query vector set into the classification loss function L_cls to determine the classification loss value.

[0148] Step S606: Determine the consistency loss value based on the initial segmentation mask set and the target segmentation mask set.

[0149] The consistency loss value is used to force the initial segmentation mask set M_pred to converge towards the target segmentation mask set M_ref after multiple refinements.

[0150] Optionally, the terminal substitutes the initial segmentation mask set and the target segmentation mask set into formula (3) to determine the consistency loss value.

[0151] L_cons= Formula (3) is: ·‖M_pred-M_ref‖2

[0152] In the formula, |||2 represents the L2 norm, and L_cons represents the consistency loss function.

[0153] Step S608: Determine the main loss function value based on the consistency loss value, the segmentation loss value, and the classification loss value.

[0154] Optionally, the terminal substitutes the consistency loss value, the segmentation loss value, and the classification loss value into formula (4) to determine the main loss function value.

[0155] L_total = L_seg + L_cls + L_cons (Formula 4)

[0156] In the formula, L_total represents the main loss function, L_seg represents the segmentation loss function, L_cls represents the classification loss function, and L_cons represents the consistency loss function.

[0157] In this embodiment, by introducing a consistency loss L_cons while calculating the segmentation loss function and the classification loss function, a constraint relationship is established between the initial segmentation mask and the refined mask. This enables the trained model to output more accurate masks and classification results during inference.

[0158] In one exemplary embodiment, such as Figure 7 As shown, based on the initial segmentation mask set and the true segmentation mask set of the first sample image, the segmentation loss value is determined, including steps S702 to S706. Wherein:

[0159] Step S702: Determine the mask segmentation loss value based on the initial segmentation mask set and the real segmentation mask set of the first sample image.

[0160] Optionally, the terminal substitutes the initial segmentation mask set M_pred and the real segmentation mask set M_gt of the first sample image into formula (5) to determine the mask segmentation loss value.

[0161] Formula (5)

[0162] In the formula, It is the probability that the model predicts the i-th pixel as a defect. The label represents the true label of the i-th pixel, and N represents the total number of pixels in the image. Represents the smoothing term. This represents the masking segmentation loss function.

[0163] Step S704: Determine the pixel segmentation loss value based on the pixel binary classification of the initial segmentation mask set and the pixel binary classification of the real segmentation mask set.

[0164] Pixel binary classification refers to whether the pixel is a defect or part of the background.

[0165] Optionally, the terminal substitutes the pixel binary classification of the initial segmentation mask set and the pixel binary classification of the real segmentation mask set into the focal loss, as shown in formula (6), to determine the pixel segmentation loss value.

[0166] Formula (6)

[0167] In the formula, This represents the model's predicted probability for the correct class. For a binary classification task for each pixel: if the true label is 1 (positive sample), then... (The probability that the model predicts a positive result); if the true label is 0 (a negative sample), then... For multi-classification tasks, Take the probability value corresponding to the true class index in the model output. Value range: (0,1), the closer to 1, the more accurate the prediction. γ represents the balance factor used to adjust the weights of positive and negative samples; γ represents the focusing parameter used to control the degree of weight decay for easily separated samples. log(p_t) represents the cross-entropy loss component, when... The smaller the value, the larger its absolute value, and the stronger the punishment.

[0168] Focal Loss uses (1-pt) γ The method dynamically reduces the loss weight of easily separable samples, causing the model to focus more on difficult-to-separate samples (such as defect boundaries, small targets, and low-contrast regions) during training, thereby improving the accuracy of segmentation and detection. In industrial defect detection, background pixels far outnumber defect pixels, and Focal Loss can effectively alleviate the problem of extreme class imbalance.

[0169] Step S706: Determine the segmentation loss value based on the mask segmentation loss value and the pixel segmentation loss value.

[0170] Optionally, the terminal substitutes the mask segmentation loss value and the pixel segmentation loss value into formula (7) to determine the segmentation loss value.

[0171] L_seg= ·L_dice+ ·L_focal formula (7)

[0172] In this embodiment, the Dice loss guarantee defect as a whole is detected, and the Focal loss guarantee defect at its edge is detected. The two complement each other and avoid the shortcomings of a single loss.

[0173] In one exemplary embodiment, such as Figure 8 As shown, based on the defect prediction classification set, the true category set of the first sample image, and the target query vector group, the classification loss value is determined, including steps S802 to S806. Wherein:

[0174] Step S802: Determine the cross-entropy loss value based on the defect prediction classification set and the true category set of the first sample image.

[0175] Optionally, the terminal substitutes the defect prediction classification in the last layer defect prediction classification set and the real category in the real category set of the first sample image into the cross-entropy loss function, as shown in formula (8), to determine the cross-entropy loss value.

[0176] … Formula (8)

[0177] In the formula, L_cross_entropy represents the cross-entropy loss value. One-hot encoding representing the actual label; This represents the probability of the c-th class output by the classification header, where C represents the total number of defect categories.

[0178] Step S804: Determine the triplet loss value based on the target query vector group.

[0179] Optionally, the terminal randomly selects anchor points, positive samples (of the same class), and negative samples (of different classes) from the target query vector group, and substitutes them into formula (9) to calculate the triplet loss value. This helps the classification head distinguish defect types.

[0180] L_triplet=max(0,d(anchor,positive)-d(anchor,negative)+margin) Formula (9)

[0181] In the formula, d is the Euclidean distance or cosine distance, margin is a preset interval, and L_triplet represents the triplet loss function.

[0182] Step S806: Determine the classification loss value based on the cross-entropy loss value and the triplet loss value.

[0183] Optionally, the terminal substitutes the cross-entropy loss value and the triplet loss value into formula (10) to obtain the classification loss value.

[0184] L_cls= ·L_cross_entropy+ ·L_triplet formula (10)

[0185] In this embodiment, cross-entropy ensures classification accuracy, while triples enhance feature discriminative power; the two complement each other. They share the same set of query vectors, and through joint optimization, the model simultaneously possesses the ability to classify correctly and distinguish features.

[0186] In an exemplary embodiment, obtaining a feature map of a first sample image includes: obtaining an original sample image and preprocessing the original sample image to obtain a first sample image; determining a multi-scale image based on the first sample image; and determining a feature map of the first sample image based on the multi-scale image.

[0187] Optionally, the terminal acquires the raw sample images captured by the industrial camera, performs preprocessing operations including denoising, normalization, and contrast enhancement, to obtain a first sample image. A multi-scale image pyramid is constructed based on the first sample image, the image pyramid containing at least three scale levels: the original resolution scale... Downsampling 1 / 2 scale and downsampling 1 / 4 scale The terminal inputs the multi-scale image pyramid into the cross-scale feature collaborative extraction module to generate feature maps, including: respectively... , , Independent convolutional feature extraction is performed on images at three scale levels to obtain initial feature maps corresponding to each scale. , and Low-scale feature maps Bilinear upsampling to With the same spatial resolution, After channel splicing, it is generated through the attention fusion layer. The process can be shown in formula (11).

[0188] =Attention(Concat(Up( ), )) Formula (11)

[0189] Will Further upsampling to With the same spatial resolution, After channel splicing, the feature map F_co is generated through the attention fusion layer, as shown in formula (12).

[0190] F_co = Attention(Concat(Up( ), )) Formula (12)

[0191] The attention fusion layer adopts a self-attention mechanism to model the cross-scale dependency relationship of the spliced ​​multi-scale features. The calculation formula is shown in formula (13).

[0192] Attention(Q,K,V)=softmax(QK^T / √d)V Formula (13)

[0193] In this context, Q, K, and V all originate from the concatenated features, and d represents the feature dimension.

[0194] Optionally, an industrial camera is installed on the PCB production line, 30cm from the PCB surface, using a ring-shaped white LED light source. When the PCB is conveyed to the center of the camera's field of view, a picture is taken, capturing an RGB three-channel image to obtain the raw sample image. The camera transmits the raw sample image to a terminal, which performs preprocessing operations such as denoising and normalization on the acquired raw sample image, such as using a 3×3 Gaussian kernel for image denoising. =0.8); linearly normalize the pixel values ​​from [0,255] to the [0,1] interval; use adaptive histogram equalization to enhance contrast, with parameters set to clipLimit=2.0 and tileGridSize=(8,8), to obtain the first sample image. The terminal constructs a three-scale image pyramid based on the first sample image: Maintain the original resolution, for example, 2448×2048. Bilinear downsampling to 1224×1024 Further downsample to 612×512. , , The inputs are fed into the cross-scale feature collaborative extraction module. Convolutional feature extraction uses ResNet-50 as the backbone network to extract initial feature maps for each scale. (C=2048, H=77, W=61) (C=1024, H=38, W=30) (C=512, H=19, W=15).

[0195] Will Bilinear upsampling to With the same spatial resolution (38×30), and After channel splicing, the input is fed into the attention fusion layer to generate The attention fusion layer employs a multi-head self-attention mechanism (8 heads) to model the dependencies between the spatial locations of the concatenated features. Further upsampling to At the same resolution (77×61), and After channel concatenation, the feature map F_co is generated by passing it through an attention fusion layer, with dimensions C=2048, H=77, and W=61.

[0196] In this embodiment, a hierarchical cross-scale attention fusion mechanism is introduced. First, low-scale features (corresponding to a large field of view and low resolution) are upsampled and fused with mid-scale features. Then, the fused features are further upsampled and fused with high-scale features (corresponding to a small field of view and high resolution), forming a bottom-up progressive fusion path. Each fusion level uses a self-attention mechanism to perform correlation modeling on features from different scales, enabling the model to simultaneously perceive the fine texture of small defects and the macroscopic structure of large defects, thereby effectively solving the problem of collaborative perception of multi-scale defects.

[0197] In one exemplary embodiment, such as Figure 9 As shown, the incremental learning and target defect detection model update includes steps S902 to S906. Wherein:

[0198] Step S902: When the second sample image corresponding to the new defect type is obtained, the second sample image is input into the target defect detection model, and the sample query vector corresponding to the second sample image output by the target defect detection model is received.

[0199] Optionally, when a new defect type is detected, the terminal acquires N second sample images corresponding to the new defect type, where N is at least 5. The terminal extracts the sample query vector for each sample using the target defect detection model, such as inputting the second sample image into the target defect detection model and receiving the sample query vector corresponding to the second sample image output by the target defect detection model.

[0200] Step S804: Based on each sample query vector, determine a new sample query vector; add the new sample query vector to the category library.

[0201] The category library stores query vectors for known defect types. The query vector here is a prototype query vector, which is obtained by averaging the query vectors of all defect instances in that category.

[0202] Optionally, the terminal averages the query vectors of each sample to determine a new query vector. _new. The terminal will send the new sample query vector. _new is added to the category library, and the new sample query vector is recorded. The distance between _new and the existing prototype query vector.

[0203] Step S906: Fine-tune the target defect detection model using the updated category library through knowledge distillation.

[0204] Optionally, a knowledge distillation strategy is used during incremental learning to maintain the model's ability to identify known defect types. The distillation loss L_distill is defined as the KL divergence between the output distribution of the new model for old category samples and the output distribution of the old model, as shown in formula (14).

[0205] L_distill=KL(p_old∥p_new) Formula (14)

[0206] Where p_old is the output probability distribution of the old model (the object detection model before fine-tuning), and p_new is the output probability distribution of the new model (the object detection model after fine-tuning).

[0207] Furthermore, when the category library reaches the preset capacity limit, a prototype compression strategy is executed: prototype query vectors with similarity higher than the similarity threshold are merged based on the similarity between prototype query vectors, and the merged new prototype query vector is a weighted average of the two.

[0208] In this embodiment, by using a category library and knowledge distillation strategy, it is not necessary to retrain the entire target detection model when adding new defect types, thereby improving the update efficiency of the target detection model and significantly reducing the model deployment cost when changing production lines.

[0209] In one embodiment, a defect detection method is also provided. This embodiment illustrates the method by applying it to a terminal. It is understood that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes: acquiring a target image to be detected; inputting the target image to be detected into a target defect detection model in any of the above embodiments, and acquiring the instance segmentation mask, instance prediction category, and classification confidence corresponding to the instance prediction category for each defect instance output by the target defect detection model.

[0210] Optionally, taking PCB surface defect detection as an example, an industrial camera is installed on the PCB production line, 30cm away from the PCB surface, using a ring-shaped white LED light source. When the PCB is transported to the center of the camera's field of view, a picture is taken, capturing an RGB three-channel image with a resolution of 2448×2048 pixels.

[0211] The following preprocessing operations were performed on the acquired raw images: denoising was performed on the images using a 3×3 Gaussian kernel. =0.8); linearly normalize pixel values ​​from [0,255] to the [0,1] interval; use adaptive histogram equalization to enhance contrast, with parameters set to clipLimit=2.0 and tileGridSize=(8,8). Construct a three-scale image pyramid based on the preprocessed image: Maintain the original resolution of 2448×2048. Bilinear downsampling to 1224×1024 Further downsampling to 612×512.

[0212] Will , , The inputs are fed into the cross-scale feature collaborative extraction module. Convolutional feature extraction uses ResNet-50 as the backbone network to extract initial feature maps for each scale. (C=2048, H=77, W=61) (C=1024, H=38, W=30) (C=512, H=19, W=15). [The rest of the text appears to be incomplete and requires further context.] Bilinear upsampling to With the same spatial resolution (38×30), and After channel splicing, the input is fed into the attention fusion layer to generate The attention fusion layer employs a multi-head self-attention mechanism (8 heads) to model the dependencies between the spatial locations of the concatenated features. Further upsampling to At the same resolution (77×61), and After channel concatenation, the feature map F_co is generated by passing it through an attention fusion layer, with dimensions C=2048, H=77, and W=61.

[0213] F_co is input into the Transformer instance segmentation head. The instance segmentation head adopts a standard architecture of 6 layers of encoder and 6 layers of decoder, and sets 100 learnable instance query vectors. Through cross-attention interaction between the encoder and decoder, the mask parameters and class embedding vectors of up to 50 defect instances are decoded in parallel. The initial segmentation mask set M_init is output, including instances such as cold solder joint defects (12 in total), solder ball defects (8 in total), scratch defects (3 in total), and foreign object defects (2 in total), and each mask is accompanied by an initial confidence score.

[0214] The M_init is input into the progressive mask refinement module, with a maximum iteration count T=3 and a refinement quality threshold Q_threshold=0.92. After the first iteration, the boundary refinement branch identifies and corrects the jagged edges of the solder ball defect mask; the void filling branch fills two small voids in the cold solder defect mask; and the topology preservation branch merges two breaks in the scratch defect caused by insufficient contrast. After the first step, Q=0.89, which does not reach the threshold, so iteration continues. After the second iteration, the overall Q=0.94, which exceeds the threshold Q_threshold=0.92, so the iteration is terminated early, and the refined mask M_ref is output.

[0215] In each decoding layer of the instance segmentation head, a classification branch is connected in parallel. This classification branch consists of two fully connected layers (256 hidden layers) and a Softmax output layer, outputting the probability distributions of seven preset categories of PCB defects (cold solder joints, solder balls, scratches, foreign objects, short circuits, open circuits, and others). Weighting coefficients are set in the joint loss function. =0.5, =0.5, =0.6, =0.4, =0.3, classification confidence threshold _c=0.7. During training, L_seg and L_cls are jointly optimized using gradient descent, and L_cons ensures that the segmentation mask and the refinement mask tend to be consistent.

[0216] In PCB board inspection, a new defect type—"copper slag residue"—was encountered for the first time, with only 8 images as the initial sample. An incremental learning process was executed: the class embedding vectors of the 8 samples were extracted, and the prototype vector was calculated. _new. Calculated, The highest cosine similarity between _new and the existing 7 prototype classes is 0.52 (with the "foreign object" class), which is below the merging threshold. _sim=0.85, indicating a new type. `_new` was added to the prototype memory, expanding its capacity from 7 to 8. Simultaneously, a knowledge distillation strategy was employed to fine-tune the model: the parameters of the first four layers of the old model were frozen, and only the last two layers and the classification head were fine-tuned, with a distillation loss of L_distill=0.12. The entire incremental learning process was significantly reduced in time.

[0217] Final output inspection results: 12 cold solder joint defects (average confidence level 0.96), 8 solder ball defects (average confidence level 0.94), 3 scratch defects (average confidence level 0.91), 2 foreign object defects (average confidence level 0.89), and 8 copper slag residues (average confidence level 0.87). Each defect outputs a corresponding instance mask (represented as a binary image) and confidence score.

[0218] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0219] Based on the same inventive concept, this application also provides a defect detection model training device for implementing the defect detection model training method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more defect detection model training device embodiments provided below can be found in the limitations of the defect detection model training method described above, and will not be repeated here.

[0220] In one exemplary embodiment, such as Figure 10 As shown, a defect detection model training device is provided, including: a first acquisition module 1001, a segmentation head 1002, a classification head 1003 connected in parallel with the segmentation head, and a calculation and fine-tuning module 1004, wherein:

[0221] The first acquisition module 1001 is used to acquire the feature map of the first sample image.

[0222] Segmentation head 1002 is used to perform segmentation masking on the feature map to obtain the target segmentation mask set and target query vector group of defect instances.

[0223] The classification head 1003, which is connected in parallel with the segmentation head, is used to perform defect prediction classification and obtain a defect prediction classification set of defect instances.

[0224] The fine-tuning module 1004 is used to calculate the joint loss value based on the target segmentation mask set, the target query vector set, and the defect prediction classification set, and to adjust the parameters in the initial defect detection model based on the joint loss value to obtain the target defect detection model.

[0225] In an exemplary embodiment, the current layer decoder of the segmentation head 1002 is used to perform cross-attention interaction with the feature map based on the historical query vector group output by the previous layer decoder of the current layer to obtain a new query vector group for the current layer.

[0226] The classification head 1003, which is connected in parallel with the current layer decoder, is used to obtain the defect prediction classification set of the current layer based on the corresponding new query vector set; wherein, while the next layer decoder of the current layer executes the new query vector set output by the current layer decoder as the historical query vector set, the classification head connected in parallel with the current layer decoder executes the corresponding new query vector set.

[0227] The determination module is used to select the new query vector group of the last layer as the target query vector group until the current layer is the last layer.

[0228] The mask processing module of the segmentation head 1002 is used to obtain the target segmentation mask set based on the target query vector group.

[0229] In an exemplary embodiment, the mask processing module of the segmentation head 1002 is further configured to obtain an initial segmentation mask set based on the target query vector group; and to perform iterative mask refinement on the initial segmentation mask set to obtain a target segmentation mask set.

[0230] In an exemplary embodiment, the mask processing module of the segmentation head 1002 is further configured to: obtain the feature region corresponding to each mask in the initial segmentation mask set according to the initial segmentation mask set and the feature map; perform at least one of boundary refinement, hole filling and topology preservation processing on each feature region to obtain at least one of boundary refinement mask, hole filling mask and topology preservation mask respectively; obtain the refined mask update amount based on at least one of boundary refinement mask, hole filling mask and topology preservation mask; and determine the target segmentation mask set based on the refined mask update amount and the initial segmentation mask set.

[0231] In an exemplary embodiment, a defect detection model training apparatus further includes: a backcorrection module, configured to: determine an initial segmentation mask corresponding to a defect instance from an initial segmentation mask set when the classification confidence of the defect prediction classification corresponding to any defect instance in the defect prediction classification set is lower than a preset classification confidence threshold; obtain a new feature map based on the initial segmentation mask and the feature map; process the new feature map through a target number layer decoder to obtain a new query vector for the corresponding layer; obtain a new defect prediction classification for the corresponding layer based on the new query vector through a classification head connected in parallel with the target number layer decoder; and process the new query vector through a mask processing module in the initial defect detection model to obtain a new segmentation mask.

[0232] In an exemplary embodiment, the fine-tuning module 1004 is further configured to determine an auxiliary loss function value based on the defect prediction classification set; determine a main loss function value based on the segmentation mask set, the target query vector set, and the defect prediction classification set; and determine a joint loss value based on the auxiliary loss function and the main loss function.

[0233] In an exemplary embodiment, the fine-tuning module 1004 is further configured to: determine a segmentation loss value based on an initial segmentation mask set and a true segmentation mask set of the first sample image; determine a classification loss value based on a defect prediction classification set, a true category set of the first sample image, and a target query vector set; determine a consistency loss value based on an initial segmentation mask set and a target segmentation mask set; and determine a main loss function value based on the consistency loss value, the segmentation loss value, and the classification loss value.

[0234] In an exemplary embodiment, the fine-tuning module 1004 is further configured to determine a mask segmentation loss value based on the initial segmentation mask set and the real segmentation mask set of the first sample image; determine a pixel segmentation loss value based on the pixel binary classification of the initial segmentation mask set and the pixel binary classification of the real segmentation mask set; and determine a segmentation loss value based on the mask segmentation loss value and the pixel segmentation loss value.

[0235] In an exemplary embodiment, the fine-tuning module 1004 is further configured to determine the cross-entropy loss value based on the defect prediction classification set and the true category set of the first sample image; determine the triplet loss value based on the target query vector set; and determine the classification loss value based on the cross-entropy loss value and the triplet loss value.

[0236] In an exemplary embodiment, the first acquisition module 1001 is further configured to acquire an original sample image, preprocess the original sample image to obtain a first sample image, determine a multi-scale image based on the first sample image, and determine a feature map of the first sample image based on the multi-scale image.

[0237] In an exemplary embodiment, a defect detection model training device further includes: an incremental update module, configured to, when a second sample image corresponding to a new defect type is acquired, input the second sample image into a target defect detection model, receive a sample query vector corresponding to the second sample image output by the target defect detection model; determine a new sample query vector based on each sample query vector; add the new sample query vector to a category library; wherein the category library stores query vectors of known defect types; and fine-tune the target defect detection model using the updated category library through knowledge distillation.

[0238] In one exemplary embodiment, such as Figure 11 As shown, a defect detection device is provided, including: a second acquisition module 1101 and a detection module 1102, wherein:

[0239] The second acquisition module 1101 is used to acquire the target image to be detected.

[0240] The detection module 1102 is used to input the target image to be detected into the target defect detection model as described in any of the above embodiments, and to obtain the instance segmentation mask, instance prediction category, and classification confidence corresponding to the instance prediction category for each defect instance output by the target defect detection model.

[0241] The aforementioned defect detection model training device and each module in the defect detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0242] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores defect data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a defect detection model training method.

[0243] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0244] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0245] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0246] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0247] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0248] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0249] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A defect detection model training method, characterized in that, The method includes: Obtain the feature map of the first sample image; The feature map is segmented using the segmentation head in the initial defect detection model, and defect prediction classification is performed simultaneously using the classification head connected in parallel with the segmentation head. This yields the target segmentation mask set and target query vector set of the defect instances output by the segmentation head, as well as the defect prediction classification set output by the classification head. Based on the target segmentation mask set, the target query vector set, and the defect prediction classification set, a joint loss value is obtained. The parameters in the initial defect detection model are adjusted based on the joint loss value to obtain the target defect detection model.

2. The method of claim 1, wherein, The process involves performing segmentation masking on the feature map using the segmentation head in the initial defect detection model, and simultaneously performing defect prediction classification using a classification head connected in parallel with the segmentation head. This yields a target segmentation mask set and a target query vector set for the defect instances output by the segmentation head, as well as a defect prediction classification set output by the classification head, including: The current layer decoder of the segmentation head performs cross-attention interaction with the feature map based on the historical query vector group output by the previous layer decoder of the current layer to obtain a new query vector group for the current layer. The current layer's defect prediction classification set is obtained by using the classification head connected in parallel with the current layer decoder based on the corresponding new query vector set; wherein, while the next layer decoder of the current layer executes the new query vector set output by the current layer decoder as the historical query vector set, the classification head connected in parallel with the current layer decoder executes the corresponding new query vector set. Until the current layer is the last layer, the new query vector group of the last layer is used as the target query vector group; the target segmentation mask set is obtained by the mask processing module of the segmentation head according to the target query vector group.

3. The method according to claim 2, characterized in that, The step of obtaining the target segmentation mask set based on the target query vector group includes: Based on the target query vector group, an initial segmentation mask set is obtained; The initial segmentation mask set is iteratively refined to obtain the target segmentation mask set.

4. The method according to claim 3, characterized in that, The iterative mask refinement of the initial segmentation mask set to obtain the target segmentation mask set includes: Based on the initial segmentation mask set and the feature map, the feature region corresponding to each mask in the initial segmentation mask set is obtained; For each of the aforementioned feature regions, at least one of boundary refinement, hole filling, and topology preservation processing is performed to obtain at least one of the boundary refinement mask, hole filling mask, and topology preservation mask, respectively. Based on at least one of the boundary refinement mask, the hole filling mask, and the topology preservation mask, the refinement mask update amount is obtained; Based on the refined mask update amount and the initial segmentation mask set, the target segmentation mask set is determined.

5. The method according to claim 3, characterized in that, The method further includes: If the classification confidence of the defect prediction classification corresponding to any defect instance in the defect prediction classification set is lower than a preset classification confidence threshold, the initial segmentation mask corresponding to the defect instance is determined from the initial segmentation mask set. Based on the initial segmentation mask and the feature map, a new feature map is obtained; The new feature map is processed by the target quantity layer decoder to obtain a new query vector for the corresponding layer; Based on the new query vector, the classification head connected in parallel with the target number layer decoder obtains a new defect prediction classification for the corresponding layer. The new query vector is processed by the mask processing module in the initial defect detection model to obtain a new segmentation mask.

6. The method according to claim 3, characterized in that, The step of obtaining the joint loss value based on the target segmentation mask set, the target query vector set, and the defect prediction classification set includes: Based on the aforementioned defect prediction classification set, determine the auxiliary loss function value; The main loss function value is determined based on the segmentation mask set, the target query vector set, and the defect prediction classification set; The joint loss value is determined based on the auxiliary loss function value and the main loss function value.

7. The method according to claim 6, characterized in that, The step of determining the main loss function value based on the segmentation mask set, the target query vector set, and the defect prediction classification set includes: Based on the initial segmentation mask set and the true segmentation mask set of the first sample image, the segmentation loss value is determined; Based on the defect prediction classification set, the true category set of the first sample image, and the target query vector group, the classification loss value is determined; Based on the initial segmentation mask set and the target segmentation mask set, the consistency loss value is determined; The main loss function value is determined based on the consistency loss value, the segmentation loss value, and the classification loss value.

8. The method according to claim 7, characterized in that, The step of determining the segmentation loss value based on the initial segmentation mask set and the true segmentation mask set of the first sample image includes: Based on the initial segmentation mask set and the real segmentation mask set of the first sample image, the mask segmentation loss value is determined; The pixel segmentation loss value is determined based on the pixel binary classification of the initial segmentation mask set and the pixel binary classification of the real segmentation mask set. The segmentation loss value is determined based on the mask segmentation loss value and the pixel segmentation loss value; The step of determining the classification loss value based on the defect prediction classification set, the true category set of the first sample image, and the target query vector group includes: Based on the defect prediction classification set and the true category set of the first sample image, the cross-entropy loss value is determined; Based on the target query vector set, determine the triplet loss value; The classification loss value is determined based on the cross-entropy loss value and the triplet loss value.

9. The method according to claim 1, characterized in that, The process of obtaining the feature map of the first sample image includes: Acquire the original sample image and preprocess the original sample image to obtain the first sample image; Based on the first sample image, determine the multi-scale image; Based on the multi-scale image, the feature map of the first sample image is determined.

10. The method according to claim 1, characterized in that, The method further includes: When a second sample image corresponding to a new defect type is obtained, the second sample image is input into the target defect detection model, and the sample query vector corresponding to the second sample image output by the target defect detection model is received. Based on the aforementioned sample query vectors, a new sample query vector is determined; The new sample query vector is added to the category library; wherein the category library stores query vectors of known defect types; The target defect detection model is fine-tuned using knowledge distillation with the updated category library.

Images