Defect detection method and device based on defect detection model, program product

Abstract

The defect detection method and device based on a defect detection model, and the computer program product, the defect detection method comprising: acquiring an image of an object to be detected; inputting the image of the object to be detected into an encoder for multi-scale feature extraction to obtain a first feature map of multiple layers with different resolutions; inputting the first feature map of multiple layers with different resolutions into a decoder for fusion, and performing dimensionality lifting and information recovery using the fused features to obtain a fused feature map; using the fused feature map by a classification head module to perform pixel-level classification prediction to obtain a defect segmentation result, the defect segmentation result comprising a classification result of whether each pixel in the image of the object to be detected belongs to a defect pixel; and determining a defect region based on the defect pixel. Since the defect detection model is composed of an encoder, a decoder and a classification head module, the structure is relatively simple, the model can be lightened, the calculation speed is improved, and the time consumption of the hyperparameter optimization process is reduced.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, specifically to a defect detection method and apparatus based on a defect detection model, and a computer program product. Background Technology

[0002] Deep learning technology has been increasingly widely applied in image processing tasks such as defect detection. For example, in the deployment of industrial defect detection, neural network-based defect segmentation technology has been widely adopted in the market due to its advantages such as pixel-by-pixel fine localization, strong scene adaptability, and high accuracy, and is often the preferred technology for industrial inspection. When training a deep learning model, it is necessary to set hyperparameters such as batch size, learning rate, optimizer, regularization, and data augmentation methods. With the continuous development of technology, determining the appropriate or optimal combination of hyperparameters has become a key factor restricting the performance improvement of deep learning models. Inappropriate hyperparameter settings can lead to long training times and poor detection results.

[0003] To obtain the optimal hyperparameter combination, hyperparameter optimization is required, typically using automatic hyperparameter search tools, including grid search, random search, and parameter space search based on Bayesian optimization. In practice, grid search and random search suffer from significant computational overhead for deep learning networks due to the high resolution of industrial images, resulting in excessively long optimization processes. Bayesian optimization continuously adjusts the hyperparameter search space using historical data, finding suitable hyperparameter combinations with relatively few attempts. However, Bayesian optimization uses surrogate models such as Gaussian process regression. In deep learning, there are often numerous hyperparameters requiring tuning, and finding an accurate approximation in a high-dimensional parameter space is difficult, leading to performance degradation. Furthermore, the objective function of deep learning models is usually non-convex, making it difficult to fit using surrogate models like Gaussian process regression, while directly using the deep learning model further increases the time consumption. Summary of the Invention

[0004] The main technical problem this invention addresses is how to reduce the time consumed in the hyperparameter optimization process.

[0005] According to a first aspect, one embodiment provides a defect detection method based on a defect detection model, the defect detection model including an encoder, a decoder, and a classification head module, the defect detection method comprising:

[0006] Acquire an image of the object being detected;

[0007] The image of the detected object is input into the encoder for multi-scale feature extraction to obtain a first feature map with multiple layers of different resolutions.

[0008] The first feature maps of different resolutions in the multiple layers are input into the decoder for fusion, and the fused features are used for dimensionality upscaling and information recovery to obtain a fused feature map;

[0009] The classification head module uses the fused feature map to perform pixel-level classification prediction to obtain defect segmentation results. The defect segmentation results include classification results of whether each pixel in the image of the detected object belongs to a defect pixel.

[0010] Defect regions are determined based on defective pixels.

[0011] According to a second aspect, one embodiment provides a defect detection device based on a defect detection model, the defect detection model including an encoder, a decoder, and a classification head module, the defect detection device comprising:

[0012] The image acquisition module is used to acquire images of the detected object;

[0013] The defect segmentation module is used for:

[0014] The image of the detected object is input into the encoder for multi-scale feature extraction to obtain a first feature map with multiple layers of different resolutions.

[0015] The first feature maps of different resolutions in the multiple layers are input into the decoder for fusion, and the fused features are used for dimensionality upscaling and information recovery to obtain a fused feature map;

[0016] The classification head module uses the fused feature map to perform pixel-level classification prediction to obtain defect segmentation results. The defect segmentation results include classification results of whether each pixel in the image of the detected object belongs to a defect pixel.

[0017] Defect regions are determined based on defective pixels.

[0018] According to a third aspect, one embodiment provides a computer program product including a computer program and / or instructions, which, when executed by a processor, implement the defect detection method described in the first aspect.

[0019] According to the defect detection method, apparatus, and computer program product based on the defect detection model described in the above embodiments, defect detection is performed based on a defect detection model including an encoder, a decoder, and a classification head module. First, the image of the object to be detected is input into the encoder for multi-scale feature extraction, obtaining multiple layers of first feature maps at different resolutions. These first feature maps are then input into the decoder for fusion. The fused features are then used for dimensionality upscaling and information recovery to obtain a fused feature map. Finally, the classification head module uses the fused feature map to perform pixel-level classification prediction to obtain the defect segmentation result. Since the defect detection model consists of an encoder, a decoder, and a classification head module, its structure is relatively simple, enabling lightweight model design, thereby improving computational speed and reducing the time spent on hyperparameter optimization. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall processing flow of a hyperparameter optimization method according to one embodiment.

[0021] Figure 2 A schematic diagram of the overall processing flow of a hyperparameter optimization method according to another embodiment;

[0022] Figure 3 Here are flowcharts of hyperparameter optimization methods for some embodiments;

[0023] Figure 4 A flowchart illustrating the training method of a defect segmentation model in some embodiments;

[0024] Figure 5 This is a flowchart of obtaining the first sample image set in some embodiments;

[0025] Figure 6 Here are flowcharts of defect detection methods for some embodiments;

[0026] Figure 7 This is a schematic diagram of the structure and loss function of the defect segmentation model in some embodiments;

[0027] Figure 8 A flowchart illustrating how, in some embodiments, an image of the object to be detected is input into a pre-trained defect segmentation model to obtain defect regions in the image of the object to be detected.

[0028] Figure 9 This is a schematic diagram of the structure and processing flow of the decoder in some embodiments;

[0029] Figure 10 The flowchart illustrates how, in some embodiments, first feature maps of different resolutions from multiple layers are input into a decoder for fusion, and the fused features are used for dimensionality upscaling and information recovery to obtain a fused feature map.

[0030] Figure 11This is a flowchart illustrating the process of using a first multi-class score map to enhance the spliced ​​feature map in some embodiments, resulting in a target-enhanced feature map.

[0031] Figure 12 This is a schematic diagram of the process of performing target enhancement on the spliced ​​feature map to obtain the target enhanced feature map in a specific implementation;

[0032] Figure 13 This is a schematic diagram of the structure and processing flow of the self-attention module in some embodiments;

[0033] Figure 14 The flowchart below shows how, in some embodiments, the stitched feature map and the target enhancement feature map are input into the self-attention module for processing by the self-attention mechanism to obtain the fused feature map.

[0034] Figure 15 This is a schematic diagram showing the result of defect detection of an image of an object using a defect segmentation model in one embodiment.

[0035] Figure 16 This is a schematic diagram of the structure of a hyperparameter optimization device according to some embodiments;

[0036] Figure 17 This is a schematic diagram of the structure of a defect detection device according to some embodiments. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0038] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0039] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0040] Currently, in industrial image defect segmentation methods, Bayesian optimization is typically used for automated hyperparameter optimization to obtain the optimal hyperparameter combination for training the defect segmentation model. Compared to grid search and random search, Bayesian optimization can find suitable hyperparameter combinations with relatively fewer attempts; however, it also has the following problems: First, the challenge of high-dimensional parameter space: In deep learning, there are many hyperparameters that need to be tuned. In high-dimensional parameter space, it is difficult for the Bayesian optimization process to find an accurate approximation of the surrogate model. If a deep learning model is used directly, especially for the deep learning models used in high-resolution images for industrial inspection, it will lead to excessive computational overhead and excessively long optimization time. Second, fitting non-convex objective functions: The simple surrogate models such as Gaussian process regression used in the Bayesian optimization process are difficult to fit the non-stationary and non-convex objective functions in deep learning, resulting in a decrease in optimization performance. Third, model structure design: In order to obtain more accurate optimization results, the Bayesian optimization process directly uses a defect segmentation model based on a deep learning network as a surrogate model. However, the lightweight structure design of the defect segmentation model is insufficient, and there is a lack of dedicated optimization design for the segmentation network, which leads to a large amount of computational resources and time required.

[0041] To address some problems existing in current technologies, this invention employs a novel solution, optimizing the design from both data and model perspectives. It can automatically optimize the model's hyperparameters to obtain the optimal hyperparameter combination, significantly reducing optimization time while improving model performance. This invention provides a hyperparameter optimization method based on active learning. Compared to similar methods currently available, the goal of this invention's hyperparameter optimization method is to achieve accurate and automated optimization of hyperparameters with lower processing time, obtaining the optimal hyperparameter combination for model training and improving the detection performance of industrial images. Specifically, the various embodiments of the present invention have some or all of the following advantages: (1) Accuracy: In the hyperparameter search process, in an active learning manner, difficult samples are used as surrogate samples for subsequent training, and the generation of surrogate samples is continuously iterated and optimized. The model can obtain the best hyperparameter optimization results in the most important and representative samples, while also reducing the sample size; (2) Efficiency: In some embodiments, from the perspective of data, based on the active learning method, key learning regions in the sample images are selected to form surrogate samples for hyperparameter optimization, and / or from the perspective of the model, the model is specially optimized and designed. The constructed model has fewer parameters and faster calculation speed, and can obtain the best hyperparameter combination in a short time; (3) Adaptability and scalability: It can adapt to different datasets and tasks, and can be used in good combination with other hyperparameter search methods to improve the optimization effect.

[0042] The following is a brief introduction to the concept of this invention. After analyzing and exploring the problem of excessively long hyperparameter optimization time in deep learning network models, the inventors found that the main influencing factors include the following two aspects: First, data: For deep networks, the network's running time is linearly positively correlated with the image resolution. Industrial inspection images have high resolution, requiring the network to spend more time on computation. At the same time, image samples have redundancy. For general samples or easily fitted samples, the conventionally used unoptimized hyperparameter combinations can also achieve good detection results for model training and prediction. However, the most representative important samples and difficult samples require the configuration of optimal hyperparameter combinations to improve the model's detection performance. Second, network structure: Insufficient lightweight design of the network structure and lack of dedicated optimization design result in high computational load and poor real-time performance, leading to excessively long optimization time when using hyperparameter search tools.

[0043] Therefore, based on the above two points of analysis, the inventors believe that the optimization design of data and models is crucial. This invention optimizes the design from both data and model perspectives. Based on active learning, it acquires challenging samples for hyperparameter optimization. In some embodiments, for defect detection tasks, the structure of the defect segmentation model is optimized to make the model lightweight. These optimizations significantly reduce the time spent on the hyperparameter optimization process, obtain the optimal hyperparameter combination, and improve the model's detection performance. The core idea is: based on active learning, select the most important and difficult samples that the model struggles to fit, greatly reducing the amount of sample data used in the hyperparameter optimization process, and achieving efficient and accurate acquisition of the optimal hyperparameter combination. Furthermore, in some embodiments, for defect detection tasks, since industrial images have high resolution and few and small defect regions, a portion of the image with defective regions is extracted and used as actual training sample images, significantly reducing the computational load of the model. In some embodiments, an optimized model structure is adopted, reducing the computational load and achieving excellent real-time performance. In some embodiments, the optimization of both data and model works together to significantly reduce the time spent on hyperparameter optimization, obtaining the optimal hyperparameter combination in a short time, thereby improving the detection performance of the defect segmentation model and reducing training time, achieving cost reduction and efficiency improvement.

[0044] Based on the hyperparameter optimization method of this invention, a complete image processing task is divided into two stages. The first stage involves using the hyperparameter optimization method of this invention to search for hyperparameters and obtain the optimal hyperparameter combination based on the optimization of data and / or the model. The second stage involves using the obtained optimal hyperparameter combination to complete high-quality training of the network model, improving detection performance, and finally using the trained network model to perform the corresponding image processing task. In the entire process, the first and second stages are executed sequentially. Only after the first stage is completed and the optimal hyperparameter combination is obtained can the training and detection process of the second stage be run, ultimately achieving performance improvement of the network model.

[0045] The technical solution of this invention is described below. Please refer to... Figure 1 , Figure 1 This is a schematic diagram of the overall processing flow of the hyperparameter optimization method proposed in this invention. The sample image set used includes an image set X and a corresponding annotation set Y. Each sample image in the image set X has a corresponding annotation in the annotation set Y. Figure 1The diagram illustrates several functional modules or program modules. F represents the deep learning model used to execute the target task (referred to as the "target task model"). The management module P implements some processing steps related to hyperparameters in the entire processing flow. Its main functions include two aspects: first, managing the hyperparameters of the target task model F and all its combinations, and providing the hyperparameter combination para to the target task model F for active learning training and experimentation; second, recording the performance of the target task model F under all hyperparameter combinations para, and finally selecting the hyperparameter combination corresponding to the best performance of the target task model F as the optimal hyperparameter combination best para. The management module P outputs each of the preset multiple hyperparameter combinations to the target task model F. The target task model F receives sample images and corresponding annotations (x, y, y) from the sample image set. train y train The model is trained using hyperparameter combinations `para` output by the management module P, employing an active learning approach and updating the sample image set based on the training progress. Next, at the end of training for each hyperparameter combination, the management module P performs a performance evaluation `eval` on the trained target task model F, recording the hyperparameter combination used and the performance metric for each iteration. Finally, after training trials for all hyperparameter combinations, the best hyperparameter combination is output as the optimal hyperparameter combination for the current task, completing the hyperparameter optimization process. The optimal hyperparameter combination is then used to perform final training on the target task model F, thus completing the training for executing the target task.

[0046] like Figure 1 As shown, in some embodiments, the sample image set is divided into a training set (x, y, z) according to a preset ratio. train y train ) and validation set (x val y val ), training set (x train y train ) is used for searching and experimenting with hyperparameter optimization, and the validation set (x val y val The hyperparameters (x) are used to select the hyperparameter combination. Specifically, the target task model F uses the training set (x) train y train Training is performed using an active learning approach, and the training set (x) is updated based on the training progress. train y train At the end of training for each hyperparameter combination, the management module P uses the validation set (x) on the trained target task model F. val y valThe system performs a performance evaluation using the `eval` function, recording the hyperparameter combination and performance metrics used each time. Finally, after training all hyperparameter combinations, it determines the optimal performance metrics, outputs the corresponding hyperparameter combination, and obtains the best hyperparameter combination.

[0047] Please refer to Figure 2 In some embodiments, for defect detection tasks (i.e., the target task model F is a defect segmentation model, the sample images include defect regions, the annotations are specifically defect segmentation annotation maps, the defect segmentation annotation maps have defect-marked regions, and the regions in the sample images corresponding to the defect-marked regions in the corresponding defect segmentation annotation maps are the defect regions), a preprocessing module Q is also designed. Its main function is to extract regions of interest from the sample image set for subsequent processing. In the preprocessing module Q, the original defect segmentation annotation maps in the original annotation set Y are cropped to extract a portion of the image that includes the defect-marked region, resulting in new defect segmentation annotation maps. These new defect segmentation annotation maps constitute a new annotation set Y1. For the original sample images in the original image set X, the regions corresponding to the cropped portions of the original defect segmentation annotation maps are extracted to obtain new sample images. These new sample images constitute a new image set X1. During hyperparameter optimization, the defect segmentation model F accepts sample images from a new image set X1, corresponding defect segmentation annotations from a new annotation set Y1, and the hyperparameter combination para output by the management module P. Based on active learning, the sample image set (X1, Y1) is updated according to the training status. At the end of training for each hyperparameter combination, the management module P performs a performance evaluation (eval) on the trained defect segmentation model F, recording the hyperparameter combination used and the defect segmentation performance for each step. Finally, after training trials for all hyperparameter combinations, the hyperparameter combination with the best defect segmentation performance is selected as the best hyperparameter combination for the current task. Then, the best hyperparameter combination is used to perform final defect segmentation training on the defect segmentation model F. During inference (i.e., when performing defect detection), the image of the object to be detected is input into the defect segmentation model F, and the defect segmentation model F outputs the defect segmentation result.

[0048] Similarly, the new sample image set (X1, Y1) obtained after preprocessing by the preprocessing module Q can also be divided into a training set (x1, Y1, and x2). train y train ) and validation set (x val y val ), training set (x train y train ) is used for searching and experimenting with hyperparameter optimization, and the validation set (x val y valThis is used to select the best combination of hyperparameters.

[0049] Please refer to Figure 3 The hyperparameter optimization method in some embodiments of the present invention specifically includes steps 1100 to 1300, which are described below.

[0050] Step 1100: Obtain the sample image set.

[0051] The sample image set includes multiple sample images and corresponding annotations. The content of the annotations varies depending on the task. For example, for object detection, the position and size of the bounding box of the object in the sample image can be annotated; for defect detection, the defect region in the sample image can be annotated.

[0052] Step 1200: For each of the preset multiple hyperparameter combinations, use that hyperparameter combination to train the target task model using the sample image set, and evaluate the performance index of the trained target task model when performing the target task; record the performance index of the target task model trained using each hyperparameter combination when performing the target task.

[0053] A target task model is a model for performing a target task, such as target detection or defect detection. Those skilled in the art can set the structure of the target task model and the performance indicators used for evaluation according to the task type and actual detection needs.

[0054] Hyperparameter combinations are preset and saved. For example, if the learning rate is 0.3 or 0.6, and the batch size is 30 or 80, then there are four possible hyperparameter combinations for [learning rate, batch size]: [0.3, 30], [0.3, 80], [0.6, 30], and [0.6, 80]. These combinations can be set using various hyperparameter search methods, such as grid search, random search, and Bayesian optimization-based search.

[0055] In deep learning training, sample image sets are reused multiple times. In this invention, during the training of the target task model using the sample image set for each hyperparameter combination, an active learning approach is used to update the sample image set based on the prediction results of the target task model. Specifically, if the prediction error of the target task model for any sample image is less than a second error threshold, the sample image and its corresponding annotation are discarded and will not participate in subsequent training rounds, effectively reducing the number of samples in the sample image set. The prediction error can be determined based on the difference between the target task model's prediction result for the sample image and its corresponding annotation.

[0056] Essentially, this approach uses active learning to select the most important and challenging samples for training and fitting, reducing the number of samples and accelerating the hyperparameter optimization search time. By training the model on the most important and challenging samples, it can efficiently and accurately obtain the optimal hyperparameter combination, thereby improving model performance.

[0057] In some embodiments, training the target task model using a sample image set and evaluating the performance metrics of the trained target task model when performing the target task includes: dividing the sample image set into a training set and a validation set according to a preset ratio (e.g., a training set: validation set ratio of 7:3); training the target task model using the training set; and evaluating the performance metrics of the trained target task model when performing the target task using the validation set after training. It can be understood that, in this active learning approach, the sample images and corresponding annotations in the training set are discarded, i.e., the training set is updated. After training, the parameters of the target task model are fixed, it receives sample images from the validation set as input, outputs prediction results, and evaluates the performance metrics of the trained target task model based on the prediction results and corresponding annotations.

[0058] Step 1300: Select the hyperparameter combination corresponding to the trained target task model with the best performance index as the final hyperparameter combination used to train the target task model. Subsequently, the target task model will use the best para as the training hyperparameter combination, which can complete high-quality training in a shorter time and obtain excellent performance.

[0059] Please refer to Figure 4 The hyperparameter optimization method of the present invention is applied to the defect detection task. The process of training the defect segmentation model includes steps 2100-2300, which are described in detail below.

[0060] Step 2100: Obtain the first sample image set.

[0061] The first sample image set includes multiple first sample images and corresponding defect segmentation annotations, which are used to indicate defect areas in the first sample images.

[0062] Defect segmentation annotations are typically defect segmentation annotation maps with the same resolution as the sample images. Each pixel in the defect segmentation annotation map indicates whether the corresponding pixel in the sample image belongs to a defect. If multiple types of defects exist, it can also indicate which type of defect the corresponding pixel in the sample image belongs to. Specifically, different pixel values ​​or colors can be used to mark the background and (different types of) defects. The pixels marked as defects form the defect marking region. Therefore, the region in the sample image corresponding to the defect marking region in the corresponding defect segmentation annotation map is the defect region. Based on this, the first sample image set includes multiple first sample images and corresponding first defect segmentation annotation maps. The first defect segmentation annotation map has defect marking regions, and the region in the first sample image corresponding to the defect marking region in the corresponding first defect segmentation annotation map is the defect region.

[0063] Step 2200: For each of the preset multiple hyperparameter combinations, use that hyperparameter combination to train the defect segmentation model using the first sample image set, and evaluate the defect segmentation performance of the trained defect segmentation model; record the defect segmentation performance of the defect segmentation model trained using each hyperparameter combination.

[0064] For setting the hyperparameter combination, please refer to step 120. Defect segmentation performance can be evaluated using metrics such as pixel segmentation accuracy and average IoU (Intersection over Union). Those skilled in the art can set the structure of the defect segmentation model and the metrics used for evaluation according to actual inspection needs.

[0065] During the training of the defect segmentation model using the first sample image set for each hyperparameter combination, if the defect segmentation error of the defect segmentation model for any first sample image is less than a first error threshold, then that first sample image and its corresponding defect segmentation label are discarded and will not participate in subsequent training rounds. The first sample image is input into the defect segmentation model to obtain the corresponding defect segmentation result. This result can be understood as the classification result of whether each pixel in the first sample image belongs to a defect pixel. The defect segmentation error can be determined based on the difference between the defect segmentation model's defect segmentation result for the first sample image and the corresponding defect segmentation label; for example, the number of incorrectly predicted pixels can be used as the defect segmentation error.

[0066] In some embodiments, training a defect segmentation model using a first sample image set and evaluating the defect segmentation performance of the trained model includes: dividing the first sample image set into a training set and a validation set according to a preset ratio (e.g., a training set: validation set ratio of 7:3); training the defect segmentation model using the training set; and evaluating the defect segmentation performance of the trained model using the validation set after training. In this case, the active learning approach discards the first sample image and its corresponding defect segmentation annotation in the training set, i.e., it updates the training set. After training, the parameters of the defect segmentation model are fixed, it receives the first sample image from the validation set as input, and outputs the defect segmentation result; the defect segmentation performance of the trained model is evaluated based on the defect segmentation result and the corresponding defect segmentation annotation.

[0067] Step 2300: Select the hyperparameter combination corresponding to the trained defect segmentation model with the best defect segmentation performance as the final hyperparameter combination best para. Using the final hyperparameter combination best para to train the defect segmentation model can complete high-quality training in a short time and obtain a trained defect segmentation model with excellent defect segmentation performance.

[0068] Please refer to Figure 5 In some embodiments, step 2100 includes steps 2110 to 2140, which will be described in detail below.

[0069] Step 2110: Obtain the original sample image set.

[0070] The original sample image set includes multiple original sample images and corresponding original defect segmentation annotation maps. The original defect segmentation annotation maps have defect-marked regions. The regions in the original sample images that correspond to the defect-marked regions in the corresponding original defect segmentation annotation maps are the defect regions.

[0071] The original sample images here can be images obtained by imaging the product using a high-resolution industrial camera, which have high resolution.

[0072] Step 2120: Extract a portion of the image including the defect mark region from the original defect segmentation annotation map to obtain the first defect segmentation annotation map; extract the region in the original sample image that corresponds to the first defect segmentation annotation map extracted from the original defect segmentation annotation map to obtain the first sample image corresponding to the first defect segmentation annotation map.

[0073] By cropping the original defect segmentation annotation map and the original sample image, the resolution of both can be reduced, thereby reducing the amount of computation and accelerating the training of the model. When cropping, it is only necessary to ensure that the first defect segmentation annotation map includes the defect marking region of the original defect segmentation annotation map (i.e., the region of interest of the original defect segmentation annotation map) and the first sample image includes the defect region of the original sample image (i.e., the region of interest of the original sample image). The specific size of the image to be cropped can be determined according to the processor's computing power, training efficiency requirements, etc.

[0074] In one embodiment, extracting a portion of the image including the defect marking region from the original defect segmentation annotation map specifically includes: determining the minimum bounding rectangle of the defect marking region, and extracting the image of the portion of the minimum bounding rectangle from the original defect segmentation annotation map.

[0075] Step 2130: Unify the resolution of all first sample images and first defect segmentation annotation maps. This can be achieved through interpolation.

[0076] Step 2140: The first sample image set is composed of all the first sample images and the corresponding first defect segmentation annotation map.

[0077] In one embodiment, when the defect segmentation model is finally trained in step 2300 using the final hyperparameter combination best para, the original sample image set is used.

[0078] Currently, in actual industrial product inspection, manufacturers generally use high-resolution industrial cameras to image products to ensure quality and maximize inspection accuracy. However, this results in large image resolutions, such as 8K×10K. Using these high-resolution images to perform hyperparameter optimization search for defect segmentation models is unacceptably time-consuming. Even using Bayesian optimization-based search methods with slightly lower computational cost is still time-consuming and impractical. To address this pain point, the inventors, after analysis, believe that high-resolution industrial images have fewer defect regions, and these regions are also smaller. Compared to 8K×10K resolution, the height and width of most defect regions are only one-thousandth or even one-ten-thousandth of the original image. The defect segmentation model struggles not with large, normal areas, but rather with the few small defect regions. Therefore, in this embodiment, by extracting partial images of defective regions and corresponding defect segmentation annotations, the amount of sample data is reduced. Combined with active learning, this further accelerates the hyperparameter optimization search process for the defect segmentation model, significantly reducing the time required for hyperparameter optimization. Furthermore, the hyperparameter optimization method and defect segmentation model training method of this invention can be combined with various hyperparameter search methods, such as grid search, random search, and Bayesian optimization-based search, to achieve highly compatible, efficient, and accurate optimization search, and obtain the best hyperparameter optimization results required by the target task model in a short time.

[0079] Based on the training method of the aforementioned defect segmentation model, this invention provides a defect detection method, such as... Figure 6 As shown, some embodiments of the defect detection method include the following steps:

[0080] Step 3100: Acquire an image of the object being detected;

[0081] Step 3200: Input the image of the object to be detected into the pre-trained defect segmentation model to obtain the defect region in the image of the object to be detected. The location of the defect on the object to be detected can be determined based on the defect region in the image of the object to be detected.

[0082] The objects being inspected here can be products on an industrial assembly line, mechanical parts in a bin, tools on a workbench, etc., without specific limitations. The defect segmentation model is trained using the aforementioned defect segmentation model training method.

[0083] The above embodiments are data-related optimizations. Some embodiments of this invention optimize from the model perspective, that is, by optimizing the structure of the defect segmentation model. This achieves lightweighting of the defect segmentation model without reducing its performance, significantly reducing the time spent on hyperparameter optimization and enhancing the real-time performance of the network model. The optimized defect segmentation model and the relevant steps of the defect detection method when using this model are described in detail below. For ease of description, unless otherwise specified, "Conv3x3" refers to a 3×3 standard convolutional block, which consists of a convolutional layer with a kernel size of 3×3 and a stride of 1×1, plus a normalization layer (e.g., a batch normalization layer) and an activation function (e.g., the ReLU function). "Conv1x1" refers to a 1×1 standard convolutional block, which consists of a convolutional layer with a kernel size of 1×1 and a stride of 1×1, plus a normalization layer (e.g., a batch normalization layer) and an activation function (e.g., the ReLU function).

[0084] Please refer to Figure 7 The defect segmentation model provided by this invention includes an encoder B, a decoder D, and a classification head module C. Encoder B is responsible for extracting multi-scale features of the image, decoder D is responsible for decoding, and classification head module C completes the final pixel-by-pixel classification task to predict whether each pixel belongs to a defect. Based on this, please refer to... Figure 8 Step 3200 specifically includes:

[0085] Step 3210: Input the image of the object to be detected into encoder B for multi-scale feature extraction to obtain multiple layers of first feature maps with different resolutions; here, the multiple layers of first feature maps with different resolutions can be taken from the feature maps output by multiple layers of encoder B, and the higher the number of layers, the lower the resolution.

[0086] Step 3220: Input the first feature maps of different resolutions from the multiple layers into the decoder D for fusion, and use the fused features for dimensionality upscaling and information recovery to obtain the fused feature map d;

[0087] Step 3230: Pixel-level classification prediction is performed using the fused feature map d through the classification head module C to obtain the defect segmentation result r. The defect segmentation result r includes the classification result of whether each pixel in the image of the detected object belongs to a defect pixel.

[0088] Step 3240: Determine the defect region based on the defective pixels, that is, the region composed of defective pixels is the defect region.

[0089] The defect segmentation model provided by this invention consists of only an encoder B, a decoder D, and a classification head module C. Its structure is relatively simple, which enables the model to be lightweight, thereby improving the calculation speed and reducing the time spent on hyperparameter optimization.

[0090] In some embodiments, given the high resolution and small defects of industrial images, the focus is on extracting location information, and the need for a large receptive field is relatively small. Therefore, only the three layers of the first feature maps f1, f2, and f3 are retained (e.g., Figure 7 As shown in the diagram, this approach improves defect segmentation performance while reducing computational cost. Considering both defect segmentation performance and computational cost, encoder B can employ a lightweight network widely recognized in the field, such as the ResNet-18 backbone network. The ResNet-18 backbone network has strong feature extraction capabilities and a relatively small model size, satisfying both the requirements for defect segmentation performance and computational cost. This example only uses the ResNet-18 backbone network; other similar lightweight networks with equivalent performance can also be used.

[0091] Decoder D can also employ a lightweight design. In some embodiments, decoder D includes multiple standard convolutional blocks (e.g., Conv1x1) that reduce the number of channels in the feature map without changing the resolution, as well as upsampling operations. By using these standard convolutional blocks that reduce the number of channels in the feature map without changing the resolution, the amount of data in the feature map is reduced, thereby reducing the computational cost of the model. The upsampling operation can increase the dimensionality of the feature map, which, in conjunction with convolutional layers, enables information recovery.

[0092] The classification head module C is used to perform pixel-level classification prediction using the fused feature map d to obtain the defect segmentation result r. It can include only a standard convolutional block (e.g., Conv1x1) to reduce the number of parameters.

[0093] Please refer to Figure 9 In some embodiments, the decoder D includes a first convolutional layer and a self-attention module (Attention), based on which, such as Figure 10 As shown, step 3220 specifically includes steps 3221 to 3223, which will be explained in detail below.

[0094] Step 3221: Unify the first feature maps of different resolutions in multiple layers to the resolution of the first feature map with the largest resolution, and perform channel stitching to obtain the stitched feature map f.

[0095] by Figure 7 Taking the three-layer first feature maps f1, f2 and f3 as an example, the first feature map f1 has the largest resolution. Therefore, the first feature maps f2 and f3 are upsampled to the resolution of the first feature map f1 to unify the resolution of the three.

[0096] Step 3222: Input the spliced ​​feature map f into the first convolutional layer for convolution processing to obtain the first multi-class score map p.

[0097] The first convolutional layer should be configured such that the number of channels in the output multi-class score map p is the same as the number of preset classification categories. There are multiple classification categories; at least one represents defects, and one represents the background (i.e., non-defects). When multiple different types of defects need to be represented, there can be multiple classification categories representing defects, each representing a different defect type. The first convolutional layer can be a Conv1x1 layer.

[0098] The first multi-class score map p obtained here is equivalent to a preliminary predicted score map in the middle of the whole process. Each channel corresponds to a classification category, and the value on each channel represents the probability of the corresponding classification category. Figure 9 The dashed line in the diagram represents the first multi-class score map p that the decoder D will output during training. This map, along with the corresponding defect segmentation label, is used to train the defect segmentation model under the control of the loss function L2. This is an auxiliary branch during training and will be discarded during inference, not participating in the calculation.

[0099] Step 3223: Using the self-attention module Attention, the first multi-class score map p is used to enhance the contextual information of the concatenated feature map f, resulting in the fused feature map d.

[0100] The self-attention module, based on the self-attention mechanism, can capture the relationships or similarities between pixels in an image, thus enhancing the contextual information of each pixel. This means it can capture global dependencies in the image, not just local ones, thereby improving the accuracy of defect segmentation. The first multi-class score map p provides category information.

[0101] In some embodiments, step 3223 specifically includes: using the first multi-class score map p to perform target enhancement on the concatenated feature map f to obtain a target-enhanced feature map fp; inputting the concatenated feature map f and the target-enhanced feature map fp into the self-attention module Attention for processing by the self-attention mechanism to obtain a fused feature map d, wherein the concatenated feature map f is used to form the query matrix in the self-attention mechanism, and the target-enhanced feature map fp is used to form the key matrix and value matrix in the self-attention mechanism.

[0102] Among these, an appropriate target enhancement method can be selected based on actual needs. The main purpose of target enhancement is to enhance the feature representation of the target region (i.e., the defect region) in the image. Please refer to [reference needed]. Figure 11 In some embodiments of the present invention, a first multi-class score map p is used to perform target enhancement on the spliced ​​feature map f to obtain a target enhanced feature map fp, specifically including steps 3223a and 3223b, which are described below in conjunction with... Figure 12 Please provide a detailed explanation.

[0103] Step 3223a: Perform shape transformation on the spliced ​​feature map f to form a two-dimensional matrix F, and perform shape transformation on the first multi-class score map p to form a two-dimensional matrix P. The two-dimensional matrix F and the two-dimensional matrix P can be multiplied together.

[0104] As can be understood, an image is a three-dimensional data set with three dimensions: height, width, and number of channels. Shape transformation refers to changing the height, width, and / or feature dimensions (i.e., the number of channels) to alter the shape or size of the image. The main purpose of shape transformation on the stitched feature map f and the first multi-class score map p is to convert them into two-dimensional matrices and make their dimensions conform to the requirements of matrix multiplication, thus facilitating matrix multiplication and achieving target augmentation. Any shape transformation method that achieves this goal is acceptable.

[0105] Figure 12 A specific transformation method is shown, where the shape of the concatenated feature map f is [batch, h, w, channel], where batch is the batch size, h is the height, w is the width, and channel is the number of channels. The shape of the first multi-class score map p is [batch, h, w, classes], where classes is the number of classification categories. Figure 12 As shown, the concatenated feature map f is transformed into two-dimensional data [batch, h×w, channel] by performing Reshape and Permute operations, thus obtaining a two-dimensional matrix F. The first multi-class score map p is changed to [batch, classes, h×w] by performing a Reshape operation, thus obtaining two-dimensional data. Then, a Softmax operation is performed on the h×w dimension to obtain a two-dimensional matrix P.

[0106] Step 3223b: Multiply the two-dimensional matrix F and the two-dimensional matrix P, and perform shape transformation on the result of the multiplication to restore the image data, thereby obtaining the target enhancement feature map fp.

[0107] like Figure 12 In the specific implementation shown, the two-dimensional matrix F and the two-dimensional matrix P are multiplied (Mul), transforming the data shape to [batch, classes, channel]. Then, the data shape is modified to [batch, channel, classes, 1] through the Permute and Unsqueeze (dimensional expansion) operations, restoring the three-dimensional image data and obtaining the target enhancement feature map fp. Finally, the concatenated feature map f and the target enhancement feature map fp are input into the attention module for processing, and the fused feature map d is output.

[0108] Essentially, the target augmentation feature map fp is a pixel-by-pixel similarity calculation between the concatenated feature map f and different classification categories, i.e., the enhanced representation of the target region. First, target augmentation is performed, and then the resulting target augmentation feature map and the concatenated feature map are input into the self-attention module for self-attention processing. The purpose is to obtain the feature representation of the target region by aggregating the feature representations of pixels in the corresponding target region, and then perform fine-grained pixel-level processing through the self-attention module. The core idea is to perform a "region-first, pixel-later" optimization process, making network training easier and more stable. In contrast, directly inputting the first multi-class score map and the concatenated feature map into the self-attention module is equivalent to directly providing pixel-level guidance. The network lacks the optimization process from region to pixel, making training more difficult and resulting in poor stability.

[0109] The core of the self-attention module (Attention) is the self-attention mechanism; please refer to [reference needed]. Figure 13 In some embodiments, the self-attention module features a lightweight design with fewer parameters, consisting of only four parameterized network layers: a second, third, fourth, and fifth convolutional layer. Furthermore, the second, third, and fourth convolutional layers reduce the number of channels in the feature map. Based on this, such as... Figure 14 As shown, the step of inputting the spliced ​​feature map f and the target enhancement feature map fp into the self-attention module Attention for processing by the self-attention mechanism to obtain the fused feature map d specifically includes steps 3223c~3223h, which will be explained in detail below.

[0110] Step 3223c: Reduce the number of channels in the concatenated feature map f through the second convolutional layer to obtain the second convolutional feature map; reduce the number of channels in the target enhancement feature map fp through the third convolutional layer to obtain the third convolutional feature map; and reduce the number of channels in the target enhancement feature map fp through the fourth convolutional layer to obtain the fourth convolutional feature map.

[0111] The second, third, and fourth convolutional layers can all use Conv1x1.

[0112] Step 3223d: Perform shape transformation on the second convolutional feature map to form the query matrix M. q The third convolutional feature map is shape transformed to form the key matrix M. k , where the query matrix M q Bond matrix M k It can perform matrix multiplication.

[0113] The main purpose of shape transformation on the second and third convolutional feature maps is to convert them into two-dimensional matrices, which can then be used as query matrices M. q Bond matrix Mk And ensure that their dimensions meet the requirements for matrix multiplication. Please refer to... Figure 13 In one specific implementation, the second convolutional feature map is transformed into a two-dimensional matrix by performing Permute and Reshape operations, forming the query matrix M. q The third convolutional feature map is transformed into a two-dimensional matrix by performing a reshape operation, forming the key matrix M. k .

[0114] Step 3223e: Query matrix M q Bond matrix M k Multiply the results and then perform Softmax processing to obtain the relation matrix M. r .

[0115] Step 3223f: Perform shape transformation on the fourth convolutional feature map to form the value matrix M. v The value matrix M v And relation matrix M r It can perform matrix multiplication.

[0116] like Figure 13 In the specific implementation shown, the fourth convolutional feature map is transformed into a two-dimensional matrix, forming a value matrix M, by performing Reshape and Permute operations. v .

[0117] Step 3223g: Convert the value matrix M v And relation matrix M r The images are multiplied, and the result is transformed to restore them to the same resolution as the stitched feature map f. Then, it passes through a fifth convolutional layer to restore it to the same number of channels as the stitched feature map f, resulting in the fifth convolutional feature map. The fifth convolutional layer can be a Conv1x1 layer.

[0118] like Figure 13 In the specific implementation shown, the fifth convolutional feature map is restored to image data with the same resolution as the stitched feature map f by performing Permute and Reshape operations, and then restored to the same number of channels as the stitched feature map f by the fifth convolutional layer.

[0119] Step 3223h: Perform an element-wise addition operation (Add) on the fifth convolutional feature map and the concatenated feature map f to obtain the fused feature map d. The number of channels in the fused feature map d is the same as that in the concatenated feature map f.

[0120] Essentially, the self-attention module (Attention) first calculates the relationship matrix M between the concatenated feature map f and the target augmented feature map fp through a self-attention mechanism. r Then the relation matrix M rThe sum-value matrix M v Perform matrix multiplication (Mul) and restore the number of channels through Permute, Reshape and convolutional layers to obtain the target context representation. Finally, perform element-wise addition with the concatenated feature map f to output the fused feature map d.

[0121] In some embodiments, the classification head module C includes a sixth convolutional layer, and the defect segmentation result includes a defect segmentation result map. The pixel values ​​of pixels in the defect segmentation result map are used to indicate whether the corresponding pixel in the image of the detected object belongs to a defect pixel, similar to a defect segmentation annotation map. Based on this, step 3230 specifically includes: inputting the fused feature map d into the sixth convolutional layer for convolution processing, and then upsampling it to the resolution of the image of the detected object to obtain a second multi-class score map c. The number of channels in the second multi-class score map c is the same as the number of preset classification categories. The value of each channel at each pixel position in the second multi-class score map c represents the probability that the corresponding pixel in the image of the detected object belongs to each classification category, similar to the first multi-class score map p. For each pixel in the second multi-class score map c, the classification category corresponding to the channel with the largest value is taken as the final classification category to obtain the defect segmentation result map.

[0122] During training (such as in steps 2200 and 2300), the loss function of the defect segmentation model needs to be used, such as... Figure 7 As shown, in some embodiments, the loss function Loss consists of a first loss function L1 and a second loss function L2. The first loss function L1 represents the difference between the defect segmentation result map r and the defect segmentation label, and the second loss function L2 represents the difference between the first multi-class score map p obtained by inputting the first sample image into the defect segmentation model and the defect segmentation label.

[0123] In some embodiments, taking advantage of the high resolution and small defect area characteristics of industrial images, the first loss function L1 and the second loss function L2 adopt the intersection-union (IUU) loss function. IUU is a region correlation loss function, which is good at mining information from the foreground region (i.e., the defect region), and is beneficial for the detection of small targets. Therefore, the expression for the first loss function is:

[0124] ,

[0125] Where IoU1 represents the intersection-union ratio (IoU1) of the defect segmentation result image r and the defect segmentation annotation, specifically it can be...

[0126] ,

[0127] Where label represents the defect segmentation label. This indicates finding the intersection. This indicates finding the union of sets.

[0128] The first multi-class score map p is the intermediate predicted score map in the entire process. It is used to obtain intermediate segmentation results for comparison with defect segmentation annotations. This is achieved by upsampling the first multi-class score map p (represented as...). The resolution of the image of the object being detected is used to determine the classification category (represented as follows) for each upsampled pixel, based on the channel with the highest value. The intermediate segmentation results are obtained. The expression for the second loss function is:

[0129] ,

[0130] Where IoU2 represents the intersection-union ratio (IoU2) of the intermediate segmentation results and the defect segmentation annotations, specifically it can be...

[0131] .

[0132] In some embodiments, the loss function of the defect segmentation model is expressed as follows:

[0133] ,

[0134] in and The weighting coefficients are preset and can be designed according to actual needs. In one embodiment, they are... and .

[0135] The above is an introduction to model optimization. Data optimization and model optimization can be used individually or in combination. In one embodiment, the defect segmentation model provided by this invention is used, and based on... Figure 4 The method shown obtains the optimal hyperparameter combination. This optimal hyperparameter combination is then used to train the defect segmentation model, resulting in a trained defect segmentation model F. During detection, an image of the object to be detected is acquired, and the defect segmentation result obtained from the defect segmentation model F is input as shown in the figure. Figure 15 As shown, the left image is the image of the object being detected, and the right image is the defect segmentation result.

[0136] Based on the above-described hyperparameter optimization method, this invention also provides a hyperparameter optimization device, please refer to... Figure 16 In some embodiments, the device includes a sample acquisition module 11, a hyperparameter search module 12, and a hyperparameter determination module 13.

[0137] The sample acquisition module 11 is used to acquire a sample image set, which includes multiple sample images and corresponding annotations.

[0138] The hyperparameter search module 12 is used to train a target task model using a sample image set for each of a preset set of hyperparameter combinations, and to evaluate the performance metrics of the trained target task model when performing the target task; it also records the performance metrics of the target task model trained with each hyperparameter combination when performing the target task. Specifically, during the training of the target task model using the sample image set, if the prediction error of the target task model for any sample image is less than a second error threshold, the sample image is discarded and will not participate in subsequent training rounds.

[0139] The hyperparameter determination module 13 is used to select the hyperparameter combination corresponding to the trained target task model with the best performance index as the final hyperparameter combination used to train the target task model.

[0140] Based on the above-mentioned defect detection method, the present invention also provides a defect detection device, please refer to... Figure 17 In some embodiments, the device includes an image acquisition module 210, a defect segmentation module 220, and a training module 230, which are described below.

[0141] The image acquisition module 210 is used to acquire images of the detected object.

[0142] The defect segmentation module 220 is used to input the image of the object to be detected into a pre-trained defect segmentation model to obtain the defect region in the image of the object to be detected.

[0143] Training module 230 is used to train the defect segmentation model, such as Figure 17 As shown, the training module 230 includes a sample acquisition submodule 231, a hyperparameter search submodule 232, and an optimization training submodule 233. The sample acquisition submodule 231 is used to acquire a first sample image set, which includes multiple first sample images and corresponding defect segmentation annotations. The defect segmentation annotations are used to indicate defect regions in the first sample images. In some embodiments, the sample acquisition submodule 231 is specifically used to perform... Figure 5 The process shown is to obtain the first sample image set.

[0144] The hyperparameter search submodule 232 is used to train a defect segmentation model using a first sample image set for each of a preset set of hyperparameter combinations, and to evaluate the defect segmentation performance of the trained model. It also records the defect segmentation performance of the model trained using each hyperparameter combination. Specifically, during the training of the defect segmentation model using the first sample image set, if the defect segmentation error of the model for any first sample image is less than a first error threshold, that first sample image is discarded and will not participate in subsequent training rounds. See step 2200 for details.

[0145] The optimization training submodule 233 is used to select the hyperparameter combination corresponding to the trained defect segmentation model with the best defect segmentation performance as the final hyperparameter combination, and to train the defect segmentation model using the final hyperparameter combination to obtain the trained defect segmentation model.

[0146] When adopting such Figure 7 In the defect segmentation model shown, the defect segmentation module 220 is specifically used for: inputting the image of the object to be detected into the encoder for multi-scale feature extraction, obtaining multiple layers of first feature maps with different resolutions; inputting the multiple layers of first feature maps with different resolutions into the decoder for fusion, and using the fused features for dimensionality upscaling and information recovery to obtain a fused feature map; using the fused feature map through the classification head module to perform pixel-level classification prediction to obtain the defect segmentation result, which includes the classification result of whether each pixel in the image of the object to be detected belongs to a defect pixel; and determining the defect region based on the defect pixels. Please refer to the relevant introduction above for more information on the defect segmentation model.

[0147] In some embodiments, the decoder of the defect segmentation model includes a first convolutional layer and a self-attention module. The defect segmentation module 220 inputs multiple layers of first feature maps with different resolutions into the decoder for fusion, and uses the fused features for dimensionality upscaling and information recovery to obtain a fused feature map. This includes: unifying the multiple layers of first feature maps with different resolutions to the resolution of the first feature map with the largest resolution, and performing channel concatenation to obtain a concatenated feature map; inputting the concatenated feature map into the first convolutional layer for convolution processing to obtain a first multi-class score map, wherein the number of channels of the first multi-class score map is the same as the number of preset classification categories; and using the self-attention module, using the first multi-class score map to enhance the contextual information of the concatenated feature map to obtain the fused feature map. Please refer to steps 3221-3223 above for details.

[0148] In some embodiments, the defect segmentation module 220 utilizes a self-attention module to enhance the contextual information of the concatenated feature map using a first multi-class score map to obtain a fused feature map. This includes: using the first multi-class score map to enhance the target of the concatenated feature map to obtain a target-enhanced feature map; and inputting the concatenated feature map and the target-enhanced feature map into the self-attention module for processing using the self-attention mechanism to obtain a fused feature map. The concatenated feature map is used to form the query matrix in the self-attention mechanism, and the target-enhanced feature map is used to form the key matrix and value matrix in the self-attention mechanism.

[0149] In some embodiments, the defect segmentation module 220 uses a first multi-class score map to perform target enhancement on the stitched feature map to obtain a target enhanced feature map. This includes: performing a shape transformation on the stitched feature map to form a two-dimensional matrix F; performing a shape transformation on the first multi-class score map to form a two-dimensional matrix P, wherein the two-dimensional matrix F and the two-dimensional matrix P can be multiplied; multiplying the two-dimensional matrix F and the two-dimensional matrix P; and performing a shape transformation on the result of the multiplication to restore it to image data, thereby obtaining the target enhanced feature map. Please refer to steps 3223a and 3223b above for details.

[0150] In some embodiments, the self-attention module includes a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer; the defect segmentation module 220 inputs the spliced ​​feature map and the target enhancement feature map into the self-attention module for processing by the self-attention mechanism to obtain a fused feature map, including: reducing the number of channels in the spliced ​​feature map through the second convolutional layer to obtain a second convolutional feature map, reducing the number of channels in the target enhancement feature map through the third convolutional layer to obtain a third convolutional feature map, reducing the number of channels in the target enhancement feature map through the fourth convolutional layer to obtain a fourth convolutional feature map; and performing a shape transformation on the second convolutional feature map to form a query matrix M. q The third convolutional feature map is shape transformed to form the key matrix M. k , where the query matrix M q Bond matrix M k Capable of performing matrix multiplication; query matrix M q Bond matrix M k Multiply the results and then perform Softmax processing to obtain the relation matrix M. r The fourth convolutional feature map is shape-transformed to form a value matrix M. v The value matrix M v And relation matrix M r Capable of performing matrix multiplication; transforming the value matrix M v And relation matrix M r The images are multiplied, and the result is transformed to restore the image data to the same resolution as the stitched feature map. Then, it is passed through a fifth convolutional layer to restore the image to the same number of channels as the stitched feature map, resulting in the fifth convolutional feature map. The fifth convolutional feature map and the stitched feature map are then added element-wise to obtain the fused feature map. Please refer to steps 3223c~3223h above for details.

[0151] In some embodiments, the defect segmentation result includes a defect segmentation result map, where the pixel values ​​of pixels in the defect segmentation result map are used to indicate whether the corresponding pixel in the image of the detected object belongs to a defect pixel; the classification head module includes a sixth convolutional layer. The defect segmentation module 220 uses the fused feature map through the classification head module to perform pixel-level classification prediction to obtain the defect segmentation result, including: inputting the fused feature map into the sixth convolutional layer for convolution processing, and then upsampling it to the resolution of the image of the detected object to obtain a second multi-class score map. The number of channels in the second multi-class score map is the same as the number of preset classification categories. The value of each channel at each pixel position in the second multi-class score map represents the probability that the corresponding pixel in the image of the detected object belongs to each classification category; for each pixel in the second multi-class score map, the classification category corresponding to the channel with the largest value is taken as the final classification category to obtain the defect segmentation result map.

[0152] For the loss function of the defect segmentation model, please refer to the relevant introduction above, which will not be repeated here.

[0153] The technical solution of the present invention has the following advantages compared with the prior art:

[0154] (1) High efficiency: From the perspective of data, based on active learning, difficult samples in the sample images are selected as surrogate samples for hyperparameter optimization, which greatly reduces the sample size, speeds up the hyperparameter optimization process, and improves model performance.

[0155] (2) In some embodiments, for the defect detection task, from the perspective of data, taking into account the characteristics of industrial images with high resolution and few and small defect areas, the key learning areas are extracted and used as actual training sample images, which significantly reduces the computational load of the model and further reduces computational overhead.

[0156] (3) In some embodiments, from the perspective of the model, the structure of the model is specially optimized and designed, the number of parameters of the constructed model is small, the calculation speed is fast, and the real-time performance is excellent; it can also work together with the optimization from the perspective of data to further reduce the computational overhead, greatly reduce the time consumption of the hyperparameter optimization process, and obtain the optimal hyperparameter combination in a short time.

[0157] (4) Accuracy: During the hyperparameter search process, difficult samples are selected for training, and the generation of difficult samples is continuously iterated and optimized. The model can obtain the best hyperparameter optimization results in the most important and representative samples.

[0158] (5) Adaptability and scalability: It can adapt to different datasets and tasks, and can be used in combination with other hyperparameter search methods to improve the optimization effect.

[0159] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0160] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A defect detection method based on a defect detection model, characterized in that, The defect detection model includes an encoder, a decoder, and a classification head module; the defect detection method includes: Acquire an image of the object being detected; The image of the detected object is input into the encoder for multi-scale feature extraction to obtain a first feature map with multiple layers of different resolutions. The first feature maps of different resolutions in the multiple layers are input into the decoder for fusion, and the fused features are used for dimensionality upscaling and information recovery to obtain a fused feature map; The classification head module uses the fused feature map to perform pixel-level classification prediction to obtain defect segmentation results. The defect segmentation results include classification results of whether each pixel in the image of the detected object belongs to a defect pixel. Defect regions are determined based on defective pixels.

2. The defect detection method as described in claim 1, characterized in that, The decoder includes a first convolutional layer and a self-attention module. The first feature maps of different resolutions from the multiple layers are input into the decoder for fusion. The fused features are then used for dimensionality upscaling and information recovery to obtain a fused feature map, including: The first feature maps of different resolutions are fused together, and the fused feature map is input into the first convolutional layer for convolution processing to obtain a first multi-class score map. The number of channels of the first multi-class score map is the same as the number of preset classification categories, wherein at least one of the classification categories is used to represent defects. Using the self-attention module, the first multi-class score map is used to enhance the contextual information of the fused feature map to obtain the fused feature map.

3. The defect detection method as described in claim 2, characterized in that, The process of fusing the first feature maps of different resolutions across multiple layers and inputting the fused feature map into the first convolutional layer for convolution processing to obtain a first multi-class score map includes: The first feature maps of different resolutions in the multiple layers are unified to the resolution of the first feature map with the largest resolution, and then channel stitching is performed to obtain a stitched feature map. The stitched feature map is input into the first convolutional layer for convolution processing to obtain the first multi-class score map.

4. The defect detection method as described in claim 2, characterized in that, The step of using the self-attention module to enhance the contextual information of the fused feature map with the first multi-class score map to obtain the fused feature map includes: The first multi-class score map is used to perform target enhancement on the fused feature map to obtain a target-enhanced feature map. The fused feature map and the target augmented feature map are input into the self-attention module for processing by the self-attention mechanism to obtain a fused feature map. The fused feature map is used to form the query matrix in the self-attention mechanism, and the target augmented feature map is used to form the key matrix and value matrix in the self-attention mechanism.

5. The defect detection method as described in claim 4, characterized in that, The step of using the first multi-class score map to perform target enhancement on the fused feature map to obtain a target-enhanced feature map includes: The fused feature map is subjected to shape transformation to form a two-dimensional matrix F, and the first multi-class score map is subjected to shape transformation to form a two-dimensional matrix P. The two-dimensional matrix F and the two-dimensional matrix P can be multiplied together. Multiply the two-dimensional matrix F and the two-dimensional matrix P, and perform a shape transformation on the result of the multiplication to restore the image data, thereby obtaining the target enhancement feature map.

6. The defect detection method as described in claim 4, characterized in that, The self-attention module includes a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer; the step of inputting the fused feature map and the target enhancement feature map into the self-attention module for processing by the self-attention mechanism to obtain the fused feature map includes: The second convolutional layer reduces the number of channels in the fused feature map to obtain a second convolutional feature map. The third convolutional layer reduces the number of channels in the target enhancement feature map to obtain a third convolutional feature map. The fourth convolutional layer reduces the number of channels in the target enhancement feature map to obtain a fourth convolutional feature map. The second convolutional feature map is shape-transformed to form the query matrix M. q The third convolutional feature map is then subjected to shape transformation to form a key matrix M. k , where the query matrix M q Bond matrix M k Capable of performing matrix multiplication; Query matrix M q Bond matrix M k Multiply the results and then perform Softmax processing to obtain the relation matrix M. r ; The fourth convolutional feature map is shape transformed to form a value matrix M. v The value matrix M v And relation matrix M r Capable of performing matrix multiplication; Value matrix M v And relation matrix M r Multiply the data, perform a shape transformation on the result of the multiplication to restore it to image data with the same resolution as the fused feature map, and then pass it through the fifth convolutional layer to restore it to the same number of channels as the fused feature map, thus obtaining the fifth convolutional feature map; The fifth convolutional feature map is fused with the fused feature map to obtain a fused feature map.

7. The defect detection method as described in claim 1, characterized in that, The defect segmentation result includes a defect segmentation result map, where the pixel values ​​of the pixels in the defect segmentation result map are used to indicate whether the corresponding pixel in the image of the detected object belongs to a defect pixel; the classification head module includes a sixth convolutional layer; The step of performing pixel-level classification prediction using the fused feature map through the classification head module to obtain defect segmentation results includes: The fused feature map is input into the sixth convolutional layer for convolution processing, and then upsampled to the resolution of the image of the detected object to obtain a second multi-class score map. The number of channels in the second multi-class score map is the same as the number of preset classification categories. The value of each channel at each pixel position in the second multi-class score map represents the probability that the corresponding pixel in the image of the detected object belongs to each classification category. Among them, at least one of the classification categories is used to represent defects. For each pixel in the second multi-class score map, the classification category corresponding to the channel with the largest value is taken as the final classification category to obtain the defect segmentation result map.

8. The defect detection method according to any one of claims 1 to 7, characterized in that, The defect segmentation model is trained in the following way: A first sample image set is obtained, which includes multiple first sample images and corresponding defect segmentation annotations, wherein the defect segmentation annotations are used to indicate defect regions in the first sample images; For each of the preset multiple hyperparameter combinations, the defect segmentation model is trained using the first sample image set using that hyperparameter combination, and the defect segmentation performance of the trained defect segmentation model is evaluated; the defect segmentation performance of the defect segmentation model trained using each hyperparameter combination is recorded; wherein, during the process of training the defect segmentation model using the first sample image set, if the defect segmentation error of the defect segmentation model for any first sample image is less than a first error threshold, then the first sample image is discarded and will not participate in subsequent rounds of training; The hyperparameter combination corresponding to the trained defect segmentation model with the best defect segmentation performance is selected as the final hyperparameter combination. The defect segmentation model is trained using the final hyperparameter combination to obtain the trained defect segmentation model.

9. The defect detection method as described in claim 8, characterized in that, The acquisition of the first sample image set includes: Obtain an original sample image set, which includes multiple original sample images and corresponding original defect segmentation annotation maps. The original defect segmentation annotation maps have defect marking regions, and the regions in the original sample images that correspond to the defect marking regions in the corresponding original defect segmentation annotation maps are the defect regions. A portion of the image including the defect marking region is extracted from the original defect segmentation annotation map to obtain a first defect segmentation annotation map; the region in the original sample image corresponding to the first defect segmentation annotation map extracted from the corresponding original defect segmentation annotation map is extracted to obtain a first sample image corresponding to the first defect segmentation annotation map. Unify the resolution of all first sample images and first defect segmentation annotation maps; The first sample image set consists of all the first sample images and the corresponding first defect segmentation annotation map, wherein the first defect segmentation annotation map is the defect segmentation annotation.

10. The defect detection method as described in claim 8, characterized in that, The loss function of the defect segmentation model consists of a first loss function and a second loss function. The first loss function represents the difference between the defect segmentation result map and the defect segmentation label, and the second loss function represents the difference between the first multi-class score map obtained by inputting the first sample image into the defect segmentation model and the defect segmentation label.

11. The defect detection method as described in claim 10, characterized in that, The expression for the first loss function is: ， Where IoU1 represents the intersection-union ratio of the defect segmentation result image and the defect segmentation annotation; The expression for the second loss function is: ， Where IoU2 represents the intersection-union ratio of the intermediate segmentation result and the defect segmentation annotation, and the intermediate segmentation result is obtained by upsampling the first multi-class score map to the resolution of the image of the detected object, and taking the classification category corresponding to the channel with the largest value for each pixel after upsampling; The expression for the loss function of the defect segmentation model is: ， in and These are the preset weighting coefficients.

12. A defect detection device based on a defect detection model, characterized in that, The defect detection model includes an encoder, a decoder, and a classification head module; the defect detection device includes: The image acquisition module is used to acquire images of the detected object; The defect segmentation module is used for: The image of the detected object is input into the encoder for multi-scale feature extraction to obtain a first feature map with multiple layers of different resolutions. The first feature maps of different resolutions in the multiple layers are input into the decoder for fusion, and the fused features are used for dimensionality upscaling and information recovery to obtain a fused feature map; The classification head module uses the fused feature map to perform pixel-level classification prediction to obtain defect segmentation results. The defect segmentation results include classification results of whether each pixel in the image of the detected object belongs to a defect pixel. Defect regions are determined based on defective pixels.

13. A computer program product comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by the processor, they implement the defect detection method according to any one of claims 1 to 11.

Images