Defect detection method and device and hyper-parameter optimization method and device

Abstract

The invention discloses a defect detection method and device and a hyper-parameter optimization method and device, and the hyper-parameter optimization method comprises the steps: firstly obtaining a sample image set, and then for each hyper-parameter combination in a plurality of hyper-parameter combinations, training a target task model through the sample image set by adopting the hyper-parameter combination, evaluating and recording a performance index when the trained target task model executes the target task; and finally, selecting the hyper-parameter combination corresponding to the trained target task model with the optimal performance index as a final hyper-parameter combination for training the target task model. In the model training process, if the prediction error of the target task model for any sample image is smaller than an error threshold value, the sample image is abandoned and does not participate in the subsequent round of training any more, so that the sample size used in the hyper-parameter optimization process is reduced, the time consumption is reduced, the model utilizes the most important and most representative sample, and the training efficiency is improved. And the optimal hyper-parameter combination is efficiently and accurately obtained.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a defect detection method and device, and a hyperparameter optimization method and device. Background Art

[0002] In some image processing tasks such as defect detection, deep learning technology has been increasingly widely used. For example, in the implementation of industrial defect detection, defect segmentation technology based on neural networks has many advantages such as pixel-by-pixel precise positioning, strong scene adaptability, and high accuracy, and is widely adopted by the market. It is usually the preferred technology for industrial detection. When training a deep learning model, it is necessary to set hyperparameters such as batch size, learning rate, optimizer, regularization, and data enhancement method. With the continuous development of technology, the determination of appropriate or optimal hyperparameter combinations has become a key factor restricting the performance improvement of deep learning models. Improper hyperparameter settings will lead to long model training time and poor detection results.

[0003] In order to obtain the best hyperparameter combination, hyperparameter optimization is required, and hyperparameter automatic search tools are usually used, including: grid search, random search, parameter space search based on Bayesian optimization, etc. In actual use, grid search and random search, due to the high resolution of industrial images, make the calculation overhead of deep learning networks huge, resulting in the entire optimization process taking too long; Bayesian optimization uses historical data to continuously adjust the search space of hyperparameters, and can find a suitable hyperparameter combination with a relatively small number of attempts, but Bayesian optimization uses surrogate models such as Gaussian process regression. In deep learning, there are usually a large number of hyperparameters that need to be tuned. In high-dimensional parameter space, it is difficult to find an accurate approximation of the surrogate model, resulting in performance degradation; at the same time, for deep learning models, the objective function is usually non-convex, and it is difficult to fit using surrogate models such as Gaussian process regression, and if the deep learning model is used directly, it will further increase the time consumption. Summary of the invention

[0004] The main technical problem solved by the present invention is how to reduce the time consumption of the hyperparameter optimization process.

[0005] According to the first aspect, an embodiment provides a defect detection method, including:

[0006] Acquire an image of the object to be detected;

[0007] Inputting the image of the inspected object into a pre-trained defect segmentation model to obtain a defect area in the image of the inspected object;

[0008] The defect segmentation model is trained in the following way:

[0009] Acquire a first sample image set, where the first sample image set includes a plurality of first sample images and corresponding defect segmentation annotations, where the defect segmentation annotations are used to indicate defect areas in the first sample images;

[0010] For each hyperparameter combination among the preset multiple hyperparameter combinations, the hyperparameter combination is adopted, the defect segmentation model is trained using the first sample image set, and the defect segmentation performance of the defect segmentation model after the training is evaluated; the defect segmentation performance of the defect segmentation model after training using each hyperparameter combination is recorded; wherein, in the process of training the defect segmentation model using the first sample image set, if the defect segmentation error of the defect segmentation model for any first sample image is less than the first error threshold, the first sample image is discarded and no longer participates in subsequent rounds of training;

[0011] The hyperparameter combination corresponding to the trained defect segmentation model with the best defect segmentation performance is selected as the final hyperparameter combination, and the defect segmentation model is trained using the final hyperparameter combination to obtain a defect segmentation model that has completed the training.

[0012] According to the second aspect, an embodiment provides a hyperparameter optimization method, including:

[0013] Acquire a sample image set, wherein the sample image set includes a plurality of sample images and corresponding annotations;

[0014] For each of the preset multiple hyperparameter combinations, the hyperparameter combination is adopted, the target task model is trained using the sample image set, and the performance index of the trained target task model when performing the target task is evaluated; the performance index of the target task model when performing the target task after being trained using each hyperparameter combination is recorded; wherein, in the process of training the target task model using the sample image set, if the prediction error of the target task model for any sample image is less than the second error threshold, the sample image is discarded and no longer participates in subsequent rounds of training;

[0015] The hyperparameter combination corresponding to the trained target task model with the best performance index is selected as the hyperparameter combination finally used to train the target task model.

[0016] According to a third aspect, an embodiment provides a defect detection device, comprising:

[0017] An image acquisition module, used to acquire an image of the detected object;

[0018] A defect segmentation module, used for inputting the image of the detected object into a pre-trained defect segmentation model to obtain a defect area in the image of the detected object;

[0019] A training module, used for training the defect segmentation model, includes:

[0020] A sample acquisition submodule, used to acquire a first sample image set, wherein the first sample image set includes a plurality of first sample images and corresponding defect segmentation annotations, wherein the defect segmentation annotations are used to indicate defect areas in the first sample images;

[0021] A hyperparameter search submodule is used to adopt each hyperparameter combination in a plurality of preset hyperparameter combinations, train a defect segmentation model using the first sample image set using the hyperparameter combination, and evaluate the defect segmentation performance of the defect segmentation model after the training; record the defect segmentation performance of the defect segmentation model after the training using each hyperparameter combination; wherein, in the process of training the defect segmentation model using the first sample image set, if the defect segmentation error of the defect segmentation model for any first sample image is less than a first error threshold, the first sample image is discarded and no longer participates in subsequent rounds of training;

[0022] The optimization training submodule is used to select the hyperparameter combination corresponding to the trained defect segmentation model with the best defect segmentation performance as the final hyperparameter combination, and use the final hyperparameter combination to train the defect segmentation model to obtain a defect segmentation model that has completed the training.

[0023] According to the fourth aspect, an embodiment provides a hyperparameter optimization device, including:

[0024] A sample acquisition module, used to acquire a sample image set, wherein the sample image set includes a plurality of sample images and corresponding annotations;

[0025] A hyperparameter search module is used to adopt each hyperparameter combination in a plurality of preset hyperparameter combinations, train the target task model using the sample image set, and evaluate the performance index of the trained target task model when performing the target task; record the performance index of the target task model when performing the target task after being trained using each hyperparameter combination; wherein, in the process of training the target task model using the sample image set, if the prediction error of the target task model for any sample image is less than a second error threshold, the sample image is discarded and no longer participates in subsequent rounds of training;

[0026] The hyperparameter determination module is used to select the hyperparameter combination corresponding to the trained target task model with the best performance index as the hyperparameter combination finally used for training the target task model.

[0027] According to the fifth aspect, a computer program product is provided in one embodiment, including a computer program and / or instructions, which, when executed by a processor, implement the defect detection method described in the first aspect or the hyperparameter optimization method described in the second aspect.

[0028] According to the defect detection method and device, hyperparameter optimization method and device, and computer program product of the above-mentioned embodiments, a plurality of hyperparameter combinations are first pre-set, and then each hyperparameter combination is used to train the model using the sample image set, and the performance indicators of the trained model when performing the corresponding target task are evaluated, and the performance indicators of the model trained using each hyperparameter combination when performing the target task are recorded, and finally the hyperparameter combination corresponding to the trained model with the best performance indicator is selected as the final hyperparameter combination for training the model; wherein, in the process of training the model, if the prediction error of the model for any sample image is less than the set error threshold, the sample image is discarded and no longer participates in subsequent rounds of training, which not only reduces the number of samples used in the hyperparameter optimization process, but also because the difficult samples that are difficult to fit the model are more important and more representative, these samples are used as proxy samples to train the model, and the generation of proxy samples is continuously iterated and optimized. The model can obtain the best hyperparameter optimization results in the most important and representative samples, which is also conducive to speeding up training. In summary, the technical solution of the present invention can reduce the time consumption of the hyperparameter optimization process, reduce the computational overhead, and at the same time efficiently and accurately obtain the optimal hyperparameter combination, improve model performance, and achieve cost reduction and efficiency improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the overall processing flow of a hyperparameter optimization method according to an embodiment;

[0030] Figure 2 A schematic diagram of the overall processing flow of a hyperparameter optimization method according to another embodiment;

[0031] Figure 3 A flowchart of a hyperparameter optimization method for some embodiments;

[0032] Figure 4 A flowchart of a method for training a defect segmentation model in some embodiments;

[0033] Figure 5 A flowchart for obtaining a first sample image set in some embodiments;

[0034] Figure 6 is a flow chart of a defect detection method according to some embodiments;

[0035] Figure 7 is a schematic diagram of the structure and loss function of a defect segmentation model in some embodiments;

[0036] Figure 8 A flowchart of inputting an image of an object to be inspected into a pre-trained defect segmentation model to obtain a defect region in the image of the object to be inspected in some embodiments;

[0037] Fig. 9 is a schematic diagram of the structure and processing flow of a decoder in some embodiments;

[0038] Fig.10 A flowchart of inputting multiple layers of first feature maps with different resolutions into a decoder for fusion in some embodiments, and using the fused features for dimension upgrading and information recovery to obtain a fused feature map;

[0039] Fig.11 A flowchart of performing target enhancement on a concatenated feature map using a first multi-category score map to obtain a target enhanced feature map in some embodiments;

[0040] Fig.12 A schematic diagram of a processing flow of performing target enhancement on a spliced ​​feature map to obtain a target enhanced feature map in a specific implementation manner;

[0041] Fig.13 Schematic diagram of the structure and processing flow of the self-attention module in some embodiments;

[0042] Fig.14 A flowchart of inputting a concatenated feature map and a target enhanced feature map into a self-attention module for processing by a self-attention mechanism to obtain a fused feature map in some embodiments;

[0043] Fig.15 A schematic diagram of the result of performing defect detection on an image of an object to be detected using a defect segmentation model in one embodiment;

[0044] Fig.16 It is a schematic diagram of the structure of a hyperparameter optimization device in some embodiments;

[0045] Fig.17 Schematic diagram of the structure of defect detection devices in some embodiments. DETAILED DESCRIPTION

[0046] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0047] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0048] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0049] At present, in the methods of industrial image defect segmentation, for automated hyperparameter optimization, a Bayesian optimization-based approach is usually used to obtain the optimal hyperparameter combination for training the defect segmentation model. Compared with grid search and random search, Bayesian optimization can find the right hyperparameter combination with relatively few attempts; but there are also the following problems: First, the challenge of high-dimensional parameter space: In deep learning, there are many hyperparameters that need to be tuned. In the high-dimensional parameter space, it is difficult for the Bayesian optimization process to find an accurate approximation of the proxy model. If the deep learning model is used directly, especially the deep learning model used for high-resolution images of industrial inspection, the computational overhead will be too high and the optimization time will be too long. Second, the fitting of non-convex objective functions: Simple proxy models such as Gaussian process regression used in the Bayesian optimization process are difficult to fit the non-stationary and non-convex objective functions in deep learning, resulting in a decrease in optimization performance. Third, the structural design of the model: In order to obtain more accurate optimization results, the Bayesian optimization process directly uses the defect segmentation model based on the deep learning network as the proxy model, but the lightweight structure design of the defect segmentation model is insufficient, and there is a lack of specialized optimization design for the segmentation network, which requires a lot of computing resources and time.

[0050] In view of some problems existing in the prior art, the present invention adopts a novel solution, performs optimization design from the perspectives of data and model, and can automatically optimize the hyperparameters of the model to obtain the best hyperparameter combination, thereby greatly reducing the optimization time and improving the performance of the model. The present invention provides a hyperparameter optimization method based on active learning. Compared with the current similar methods, the purpose of the hyperparameter optimization method of the present invention is to achieve accurate and automated optimization of hyperparameters with lower time consumption, obtain the best hyperparameter combination, and thus be used for model training to improve the detection performance of industrial images. Specifically, the embodiments of the present invention have some or all of the following advantages: (1) Accuracy: In the process of hyperparameter search, difficult samples are used as proxy samples for subsequent training in an active learning manner, and the generation of proxy samples is continuously iterated and optimized. The model can obtain the best hyperparameter optimization results in the most important and representative samples, while also reducing the sample size; (2) Efficiency: In some embodiments, from a data perspective, based on an active learning method, key learning areas in the sample image are selected to form proxy samples for hyperparameter optimization, and / or from a model perspective, the model is specially optimized and designed. The constructed model has fewer parameters and faster calculation speed, and the best hyperparameter combination can be obtained in a short time; (3) Adaptability and scalability: It can adapt to different data sets and tasks, and can be used well with other hyperparameter search methods to improve the optimization effect.

[0051] The concept of the present invention is first introduced below. After analyzing and exploring the problem of long time-consuming hyperparameter optimization of deep learning network models, the inventors found that the main influencing factors include the following two aspects: First, data: For deep networks, the running time of the network is linearly positively correlated with the resolution of the image. The resolution of industrial detection images is high, so the network needs to spend more time to calculate. At the same time, there is redundancy in image samples. For general samples or samples that are easy to fit, the conventionally used unoptimized hyperparameter combination can also obtain better detection results for model training and prediction, while the most representative important samples and difficult samples require the configuration of the optimal hyperparameter combination to improve the detection performance of the model; second, the network structure: the network structure is not lightweight enough, no special optimization design is performed, the calculation amount is large, and the real-time performance is poor, resulting in too long time when using hyperparameter search tools for optimization.

[0052] Therefore, based on the analysis of the above two points, the inventor believes that the optimization design of data and model is the key. The present invention optimizes the design from the two perspectives of data and model, and obtains difficult samples based on active learning to optimize hyperparameters. In some embodiments, the structure of the defect segmentation model is optimized for defect detection tasks so that the model can be lightweight. These optimization designs can greatly reduce the time consumption of the hyperparameter optimization process, obtain the optimal hyperparameter combination, and improve the detection performance of the model. The core idea is: based on active learning, select the most important difficult samples that are difficult to fit the model, greatly reduce the amount of sample data used in the hyperparameter optimization process, and achieve efficient and accurate acquisition of the optimal hyperparameter combination. In addition, in some embodiments, for defect detection tasks, since industrial images have high resolution and defect areas are few in number and small in area, partial images of defective areas are cut out and used as actual training sample images, which significantly reduces the computational complexity of the model; in some embodiments, an optimized model structure is adopted to reduce the computational complexity and provide excellent real-time performance; in some embodiments, the optimization of both data and model is combined to work together to greatly reduce the time consumption of the hyperparameter optimization process, and obtain the optimal hyperparameter combination in a short time to improve the detection performance of the defect segmentation model and reduce training time, thereby achieving cost reduction and efficiency improvement.

[0053] Based on the hyperparameter optimization method of the present invention, the process of a complete image processing task is divided into two stages. The first stage is to use the hyperparameter optimization method of the present invention to search for hyperparameters, and obtain the optimal hyperparameter combination based on the optimization of data and / or models; the second stage is to use the obtained optimal hyperparameter combination to complete the high-quality training of the network model, improve the detection performance, and finally use the trained network model to perform the corresponding image processing task. For the entire process, the first stage and the second stage are executed in series. After the first stage is completed and the optimal hyperparameter combination is obtained, the training and detection process of the second stage can be run, and finally the performance improvement of the network model can be achieved.

[0054] The technical solution of the present invention is introduced below. Figure 1 , Figure 1 This is a schematic diagram of the overall processing flow of the hyperparameter optimization method proposed in the present invention, wherein the sample image set used includes an image set X and a corresponding annotation set Y, and each sample image in the image set X has a corresponding annotation in the annotation set Y. Figure 1Several functional modules or program modules are shown in FIG. 1 , where F is a deep learning model (called “target task model”) that will be used to perform the target task. The management module P is a module that implements some processing steps related to hyperparameters in the entire processing flow. Its functions mainly include two aspects: one is to manage the hyperparameters of the target task model F and all its combinations, and provide the hyperparameter combination para to the target task model F for training and trial in an active learning manner; the other is to record the performance of the target task model F under all hyperparameter combinations para, and finally select the hyperparameter combination corresponding to the best performance of the target task model F as the optimal hyperparameter combination best para. The management module P outputs each of the preset multiple hyperparameter combinations to the target task model F, and the target task model F receives the sample image and the corresponding annotation (x in the sample image set). train ,y train ) and the hyperparameter combination para output by the management module P are trained, and the sample image set is updated based on the active learning method according to the training status. Then, at the end of the training of each hyperparameter combination, the management module P performs a performance evaluation eval on the trained target task model F, and records the hyperparameter combination and performance indicators used each time. Finally, after the training attempts for all hyperparameter combinations are completed, the corresponding hyperparameter combination is output according to the best performance indicator as the best hyperparameter combination best para for the current task, and the hyperparameter optimization process is completed. Then, the best hyperparameter combination bestpara is used to perform the final training on the target task model F, that is, the training for executing the target task is completed.

[0055] like Figure 1 As shown, in some embodiments, the sample image set is divided into a training set (x train ,y train ) and validation set (x val ,y val ), training set (x train ,y train ) is used for searching and trying hyperparameter optimization, and the validation set (x val ,y val ) is used to select hyperparameter combinations. Specifically, the target task model F uses the training set (x train ,y train ) for training, based on active learning, according to the training status, update the training set (x train ,y train ); At the end of the training of each hyperparameter combination, the management module P uses the validation set (x val ,y val) performs performance evaluation eval and records the hyperparameter combination and performance indicators used each time; finally, after the training attempts for all hyperparameter combinations are completed, the best performance indicator is determined, the corresponding hyperparameter combination is output, and the best hyperparameter combination best para is obtained.

[0056] Please refer to Figure 2 In some embodiments, for the defect detection task (i.e., the target task model F is a defect segmentation model, the sample image includes defect areas, the annotation is specifically a defect segmentation annotation map, the defect segmentation annotation map has a defect marking area, and the area in the sample image corresponding to the position of the defect marking area in the corresponding defect segmentation annotation map is the defect area), a preprocessing module Q is also designed, and its main function is to intercept the area of ​​interest from the sample image set for subsequent processing. In the preprocessing module Q, the original defect segmentation annotation map in the original annotation set Y is intercepted, and the partial image including the defect marking area is intercepted to obtain a new defect segmentation annotation map, and these new defect segmentation annotation maps constitute a new annotation set Y1; for the original sample image in the original image set X, the area corresponding to the partial image position intercepted from the corresponding original defect segmentation annotation map is intercepted to obtain a new sample image, and these new sample images constitute a new image set X1. When performing hyperparameter optimization, the defect segmentation model F accepts sample images in the new image set X1 and the corresponding defect segmentation annotation images in the new annotation set Y1, as well as the hyperparameter combination para output by the management module P. Based on the active learning method, the sample image set (X1, Y1) is updated according to the training status. At the end of the training of each hyperparameter combination, the management module P performs a performance evaluation eval on the trained defect segmentation model F, and records the hyperparameter combination and defect segmentation performance used each time. Finally, after the training attempts for all hyperparameter combinations are completed, the hyperparameter combination corresponding to the best defect segmentation performance is selected as the best hyperparameter combination best para for the current task. Then, the best hyperparameter combination bestpara is used to perform the final defect segmentation training on the defect segmentation model F. During inference (i.e., when performing defect detection), the image of the object to be detected is input into the defect segmentation model F, and the defect segmentation model F outputs the defect segmentation result.

[0057] Similarly, the new sample image set (X1, Y1) obtained after preprocessing by the preprocessing module Q can also be divided into a training set (x train ,y train ) and validation set (x val ,y val ), training set (x train ,y train ) is used for searching and trying hyperparameter optimization, and the validation set (x val ,y val) is used to select the best hyperparameter combination.

[0058] Please refer to Figure 3 The hyperparameter optimization method in some embodiments of the present invention specifically includes steps 1100 to 1300, which are described below respectively.

[0059] Step 1100: Obtain a sample image set.

[0060] The sample image set includes multiple sample images and corresponding annotations. Depending on the task, the annotation content is different. For example, for target detection, the position and size of the target box of the target in the sample image can be annotated. For defect detection, the defect area in the sample image can be annotated.

[0061] Step 1200: For each hyperparameter combination among the preset multiple hyperparameter combinations, use the hyperparameter combination to train the target task model using the sample image set, and evaluate the performance indicators of the trained target task model when performing the target task; record the performance indicators of the target task model after training using each hyperparameter combination when performing the target task.

[0062] The target task model is a model for executing the target task. The target task can be target detection, defect detection, etc. Technical personnel in this field can set the structure of the target task model and the performance indicators used in the evaluation according to the task type and actual detection needs.

[0063] Hyperparameter combinations are saved after being pre-set. For example, if the learning rate is 0.3 and 0.6, and the batch size is 30 and 80, then the hyperparameter combinations of [learning rate, batch size] are [0.3,30], [0.3,80], [0.6,30], and [0.6,80]. You can combine various hyperparameter search methods, such as grid search, random search, and search based on Bayesian optimization, to set hyperparameter combinations.

[0064] In the deep learning training process, the sample image set will be reused for multiple rounds of training. In the present invention, in the process of using each hyperparameter combination to train the target task model using the sample image set, the sample image set is updated based on the prediction result of the target task model in an active learning manner, that is, if the prediction error of the target task model for any sample image is less than the second error threshold, the sample image and the corresponding annotation are discarded and no longer participate in subsequent rounds of training, which is equivalent to reducing the samples in the sample image set. The prediction error can be determined based on the difference between the prediction result of the target task model for the sample image and the corresponding annotation.

[0065] In essence, this is based on active learning, selecting the most important and difficult samples for training and fitting, reducing the number of samples, and speeding up the time for hyperparameter optimization search. The model is trained on the most important and difficult samples, which can achieve efficient and accurate acquisition of the optimal hyperparameter combination and improve model performance.

[0066] In some embodiments, the method of using a sample image set to train a target task model and evaluating the performance indicators of the trained target task model when performing the target task includes: dividing the sample image set into a training set and a validation set according to a preset ratio (for example, the ratio of training set to validation set is 7:3), using the training set to train the target task model, and after the training, using the validation set to evaluate the performance indicators of the trained target task model when performing the target task. It can be understood that at this time, based on the active learning method, what is discarded are the sample images and corresponding annotations in the training set, that is, what is updated is the training set. After the training is completed, the parameters of the target task model are fixed, the sample images in the validation set are received as input, the prediction results are output, and the performance indicators of the trained target task model are evaluated based on the prediction results and the corresponding annotations.

[0067] Step 1300: Select the hyperparameter combination corresponding to the trained target task model with the best performance index as the hyperparameter combination best para for training the target task model. Subsequently, the target task model will use best para as the hyperparameter combination for training, which can complete high-quality training in a short time and obtain excellent performance.

[0068] Please refer to Figure 4 The hyperparameter optimization method of the present invention is applied to defect detection tasks, and the process of training the defect segmentation model includes steps 2100 to 2300, which are described in detail below.

[0069] Step 2100: Acquire a first sample image set.

[0070] The first sample image set includes a plurality of first sample images and corresponding defect segmentation annotations, where the defect segmentation annotations are used to indicate defect areas in the first sample images.

[0071] The defect segmentation annotation is usually a defect segmentation annotation map with the same resolution as the sample image. Each pixel in the defect segmentation annotation map indicates whether the corresponding pixel in the sample image is a defect. If there are multiple different types of defects, it can also indicate what kind of defect the corresponding pixel in the sample image belongs to. Specifically, different pixel values ​​or colors can be used to mark the background and (different types of) defects. The pixels marked as defects constitute the defect marking area. Therefore, the area in the sample image corresponding to the defect marking area in the corresponding defect segmentation annotation map is the defect area. On this basis, the first sample image set includes multiple first sample images and the corresponding first defect segmentation annotation map. The first defect segmentation annotation map has a defect marking area, and the area in the first sample image corresponding to the defect marking area in the corresponding first defect segmentation annotation map is the defect area.

[0072] Step 2200: For each hyperparameter combination among the preset multiple hyperparameter combinations, use the hyperparameter combination to train the defect segmentation model using the first sample image set, and evaluate the defect segmentation performance of the trained defect segmentation model; record the defect segmentation performance of the defect segmentation model after training using each hyperparameter combination.

[0073] For setting of the hyperparameter combination, please refer to step 120. The defect segmentation performance can be evaluated by using indicators such as pixel segmentation accuracy and average IoU (Intersection over Union). Those skilled in the art can set the structure of the defect segmentation model and the indicators used in the evaluation according to actual detection needs.

[0074] In the process of using each hyperparameter combination and the first sample image set to train the defect segmentation model, if the defect segmentation error of the defect segmentation model for any first sample image is less than the first error threshold, the first sample image and the corresponding defect segmentation annotation are discarded and no longer participate in subsequent rounds of training. The first sample image is input into the defect segmentation model to obtain the corresponding defect segmentation result. It can be understood that the defect segmentation result includes the classification result of whether each pixel in the first sample image is a defective pixel, and the defect segmentation error can be determined based on the difference between the defect segmentation result of the defect segmentation model for the first sample image and the corresponding defect segmentation annotation. For example, the number of pixels predicted to be wrong can be used as the defect segmentation error.

[0075] In some embodiments, the method of training the defect segmentation model using the first sample image set and evaluating the defect segmentation performance of the defect segmentation model after the training includes: dividing the first sample image set into a training set and a validation set according to a preset ratio (for example, the ratio of training set to validation set is 7:3), using the training set to train the defect segmentation model, and after the training, using the validation set to evaluate the defect segmentation performance of the defect segmentation model after the training. At this time, what is discarded based on the active learning method is the first sample image in the training set and the corresponding defect segmentation annotation, that is, what is updated is the training set. After the training is completed, the parameters of the defect segmentation model are fixed, the first sample image in the validation set is received as input, and the defect segmentation result is output; the defect segmentation performance of the defect segmentation model after the training is evaluated based on the defect segmentation result and the corresponding defect segmentation annotation.

[0076] Step 2300: Select the hyperparameter combination corresponding to the trained defect segmentation model with the best defect segmentation performance as the final hyperparameter combination best para, and use the final hyperparameter combination best para to train the defect segmentation model. This can complete high-quality training in a shorter time and obtain a trained defect segmentation model with excellent defect segmentation performance.

[0077] Please refer to Figure 5 In some embodiments, step 2100 includes steps 2110 to 2140, which are described in detail below.

[0078] Step 2110: Obtain an original sample image set.

[0079] The original sample image set includes multiple original sample images and corresponding original defect segmentation and annotation maps. The original defect segmentation and annotation map has a defect marking area. The area in the original sample image corresponding to the defect marking area in the corresponding original defect segmentation and annotation map is the defect area.

[0080] The original sample image here may be an image obtained by imaging the product using a high-resolution industrial camera, and has a high resolution.

[0081] Step 2120: Cut out a partial image including a defect marking area in the original defect segmentation and annotation map to obtain a first defect segmentation and annotation map; cut out an area in the original sample image corresponding to the first defect segmentation and annotation map cut out from the corresponding original defect segmentation and annotation map to obtain a first sample image corresponding to the first defect segmentation and annotation map.

[0082] By intercepting the original defect segmentation annotation map and the original sample image, the resolution of both can be reduced, thereby reducing the amount of calculation and accelerating the training of the model. When intercepting, it is only necessary to ensure that the first defect segmentation annotation map includes the defect marking area of ​​the original defect segmentation annotation map (that is, the area of ​​interest of the original defect segmentation annotation map) and the first sample image includes the defect area of ​​the original sample image (that is, the area of ​​interest of the original sample image). The specific size of the image to be intercepted can be determined according to the computing power of the processor, the efficiency requirements of training, etc.

[0083] In one embodiment, cutting out a partial image including a defect mark area in the original defect segmentation and annotation image specifically includes: determining a minimum bounding rectangle of the defect mark area, and cutting out an image of the minimum bounding rectangle portion of the original defect segmentation and annotation image.

[0084] Step 2130: unifying the resolutions of all first sample images and the first defect segmentation annotation map. The resolutions of all first sample images and the first defect segmentation annotation map may be unified by an interpolation operation.

[0085] Step 2140: All first sample images and corresponding first defect segmentation annotation images form the first sample image set.

[0086] In one embodiment, when the defect segmentation model is finally trained using the final hyperparameter combination best para in step 2300, the original sample image set is used.

[0087] At present, in actual industrial product inspection, in order to ensure the quality of products, manufacturers generally use high-resolution industrial cameras to image products to maximize the detection accuracy, but this leads to a large resolution of the image, for example, the resolution can reach 8k*10k. Using these high-resolution images to complete the hyperparameter optimization search of the defect segmentation model is almost unacceptable. Even if the search method based on Bayesian optimization with a slightly smaller amount of calculation is used, it is time-consuming and not very practical. In response to this pain point problem, the inventor believes after analysis that for high-resolution industrial images, the number of defective areas is small, and the area of ​​the defective areas is small. Compared with the resolution of 8k*10k, the height and width of most defective areas are only one thousandth or even one ten-thousandth of the original image, and the defect segmentation model is difficult to fit not a large area of ​​normal areas, but a few small areas of defective areas. Therefore, in this embodiment, by extracting partial images of defective areas and corresponding defect segmentation annotations, the amount of sample data is reduced; in conjunction with the active learning method, the hyperparameter optimization search process of the defect segmentation model is further accelerated, so that the time consumption of hyperparameter optimization is greatly reduced. In addition, the hyperparameter optimization method and the defect segmentation model training method of the present invention can be combined with various hyperparameter search methods, such as grid search, random search, search based on Bayesian optimization, etc., to achieve high compatibility, high efficiency, and high precision optimization search, and obtain the best hyperparameter optimization results required for the target task model in a short time.

[0088] Based on the above defect segmentation model training method, the present invention provides a defect detection method, such as Figure 6 As shown, the defect detection method in some embodiments includes the following steps:

[0089] Step 3100: Acquire an image of the detected object;

[0090] Step 3200: Input the image of the inspected object into the pre-trained defect segmentation model to obtain the defect region in the image of the inspected object. The position of the defect on the inspected object can be determined based on the defect region in the image of the inspected object.

[0091] The detected objects here can be products on an industrial assembly line, mechanical parts in an object box, tools on an operating table, etc., without specific limitation. The defect segmentation model is trained by the above-mentioned defect segmentation model training method.

[0092] The above embodiments are optimizations in terms of data. Some embodiments of the present invention optimize from the perspective of the model, that is, optimize the structure of the defect segmentation model, achieve lightweight defect segmentation model without reducing defect segmentation performance, greatly reduce the time consumption of hyperparameter optimization process, and enhance the real-time performance of the network model. The optimized defect segmentation model and the relevant steps of the defect detection method when using the model are specifically described below. For the convenience of description, unless otherwise specified, "Conv3x3" refers to a 3×3 standard convolution block, which consists of a convolution layer with a convolution kernel size of 3x3 and a step size of 1x1, plus a normalization layer (e.g., a batch normalization layer) and an activation function (e.g., a ReLU function). "Conv1x1" refers to a 1×1 standard convolution block, which consists of a convolution layer with a convolution kernel size of 1×1 and a step size of 1×1, plus a normalization layer (e.g., a batch normalization layer) and an activation function (e.g., a ReLU function).

[0093] Please refer to Figure 7 The defect segmentation model provided by the present invention includes an encoder B, a decoder D and a classification head module C. The encoder B is responsible for extracting the multi-scale features of the image, the decoder D is responsible for decoding, and the classification head module C completes the final pixel-by-pixel classification task to predict whether each pixel is a defect. On this basis, please refer to Figure 8 , step 3200 specifically includes:

[0094] Step 3210: Input the image of the detected object into encoder B for multi-scale feature extraction to obtain multiple layers of first feature maps with different resolutions; the multiple layers of first feature maps with different resolutions can be feature maps output by multiple layers of encoder B, and the higher the number of layers, the lower the resolution;

[0095] Step 3220: Input the first feature maps of multiple layers with different resolutions into the decoder D for fusion, and use the fused features for dimension upgrading and information recovery to obtain a fused feature map d;

[0096] Step 3230: Perform pixel-level classification prediction using the fused feature map d through the classification head module C to obtain a defect segmentation result r, where the defect segmentation result r includes a classification result of whether each pixel in the image of the inspected object is a defective pixel;

[0097] Step 3240: Determine the defective area based on the defective pixels, that is, the area composed of defective pixels is the defective area.

[0098] The defect segmentation model provided by the present invention is composed of only an encoder B, a decoder D and a classification head module C, has a relatively simple structure, and can achieve lightweight model, thereby improving the calculation speed and reducing the time consumption of the hyperparameter optimization process.

[0099] In some embodiments, in view of the characteristics of industrial images with high resolution and small defects, it is necessary to focus on the extraction of position information, and the demand for a large receptive field is small. Therefore, only the first feature maps f1, f2 and f3 of the three layers are retained (such as Figure 7 As shown), while reducing the amount of calculation, the defect segmentation performance is improved. Based on the comprehensive consideration of defect segmentation performance and calculation amount, encoder B can adopt a lightweight network recognized in the field, such as the backbone network Backbone of ResNet-18. The backbone network Backbone of ResNet-18 has strong feature extraction capabilities, and the model size is relatively light, which meets the dual requirements of defect segmentation effect and calculation amount. Here, only the backbone network Backbone of ResNet-18 is used as an example, and other similar lightweight networks with the same effect can be used.

[0100] The decoder D may also adopt a lightweight design. In some embodiments, the decoder D includes a plurality of standard convolution blocks (e.g., Conv1x1) and upsampling operations that can reduce the number of channels of the feature map without changing the resolution. Through the processing of these standard convolution blocks that can reduce the number of channels of the feature map without changing the resolution, the amount of data of the feature map is reduced, thereby reducing the amount of calculation of the model. The upsampling operation can increase the dimension of the feature map, and in conjunction with the convolution layer, information recovery can be achieved.

[0101] The classification head module C is used to perform pixel-level classification prediction using the fused feature map d to obtain the defect segmentation result r. It can include only one standard convolution block (such as Conv1x1) to reduce the number of parameters.

[0102] Please refer to Fig. 9 In some embodiments, the decoder D includes a first convolutional layer and a self-attention module Attention, on this basis, such as Fig.10 As shown, step 3220 specifically includes steps 3221 to 3223, which are described in detail below.

[0103] Step 3221: unify the first feature maps of multiple layers with different resolutions to the resolution size of the first feature map with the largest resolution, and perform channel splicing to obtain a spliced ​​feature map f.

[0104] by Figure 7 Taking the three-layer first feature maps f1, f2 and f3 in the example, the first feature map f1 has the largest resolution, so the first feature maps f2 and f3 are upsampled (Up) to the resolution size of the first feature map f1 to unify the resolutions of the three.

[0105] Step 3222: Input the concatenated feature map f into the first convolutional layer for convolution processing to obtain a first multi-category score map p.

[0106] The first convolutional layer needs to be set so that the number of channels of the output first multi-class score map p is the same as the number of preset classification categories. There are multiple classification categories, at least one for representing defects and one for representing background (i.e., non-defects). When multiple different types of defects need to be represented, there can be multiple classification categories for representing defects, each representing a defect type. The first convolutional layer can use Conv1x1.

[0107] The first multi-category score map p obtained here is equivalent to a preliminary predicted score map in the middle of the whole process, each channel of which corresponds to a classification category, and the value on each channel represents the probability of the corresponding classification category. Fig. 9 The dotted line in the figure indicates that the decoder D will output the first multi-category score map p during training, and the corresponding defect segmentation annotation will be used to train the defect segmentation model under the control of the loss function L2. It is an auxiliary branch during training and will be discarded during inference and will not participate in the calculation.

[0108] Step 3223: Using the self-attention module Attention, use the first multi-category score map p to enhance the context information of the spliced ​​feature map f to obtain the fused feature map d.

[0109] The self-attention module Attention is based on the self-attention mechanism, which can capture the relationship or similarity between each pixel in the image and other pixels, thereby enhancing the context information of each pixel, that is, it can capture the global dependency in the image, not just the local dependency, thereby improving the accuracy of defect segmentation. The first multi-class score map p can provide category information.

[0110] In some embodiments, step 3223 specifically includes: using the first multi-category score map p to perform target enhancement on the spliced ​​feature map f to obtain a target enhanced feature map fp; inputting the spliced ​​feature map f and the target enhanced feature map fp into the self-attention module Attention for processing by the self-attention mechanism to obtain a fused feature map d, wherein the spliced ​​feature map f is used to form a query matrix in the self-attention mechanism, and the target enhanced feature map fp is used to form a key matrix and a value matrix in the self-attention mechanism.

[0111] Among them, you can choose a suitable target enhancement method according to actual needs. The main purpose of target enhancement is to enhance the feature representation of the target area (i.e. defect area) in the image. Please refer to Fig.11 In some embodiments of the present invention, the first multi-category score map p is used to perform target enhancement on the spliced ​​feature map f to obtain the target enhanced feature map fp, which specifically includes steps 3223a and 3223b. Fig.12 Provide detailed explanation.

[0112] Step 3223a: Perform a shape transformation on the concatenated feature map f to form a two-dimensional matrix F, and perform a shape transformation on the first multi-category score map p to form a two-dimensional matrix P, and the two-dimensional matrix F and the two-dimensional matrix P can perform matrix multiplication.

[0113] It can be understood that an image is a three-dimensional data with three dimensions of height, width and number of channels. Shape transformation refers to changing the height, width and / or feature dimensions (i.e., number of channels) so that the shape or size of the image changes. The main purpose of performing shape transformation on the spliced ​​feature map f and the first multi-class score map p here is to convert the spliced ​​feature map f and the first multi-class score map p into a two-dimensional matrix and make their sizes meet the requirements of matrix multiplication, so as to facilitate matrix multiplication and achieve target enhancement. Any shape transformation method that can achieve this purpose is acceptable.

[0114] Fig.12 A specific transformation method is shown, where the shape of the concatenated feature map f is [batch, h, w, channel], where batch is the batch size of the batch processing, h is the height, w is the width, and channel is the number of channels. The shape of the first multi-class score map p is [batch, h, w, classes], where classes is the number of classification categories. Fig.12 As shown in the figure, the concatenated feature map f is transformed into a two-dimensional data [batch, h×w, channel] by performing Reshape (shape reorganization) and Permute (dimension rearrangement) operations, thereby obtaining a two-dimensional matrix F. The first multi-category score map p is changed to [batch, classes, h×w] by performing the Reshape operation, and a two-dimensional data is obtained at this time, and then the Softmax operation is performed on the h×w dimension to obtain a two-dimensional matrix P.

[0115] Step 3223b: Multiply the two-dimensional matrix F and the two-dimensional matrix P, perform shape transformation on the multiplication result to restore it to image data, and obtain the target enhanced feature map fp.

[0116] like Fig.12 In the specific implementation shown, the two-dimensional matrix F and the two-dimensional matrix P are multiplied (Mul), and the data shape is transformed into [batch, classes, channel], and then the data shape is modified to [batch, channel, classes, 1] through the Permute and Unsqueeze (dimension expansion) operations to restore the three-dimensional image data to obtain the target enhanced feature map fp. Finally, the spliced ​​feature map f and the target enhanced feature map fp are input into the self-attention module Attention for processing, and the fused feature map d is output.

[0117] In essence, the target enhancement feature map fp is the pixel-by-pixel similarity calculation of the spliced ​​feature map f and different classification categories, that is, the enhanced representation of the target area. The target is enhanced first, and then the obtained target enhancement feature map and the spliced ​​feature map are input into the self-attention module for self-attention processing. The purpose is to obtain the feature representation of the target area by aggregating the feature representation of the pixels in the corresponding target area, and then perform fine pixel-level processing through the self-attention module. The core idea is to perform "region first, then pixel", that is, "from coarse to fine" optimization process, which makes network training easier and more stable. In contrast, directly inputting the first multi-category score map and the spliced ​​feature map into the self-attention module for processing is equivalent to directly giving pixel-level guidance. The network lacks the optimization process of transitioning from region to pixel, and the training is more difficult and the stability is poor.

[0118] The core of the self-attention module Attention is the self-attention mechanism, please refer to Fig.13 In some embodiments, the structure of the self-attention module Attention is lightweight and has few parameters. It only includes four network layers with parameters, namely the second convolution layer, the third convolution layer, the fourth convolution layer and the fifth convolution layer. The second convolution layer, the third convolution layer and the fourth convolution layer can reduce the number of channels of the feature map. On this basis, Fig.14 As shown, the concatenated feature map f and the target enhanced feature map fp are input into the self-attention module Attention for processing by the self-attention mechanism to obtain the fused feature map d, which specifically includes steps 3223c to 3223h, which are described in detail below.

[0119] Step 3223c: Reduce the number of channels of the concatenated feature map f through the second convolutional layer to obtain the second convolutional feature map, reduce the number of channels of the target enhancement feature map fp through the third convolutional layer to obtain the third convolutional feature map, and reduce the number of channels of the target enhancement feature map fp through the fourth convolutional layer to obtain the fourth convolutional feature map.

[0120] The second convolutional layer, the third convolutional layer, and the fourth convolutional layer may all adopt Conv1x1.

[0121] Step 3223d: Transform the shape of the second convolutional feature map to form a query matrix M q , transform the shape of the third convolutional feature map to form a key matrix M k , where the query matrix M q and bond matrix M k Capable of matrix multiplication.

[0122] The main purpose of performing shape transformation on the second convolution feature map and the third convolution feature map is to convert the second convolution feature map and the third convolution feature map into a two-dimensional matrix, which are used as query matrices M and M respectively. q and bond matrix Mk , and make their sizes meet the requirements for matrix multiplication. Please refer to Fig.13 In a specific implementation, the second convolutional feature map is transformed into a two-dimensional matrix by performing Permute and Reshape operations to form a query matrix M q ; The third convolution feature map is transformed into a two-dimensional matrix by performing a Reshape operation to form a key matrix M k .

[0123] Step 3223e: Query matrix M q and bond matrix M k Multiply, perform Softmax processing on the result of the multiplication, and obtain the relationship matrix M r .

[0124] Step 3223f: Transform the shape of the fourth convolutional feature map to form a value matrix M v , where the value matrix M v and the relationship matrix M r Capable of matrix multiplication.

[0125] like Fig.13 In the specific implementation shown, the fourth convolution feature map is transformed into a two-dimensional matrix by performing Reshape and Permute operations to form a value matrix M v .

[0126] Step 3223g: Set the value matrix M v and the relationship matrix M r The result of the multiplication is transformed and restored to the image data with the same resolution as the spliced ​​feature map f, and then restored to the same number of channels as the spliced ​​feature map f through the fifth convolution layer to obtain the fifth convolution feature map. The fifth convolution layer can use Conv1x1.

[0127] like Fig.13 In the specific implementation shown, the fifth convolutional feature map is restored to image data with the same resolution as the spliced ​​feature map f by performing Permute and Reshape operations, and then restored to the same number of channels as the spliced ​​feature map f through the fifth convolutional layer.

[0128] Step 3223h: perform an element-wise addition operation (Add) on the fifth convolutional feature map and the concatenated feature map f to obtain a fused feature map d. The number of channels of the fused feature map d is consistent with that of the concatenated feature map f.

[0129] In essence, the self-attention module Attention uses the self-attention mechanism to first calculate the relationship matrix M between the concatenated feature map f and the target enhanced feature map fp r , then the relationship matrix M rand value matrix M v Perform the matrix multiplication Mul operation, and restore the number of channels through Permute and Reshape operations and convolutional layers to obtain the target context representation. Finally, perform element-level addition operation with the concatenated feature map f to output the fused feature map d.

[0130] In some embodiments, the classification head module C includes a sixth convolution layer, and the defect segmentation result includes a defect segmentation result map. The pixel value of the pixel in the defect segmentation result map is used to indicate whether the corresponding pixel in the image of the object to be detected is a defective pixel, similar to the defect segmentation annotation map. On this basis, step 3230 specifically includes: inputting the fused feature map d into the sixth convolution layer for convolution processing, and then upsampling to the resolution of the image of the object to be detected, to obtain a second multi-class score map c, the number of channels of the second multi-class score map c is the same as the number of preset classification categories, and the values ​​of each channel at each pixel position in the second multi-class score map c respectively represent the probability that the corresponding pixel in the image of the object to be detected belongs to each classification category, similar to the first multi-class score map p; for each pixel in the second multi-class score map c, take the classification category corresponding to the channel with the largest value as the final classification category to obtain the defect segmentation result map.

[0131] During the training process (such as in steps 2200 and 2300), the loss function of the defect segmentation model needs to be used, such as Figure 7 As shown, in some embodiments, the loss function Loss is composed of a first loss function L1 and a second loss function L2. The first loss function L1 represents the difference between the defect segmentation result image r and the defect segmentation annotation, and the second loss function L2 represents the difference between the first multi-category score image p obtained by inputting the first sample image into the defect segmentation model and the defect segmentation annotation.

[0132] In some embodiments, according to the characteristics of high resolution and small defect area of ​​industrial images, the first loss function L1 and the second loss function L2 adopt the intersection-over-union loss function, which is a region-related loss function, is good at mining the foreground area (i.e., defect area) information, and is conducive to the detection of small targets. Then the expression of the first loss function is:

[0133] L1=1-2×IoU1,

[0134] Where IoU1 represents the intersection-over-union ratio between the defect segmentation result image r and the defect segmentation annotation, which can be specifically Where label represents the defect segmentation label, ∩ represents the intersection, and ∪ represents the union.

[0135] The first multi-class score map p is the predicted score map in the middle of the whole process. It is necessary to obtain the intermediate segmentation result based on it to compare with the defect segmentation annotation. The intermediate segmentation result is obtained by upsampling the first multi-class score map p (expressed as Up(p)) to the resolution size of the image of the object to be detected. For each pixel after upsampling, the classification category corresponding to the channel with the largest value (expressed as argmax(Up(p))) is taken to obtain the intermediate segmentation result. The expression of the second loss function is:

[0136] L2=1-2×IoU2,

[0137] The IoU2 represents the intersection-over-union ratio between the intermediate segmentation result and the defect segmentation annotation, which can be

[0138]

[0139] In some embodiments, the loss function of the defect segmentation model is expressed as:

[0140] Loss=δ1L1+δ2L2,

[0141] δ1 and δ2 are preset weight coefficients, which can be designed according to actual needs. In one embodiment, δ1=1.0 and δ2=0.4.

[0142] The above is an introduction to model optimization. Data optimization and model optimization can be used separately or in combination. In one embodiment, the defect segmentation model provided by the present invention is used, and based on Figure 4 The method shown in FIG. 1 is used to obtain the optimal hyperparameter combination, and the defect segmentation model is trained with the optimal hyperparameter combination to obtain the defect segmentation model F after training. During the inspection, the image of the inspected object is obtained, and the defect segmentation result obtained by inputting the defect segmentation model F is as follows: Fig.15 As shown, the left picture is the image of the inspected object, and the right picture is the defect segmentation result.

[0143] Based on the above hyperparameter optimization method, the present invention also provides a hyperparameter optimization device, please refer to Fig.16 In some embodiments, the device includes a sample acquisition module 11, a hyperparameter search module 12 and a hyperparameter determination module 13.

[0144] The sample acquisition module 11 is used to acquire a sample image set, which includes a plurality of sample images and corresponding annotations.

[0145] The hyperparameter search module 12 is used to use each hyperparameter combination in the preset multiple hyperparameter combinations, use the hyperparameter combination to train the target task model using the sample image set, and evaluate the performance index of the trained target task model when performing the target task; and record the performance index of the target task model when performing the target task after using each hyperparameter combination for training. In the process of training the target task model using the sample image set, if the prediction error of the target task model for any sample image is less than the second error threshold, the sample image is discarded and no longer participates in subsequent rounds of training.

[0146] The hyperparameter determination module 13 is used to select the hyperparameter combination corresponding to the trained target task model with the best performance index as the hyperparameter combination finally used for training the target task model.

[0147] Based on the above defect detection method, the present invention also provides a defect detection device, please refer to Fig.17 In some embodiments, the device includes an image acquisition module 210, a defect segmentation module 220 and a training module 230, which are described below respectively.

[0148] The image acquisition module 210 is used to acquire an image of the detected object.

[0149] The defect segmentation module 220 is used to input the image of the inspected object into a pre-trained defect segmentation model to obtain the defect area in the image of the inspected object.

[0150] The training module 230 is used to train the defect segmentation model, such as Fig.17 As shown, the training module 230 includes a sample acquisition submodule 231, a hyperparameter search submodule 232, and an optimization training submodule 233. The sample acquisition submodule 231 is used to acquire a first sample image set, the first sample image set includes a plurality of first sample images and corresponding defect segmentation annotations, and the defect segmentation annotations are used to indicate defect areas in the first sample images. In some embodiments, the sample acquisition submodule 231 is specifically used to perform Figure 5 The process shown is to obtain a first sample image set.

[0151] The hyperparameter search submodule 232 is used to use each hyperparameter combination in the preset multiple hyperparameter combinations, use the hyperparameter combination to train the defect segmentation model using the first sample image set, and evaluate the defect segmentation performance of the defect segmentation model after training; record the defect segmentation performance of the defect segmentation model after training using each hyperparameter combination. In the process of training the defect segmentation model using the first sample image set, if the defect segmentation error of the defect segmentation model for any first sample image is less than the first error threshold, the first sample image is discarded and no longer participates in subsequent rounds of training. Please refer to step 2200 for details.

[0152] The optimization training submodule 233 is used to select the hyperparameter combination corresponding to the trained defect segmentation model with the best defect segmentation performance as the final hyperparameter combination, and use the final hyperparameter combination to train the defect segmentation model to obtain a trained defect segmentation model.

[0153] When using Figure 7 In the defect segmentation model shown, the defect segmentation module 220 is specifically used to: input the image of the detected object into the encoder for multi-scale feature extraction to obtain multiple layers of first feature maps with different resolutions; input multiple layers of first feature maps with different resolutions into the decoder for fusion, and use the fused features for dimension upgrading and information recovery to obtain a fused feature map; use the fused feature map to perform pixel-level classification prediction through the classification head module to obtain a defect segmentation result, which includes a classification result of whether each pixel in the image of the detected object is a defective pixel; determine the defect area based on the defective pixel. Please refer to the above introduction for the defect segmentation model.

[0154] In some embodiments, the decoder of the defect segmentation model includes a first convolutional layer and a self-attention module. The defect segmentation module 220 inputs multiple layers of first feature maps of different resolutions into the decoder for fusion, and uses the fused features for dimensionality increase and information recovery to obtain a fused feature map, including: unifying multiple layers of first feature maps of different resolutions to the resolution size of the first feature map with the largest resolution, and performing channel splicing to obtain a spliced ​​feature map; inputting the spliced ​​feature map into the first convolutional layer for convolution processing to obtain a first multi-category score map, the number of channels of the first multi-category score map is the same as the number of preset classification categories; using the self-attention module, using the first multi-category score map to enhance the context information of the spliced ​​feature map to obtain a fused feature map. Please refer to steps 3221 to 3223 above for details.

[0155] In some embodiments, the defect segmentation module 220 uses a self-attention module and adopts the first multi-category score map to perform context information enhancement on the spliced ​​feature map to obtain a fused feature map, including: using the first multi-category score map to perform target enhancement on the spliced ​​feature map to obtain a target enhanced feature map; inputting the spliced ​​feature map and the target enhanced feature map into the self-attention module for processing by the self-attention mechanism to obtain a fused feature map, wherein the spliced ​​feature map is used to form a query matrix in the self-attention mechanism, and the target enhanced feature map is used to form a key matrix and a value matrix in the self-attention mechanism.

[0156] In some embodiments, the defect segmentation module 220 uses the first multi-class score map to perform target enhancement on the spliced ​​feature map to obtain the target enhanced feature map, including: performing shape transformation on the spliced ​​feature map to form a two-dimensional matrix F, performing shape transformation on the first multi-class score map to form a two-dimensional matrix P, and the two-dimensional matrix F and the two-dimensional matrix P can be matrix multiplied; multiplying the two-dimensional matrix F and the two-dimensional matrix P, and performing shape transformation on the multiplication result to restore it to image data to obtain the target enhanced feature map. Please refer to steps 3223a and 3223b above for details.

[0157] In some embodiments, the self-attention module includes a second convolution layer, a third convolution layer, a fourth convolution layer, and a fifth convolution layer; the defect segmentation module 220 inputs the spliced ​​feature map and the target enhanced feature map into the self-attention module for processing by the self-attention mechanism to obtain a fused feature map, including: reducing the number of channels of the spliced ​​feature map through the second convolution layer to obtain a second convolution feature map, reducing the number of channels of the target enhanced feature map through the third convolution layer to obtain a third convolution feature map, reducing the number of channels of the target enhanced feature map through the fourth convolution layer to obtain a fourth convolution feature map; performing a shape transformation on the second convolution feature map to form a query matrix M q , transform the shape of the third convolutional feature map to form a key matrix M k , where the query matrix M q and bond matrix M k Able to perform matrix multiplication; query matrix M q and bond matrix M k Multiply, perform Softmax processing on the result of the multiplication, and obtain the relationship matrix M r ; Transform the shape of the fourth convolutional feature map to form a value matrix M v , where the value matrix M v and the relationship matrix M r Able to perform matrix multiplication; the value matrix M v and the relationship matrix M r Multiply, transform the result of the multiplication to restore it to the same resolution as the spliced ​​feature map, and then restore it to the same number of channels as the spliced ​​feature map through the fifth convolution layer to obtain the fifth convolution feature map; perform element-wise addition operation on the fifth convolution feature map and the spliced ​​feature map to obtain a fused feature map. For details, please refer to steps 3223c to 3223h above.

[0158] In some embodiments, the defect segmentation result includes a defect segmentation result map, and the pixel value of the pixel in the defect segmentation result map is used to indicate whether the corresponding pixel in the image of the object to be detected is a defective pixel; the classification head module includes a sixth convolutional layer. The defect segmentation module 220 uses the fused feature map through the classification head module to perform pixel-level classification prediction to obtain the defect segmentation result, including: inputting the fused feature map into the sixth convolutional layer for convolution processing, and then upsampling to the resolution of the image of the object to be detected to obtain a second multi-class score map, the number of channels of the second multi-class score map is the same as the number of preset classification categories, and the value of each channel at each pixel position in the second multi-class score map respectively represents the probability that the corresponding pixel in the image of the object to be detected belongs to each classification category; for each pixel in the second multi-class score map, the classification category corresponding to the channel with the largest value is taken as the final classification category to obtain the defect segmentation result map.

[0159] For the loss function of the defect segmentation model, please refer to the above introduction and will not go into details here.

[0160] The technical solution of the present invention has the following advantages compared with the prior art:

[0161] (1) Efficiency: From the data perspective, based on the active learning method, difficult samples in the sample images are selected as proxy samples for hyperparameter optimization, which greatly reduces the sample size, speeds up the hyperparameter optimization process, and improves model performance.

[0162] (2) In some embodiments, for defect detection tasks, from a data perspective, in view of the characteristics of industrial images with high resolution and small number and area of ​​defective areas, key learning areas are cut out and used as actual training sample images, which significantly reduces the computational workload of the model and further reduces the computational overhead;

[0163] (3) In some embodiments, from the perspective of the model, the structure of the model is specially optimized and designed, and the constructed model has fewer parameters, faster calculation speed, and excellent real-time performance; it can also work together with the optimization from the data perspective to further reduce the computing overhead, greatly reduce the time consumption of the hyperparameter optimization process, and obtain the optimal hyperparameter combination in a short time;

[0164] (4) Accuracy: During the hyperparameter search process, difficult samples are selected for training, and the generation of difficult samples is continuously iterated and optimized. The model can obtain the best hyperparameter optimization results in the most important and representative samples.

[0165] (5) Adaptability and scalability: It can adapt to different data sets and tasks, and can be used well with other hyperparameter search methods to improve optimization effects.

[0166] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0167] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. A defect detection method, characterized in that: include: Acquire an image of the object to be detected; Inputting the image of the inspected object into a pre-trained defect segmentation model to obtain a defect area in the image of the inspected object; The defect segmentation model is trained in the following way: Acquire a first sample image set, where the first sample image set includes a plurality of first sample images and corresponding defect segmentation annotations, where the defect segmentation annotations are used to indicate defect areas in the first sample images; For each hyperparameter combination among the preset multiple hyperparameter combinations, the hyperparameter combination is adopted, the defect segmentation model is trained using the first sample image set, and the defect segmentation performance of the defect segmentation model after the training is evaluated; the defect segmentation performance of the defect segmentation model after training using each hyperparameter combination is recorded; wherein, in the process of training the defect segmentation model using the first sample image set, if the defect segmentation error of the defect segmentation model for any first sample image is less than the first error threshold, the first sample image is discarded and no longer participates in subsequent rounds of training; The hyperparameter combination corresponding to the trained defect segmentation model with the best defect segmentation performance is selected as the final hyperparameter combination, and the defect segmentation model is trained using the final hyperparameter combination to obtain a defect segmentation model that has completed the training.

2. The defect detection method according to claim 1, characterized in that: The acquiring of the first sample image set comprises: Acquire an original sample image set, the original sample image set comprising a plurality of original sample images and corresponding original defect segmentation and annotation maps, the original defect segmentation and annotation maps having defect marking areas, and areas in the original sample images corresponding to the defect marking areas in the corresponding original defect segmentation and annotation maps being defect areas; Cut out a portion of the image including the defect marking area in the original defect segmentation and annotation map to obtain a first defect segmentation and annotation map; cut out an area in the original sample image corresponding to the first defect segmentation and annotation map cut out from the corresponding original defect segmentation and annotation map to obtain a first sample image corresponding to the first defect segmentation and annotation map; Unifying the resolutions of all first sample images and first defect segmentation annotation images; The first sample image set is composed of all the first sample images and the corresponding first defect segmentation annotation images, wherein the first defect segmentation annotation images are the defect segmentation annotations.

3. The defect detection method according to claim 1 or 2, characterized in that: The using the first sample image set to train the defect segmentation model and evaluating the defect segmentation performance of the trained defect segmentation model includes: The first sample image set is divided into a training set and a validation set according to a preset ratio, the training set is used to train the defect segmentation model, and after the training, the validation set is used to evaluate the defect segmentation performance of the trained defect segmentation model.

4. The defect detection method according to claim 1, characterized in that: The defect segmentation model includes an encoder, a decoder and a classification head module. The image of the detected object is input into the pre-trained defect segmentation model to obtain the defect area in the image of the detected object, including: Inputting the image of the detected object into the encoder for multi-scale feature extraction to obtain multiple layers of first feature maps with different resolutions; Inputting the first feature maps of the multiple layers with different resolutions into the decoder for fusion, and using the fused features for dimension upgrading and information recovery to obtain a fused feature map; The classification head module uses the fused feature map to perform pixel-level classification prediction to obtain a defect segmentation result, wherein the defect segmentation result includes a classification result of whether each pixel in the image of the detected object is a defective pixel; The defective area is determined based on the defective pixels.

5. The defect detection method according to claim 4, characterized in that: The decoder includes a first convolutional layer and a self-attention module, wherein the first feature maps of the multiple layers with different resolutions are input into the decoder for fusion, and the fused features are used for dimension upgrading and information recovery to obtain a fused feature map, including: Unifying the multiple layers of first feature maps with different resolutions to the resolution size of the first feature map with the largest resolution, and performing channel splicing to obtain a spliced ​​feature map; Inputting the concatenated feature map into the first convolution layer for convolution processing to obtain a first multi-category score map, wherein the number of channels of the first multi-category score map is the same as the number of preset classification categories, wherein at least one of the classification categories is used to represent defects; The self-attention module is utilized and the first multi-category score map is adopted to perform context information enhancement on the spliced ​​feature map to obtain the fused feature map.

6. The defect detection method according to claim 5, characterized in that: The using the self-attention module and the first multi-category score map to perform context information enhancement on the spliced ​​feature map to obtain the fused feature map includes: Performing target enhancement on the concatenated feature map using the first multi-category score map to obtain a target enhanced feature map; The concatenated feature map and the target enhanced feature map are input into the self-attention module for processing by the self-attention mechanism to obtain the fused feature map, wherein the concatenated feature map is used to form a query matrix in the self-attention mechanism, and the target enhanced feature map is used to form a key matrix and a value matrix in the self-attention mechanism.

7. The defect detection method according to claim 6, characterized in that: The step of performing target enhancement on the concatenated feature map by using the first multi-category score map to obtain a target enhanced feature map includes: Performing a shape transformation on the concatenated feature map to form a two-dimensional matrix F, and performing a shape transformation on the first multi-category score map to form a two-dimensional matrix P, wherein the two-dimensional matrix F and the two-dimensional matrix P can be matrix multiplied; The two-dimensional matrix F is multiplied by the two-dimensional matrix P, and the result of the multiplication is transformed in shape to be restored to image data, thereby obtaining the target enhanced feature map.

8. The defect detection method according to claim 6, characterized in that: The self-attention module includes a second convolution layer, a third convolution layer, a fourth convolution layer and a fifth convolution layer; the concatenated feature map and the target enhanced feature map are input into the self-attention module for processing by the self-attention mechanism to obtain the fused feature map, including: Reducing the number of channels of the spliced ​​feature map through the second convolution layer to obtain a second convolution feature map, reducing the number of channels of the target enhanced feature map through the third convolution layer to obtain a third convolution feature map, and reducing the number of channels of the target enhanced feature map through the fourth convolution layer to obtain a fourth convolution feature map; The second convolutional feature map is transformed to form a query matrix M q , the third convolutional feature map is transformed to form a key matrix M k , where the query matrix M q and bond matrix M k Ability to perform matrix multiplication; The query matrix M q and bond matrix M k Multiply, perform Softmax processing on the result of the multiplication, and obtain the relationship matrix M r ; The fourth convolution feature map is transformed to form a value matrix M v , where the value matrix M v and the relationship matrix M r Ability to perform matrix multiplication; The value matrix M v and the relationship matrix M r Multiply, transform the result of the multiplication into image data with the same resolution as the spliced ​​feature map, and then restore it to the same number of channels as the spliced ​​feature map through the fifth convolution layer to obtain a fifth convolution feature map; An element-wise addition operation is performed on the fifth convolutional feature map and the concatenated feature map to obtain the fused feature map.

9. The defect detection method according to claim 5, characterized in that: The defect segmentation result includes a defect segmentation result map, and the pixel values ​​of the pixels in the defect segmentation result map are used to indicate whether the corresponding pixels in the image of the object to be detected are defective pixels; the classification head module includes a sixth convolution layer; The classification head module uses the fused feature map to perform pixel-level classification prediction to obtain a defect segmentation result, including: Inputting the fused feature map into the sixth convolutional layer for convolution processing, and then upsampling to the resolution of the image of the object to be detected, to obtain a second multi-class score map, wherein the number of channels of the second multi-class score map is the same as the number of preset classification categories, and the value of each channel at each pixel position in the second multi-class score map respectively represents the probability that the corresponding pixel in the image of the object to be detected belongs to each classification category, wherein at least one of the classification categories is used to represent a defect; For each pixel of the second multi-category score map, the classification category corresponding to the channel with the largest value is taken as the final classification category to obtain the defect segmentation result map.

10. The defect detection method according to claim 9, characterized in that: The loss function of the defect segmentation model consists of a first loss function and a second loss function, wherein the first loss function represents the difference between the defect segmentation result image and the defect segmentation annotation, and the second loss function represents the difference between the first multi-category score image obtained by inputting the first sample image into the defect segmentation model and the defect segmentation annotation.

11. The defect detection method according to claim 10, characterized in that: The expression of the first loss function is: L1=1-2×IoU1, Wherein IoU1 represents the intersection-over-union ratio between the defect segmentation result image and the defect segmentation annotation; The expression of the second loss function is: L2=1-2×IoU2, Wherein IoU2 represents the intersection-over-union ratio of the intermediate segmentation result and the defect segmentation annotation, and the intermediate segmentation result is obtained by upsampling the first multi-category score map to the resolution size of the image of the object to be detected, and taking the classification category corresponding to the channel with the largest value for each pixel after upsampling; The loss function of the defect segmentation model is expressed as: Loss=δ1L1+δ2L2, Wherein δ1 and δ2 are preset weight coefficients.

12. A hyperparameter optimization method, characterized in that: include: Acquire a sample image set, wherein the sample image set includes a plurality of sample images and corresponding annotations; For each of the preset multiple hyperparameter combinations, the hyperparameter combination is adopted, the target task model is trained using the sample image set, and the performance index of the trained target task model when performing the target task is evaluated; the performance index of the target task model when performing the target task after being trained using each hyperparameter combination is recorded; wherein, in the process of training the target task model using the sample image set, if the prediction error of the target task model for any sample image is less than the second error threshold, the sample image is discarded and no longer participates in subsequent rounds of training; The hyperparameter combination corresponding to the trained target task model with the best performance index is selected as the hyperparameter combination finally used to train the target task model.

13. A defect detection device, characterized in that: include: An image acquisition module, used to acquire an image of the detected object; A defect segmentation module, used for inputting the image of the detected object into a pre-trained defect segmentation model to obtain a defect area in the image of the detected object; A training module, used for training the defect segmentation model, includes: A sample acquisition submodule, used to acquire a first sample image set, wherein the first sample image set includes a plurality of first sample images and corresponding defect segmentation annotations, wherein the defect segmentation annotations are used to indicate defect areas in the first sample images; A hyperparameter search submodule is used to adopt each hyperparameter combination in a plurality of preset hyperparameter combinations, train a defect segmentation model using the first sample image set using the hyperparameter combination, and evaluate the defect segmentation performance of the defect segmentation model after the training; record the defect segmentation performance of the defect segmentation model after the training using each hyperparameter combination; wherein, in the process of training the defect segmentation model using the first sample image set, if the defect segmentation error of the defect segmentation model for any first sample image is less than a first error threshold, the first sample image is discarded and no longer participates in subsequent rounds of training; The optimization training submodule is used to select the hyperparameter combination corresponding to the trained defect segmentation model with the best defect segmentation performance as the final hyperparameter combination, and use the final hyperparameter combination to train the defect segmentation model to obtain a defect segmentation model that has completed the training.

14. A hyperparameter optimization device, characterized in that: include: A sample acquisition module, used to acquire a sample image set, wherein the sample image set includes a plurality of sample images and corresponding annotations; A hyperparameter search module is used to adopt each hyperparameter combination in a plurality of preset hyperparameter combinations, train the target task model using the sample image set, and evaluate the performance index of the trained target task model when performing the target task; record the performance index of the target task model when performing the target task after being trained using each hyperparameter combination; wherein, in the process of training the target task model using the sample image set, if the prediction error of the target task model for any sample image is less than a second error threshold, the sample image is discarded and no longer participates in subsequent rounds of training; The hyperparameter determination module is used to select the hyperparameter combination corresponding to the trained target task model with the best performance index as the hyperparameter combination finally used for training the target task model.

15. A computer program product comprising a computer program and / or instructions, characterized in that: When the computer program and / or instructions are executed by a processor, the defect detection method according to any one of claims 1 to 11 or the hyperparameter optimization method according to claim 12 is implemented.

Images