Defect detection method and system and computer program product

Abstract

The invention discloses a defect detection method and system and a computer program product, and the method comprises the steps: inputting a to-be-detected image of a detected object into a defect detection model, and obtaining a defect detection result of the to-be-detected image; wherein the defect detection model is constructed based on a convolutional neural network, and the training process comprises the following steps: obtaining a training sample image and a corresponding defect label; performing feature extraction on the training sample image through a defect detection model to obtain a first feature map, and performing defect prediction by using the first feature map to obtain a first defect prediction result of the training sample image; performing global feature extraction on the training sample image to obtain a second feature map; carrying out weight adjustment on the defect detection model according to the loss function; wherein the loss function is determined at least according to the difference between the first feature map and the second feature map and the difference between the first defect prediction result and the defect label. According to the method, the detection performance and the real-time performance of the model are improved, the problem of data saturation is solved, and the training fitting speed is high.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a defect detection method and system, and a computer program product. Background Art

[0002] With the development of computer vision technology, it is being used more and more widely in the detection of defects or anomalies in objects or scenes. The network model based on deep learning learns complex features from large amounts of data in a data-driven manner, and can accurately identify defects or anomalies in images. It has excellent results and is widely used.

[0003] However, in vertical industries, a large amount of data is involved in the field, which is huge in quantity and diverse in types. This is the case in the field of industrial inspection. Industrial inspection realizes the analysis and detection of surface defects and quality of objects. With the development of society and the improvement of production technology, the product data in certain industries has exploded and the quantity is huge. Conventional deep learning detection models are unable to cope with it, and their accuracy and generalization are poor. The problems mainly include three aspects: (1) For the massive data in this field, conventional detection models have limited data storage performance and poor detection performance: general detection models based on convolutional neural networks (CNNs) have "data saturation" phenomenon and cannot cope with massive data; (2) Deeper and more complex network model structures (for example, network models based on Transformer architecture) can capture richer features and relationships and improve model detection capabilities, but the training time is long and the parameter adjustment is difficult, which also increases the technical threshold for use; (3) Deeper and more complex network models, due to their inherent high complexity, have poor real-time detection performance and are not conducive to production deployment. Summary of the Invention

[0004] The present invention mainly solves the technical problem that the existing defect detection technology based on deep learning cannot maintain training efficiency and real-time detection while coping with large amounts of data learning.

[0005] According to the first aspect, an embodiment provides a defect detection method, including:

[0006] Acquire an image of the object to be detected;

[0007] Inputting the image to be inspected into a pre-trained defect detection model to obtain a defect detection result for the image to be inspected, wherein the defect detection model is constructed based on a convolutional neural network and is used to extract features from the image to be inspected and predict defects using the extracted features;

[0008] The training process of the defect detection model includes:

[0009] Obtain training sample images and corresponding defect annotations;

[0010] Extracting features from the training sample image using the defect detection model to obtain a first feature map, and performing defect prediction using the first feature map to obtain a first defect prediction result for the training sample image;

[0011] Performing global feature extraction on the training sample image to obtain a second feature map;

[0012] The defect detection model is weighted according to a preset loss function; wherein the loss function is determined at least based on the difference between the first feature map and the second feature map, and the difference between the first defect prediction result and the defect annotation.

[0013] According to a second aspect, an embodiment provides a defect detection system, comprising:

[0014] The device for acquiring an image to be detected is used to acquire an image to be detected of the object to be detected;

[0015] A detection device, configured to input the image to be detected into a pre-trained defect detection model to obtain a defect detection result for the image to be detected, wherein the defect detection model is constructed based on a convolutional neural network and is configured to extract features from the image to be detected and predict defects using the extracted features;

[0016] A training device for training the defect detection model; the training device comprises:

[0017] Sample image acquisition module, used to obtain training sample images and corresponding defect annotations;

[0018] a prediction module, configured to extract features of the training sample image using the defect detection model to obtain a first feature map, and perform defect prediction using the first feature map to obtain a first defect prediction result of the training sample image;

[0019] A global feature extraction module, configured to perform global feature extraction on the training sample image to obtain a second feature map;

[0020] A weight adjustment module is used to adjust the weight of the defect detection model according to a preset loss function; wherein the loss function is determined based on at least the difference between the first feature map and the second feature map, and the difference between the first defect prediction result and the defect annotation.

[0021] According to a third aspect, an embodiment provides a computer program product, comprising a computer program and / or instructions, which, when executed by a processor, implement the defect detection method described in the first aspect.

[0022] According to the defect detection method and system, and computer program product of the above-mentioned embodiments, the defect detection model used is constructed based on a convolutional neural network architecture. During training, the defect detection model is used to extract features of the training sample image to obtain a first feature map, and the first feature map is used to predict defects to obtain a first defect prediction result of the training sample image; global features are extracted from the training sample image, and the obtained second feature map contains rich global semantic information; a preset loss function is used to adjust the weight of the defect detection model, and the loss function is determined at least based on the difference between the first feature map and the second feature map, and the difference between the first defect prediction result and the defect annotation. Since convolutional neural networks focus on the extraction of local features, and the second feature map contains rich global semantic information, by designing a special loss function and introducing the difference between the first feature map and the second feature map, the defect detection model is trained. The global semantic information can be transferred to the defect detection model, so that it can simultaneously integrate global semantic information and spatial detail information, and be compatible with accurate detection of targets of different sizes, thereby achieving performance improvement of defect detection models in vertical fields. For massive data, the training and fitting speed is fast, the number of effective samples increases, and the performance of the model is improved accordingly, solving the "data saturation" problem; a hybrid architecture of global feature extraction plus convolutional neural networks is adopted, which has fast training and fitting speed, does not require complex parameter adjustment, is easy to train and use, and reduces the technical threshold for use; only the defect detection model with a convolutional neural network structure is used for detection. Due to its simpler structure, fewer parameters, and faster inference speed, it is conducive to ensuring real-time detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of a hybrid architecture in one embodiment;

[0024] Figure 2 is a schematic diagram of a hybrid architecture in another embodiment;

[0025] Figure 3 is a schematic diagram of a hybrid architecture in yet another embodiment;

[0026] Figure 4 A flowchart of a method for training a defect detection model according to an embodiment;

[0027] Figure 5 Schematic diagram of the structure and processing flow of the guidance module S in some embodiments;

[0028] Figure 6Schematic diagram of the structure and processing flow of a decoder D in some embodiments;

[0029] Figure 7 A flowchart of using a first feature map to perform dimension upgrading and information recovery through a decoder to generate a first defect prediction result in some embodiments;

[0030] Figure 8 A flowchart of using the second feature map to perform dimension upgrading and information recovery through a decoder to generate a second defect prediction result in some embodiments;

[0031] Figure 9 A schematic diagram of the structure and processing flow of the feature fusion submodule Fusion in some embodiments;

[0032] Figure 10 The processing flow chart of each feature fusion submodule Fusion;

[0033] Figure 11 is a flow chart of a defect detection method according to an embodiment;

[0034] Figure 12 A schematic diagram of defect detection results obtained by performing defect detection on a detected object using the defect detection method of the present invention in an embodiment;

[0035] Figure 13 is a schematic structural diagram of a defect detection system in some embodiments;

[0036] Figure 14 Schematic diagram of the structure of the defect detection system in some other embodiments. DETAILED DESCRIPTION

[0037] The present invention will be further described in detail below by means of specific embodiments in conjunction with the accompanying drawings. Similar elements in different embodiments are numbered with associated similar elements. In the following embodiments, many detailed descriptions are provided to enable the present application to be better understood. However, those skilled in the art will readily appreciate that some of the features may be omitted in different circumstances, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the present application are not shown or described in the specification. This is to avoid the core portion of the present application being overwhelmed by excessive descriptions, and for those skilled in the art, it is not necessary to describe these related operations in detail. They will fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0038] In addition, the features, operations, or characteristics described in the specification may be combined in any appropriate manner to form various embodiments. Furthermore, the steps or actions in the method description may be reordered or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various sequences in the specification and drawings are provided solely for the purpose of clearly describing a particular embodiment and are not intended to be mandatory, unless otherwise specified.

[0039] The serial numbers assigned to components herein, such as "first," "second," and so on, are used solely to distinguish the objects being described and do not convey any sequential or technical meaning. "Multiple" means two or more. References to "connected" and "coupled" in this application, unless otherwise specified, include both direct and indirect connections (couplings).

[0040] As pointed out in the background technology, with the development of society and the improvement of production technology, the data in certain industries has exploded, with huge quantity and various types. The detection model based on convolutional neural network (CNN) has poor detection effect due to its inherent "data saturation" phenomenon, and is difficult to use in actual production.

[0041] Currently, in industrial inspection, vertical defect segmentation models are usually designed and developed based on deeper and more complex network architectures (e.g., the Swin Transformer segmentation model based on the Transformer architecture). These models have the following characteristics: (1) They perform well in modeling long-distance contexts, are effective in fitting massive data from the same industry, and have strong generalization performance. However, due to their very complex structure, the inference time increases, resulting in poor real-time performance. When processing high-resolution industrial images, the delay is greatly increased, making it difficult to actually use them. (2) The network parameters are complex to adjust and training is difficult, which places high demands on the users. (3) They are friendly to big data, but for some industries, data is scarce, and the model effect is poor. The combined effect of these three aspects leads to the decline in the detection performance of these models.

[0042] In response to these problems, the applicant believes through research that the existing technologies all simply adopt a network model of a certain architecture, resulting in inherent deficiencies, and proposes a novel defect detection method based on a hybrid architecture, which can improve model performance and speed, solve the "data saturation" problem, and adapt to defect detection in vertical fields. This architecture does not appear in the existing technology, and there is a lack of means for hybridization. The present invention proposes a hybrid architecture of global feature extraction plus CNN, which has the following advantages: during training, global semantic information is provided by global feature extraction, and CNN focuses on local feature extraction. Through a specially designed training alignment method, the global semantic information is passed to the CNN branch, so that the network can simultaneously fuse global semantic information and spatial detail information to achieve accurate detection of defects; the training and fitting speed is fast, which is convenient for application personnel to use; when deployed, only CNN is used for detection, and the real-time performance is good.

[0043] In some embodiments, a hybrid architecture combining a second feature extraction network with a self-attention mechanism and a CNN can complement the strengths of the two architectures, achieving significant improvements in accuracy and speed, and improving detection performance. During training, the second feature extraction network provides long-range contextual information, which is passed to the CNN, enabling the CNN to simultaneously integrate semantic information and spatial detail information for precise detection. This architecture fully combines the advantages of both networks, reducing model complexity, facilitating rapid training and fitting, and making it easier for application personnel to use. The global modeling capabilities of the second feature extraction network are utilized to align semantic representations, resulting in strong network feature extraction capabilities and excellent detection performance.

[0044] Please refer to Figure 1 In the technical solution of the present invention, deep learning technology is used to build a network structure, where x represents the training sample image and y is the corresponding defect annotation. During training, the entire network architecture F includes a global feature extraction branch module T and a CNN branch module C. The global feature extraction branch module T focuses on the extraction of global information, and the CNN branch module C focuses on the grasp of local detail information of the modeling hierarchy. During training, the global semantic information can be transferred to the CNN branch module C for fusion through a specially designed loss function Loss, so that it can learn how to extract rich global context information and detailed spatial detail information, and can simultaneously fuse semantic information and spatial detail information to enhance the overall feature representation. During reasoning, only a single branch of the CNN branch module C is deployed to directly output the defect detection results in an end-to-end manner to complete the task of industrial image defect detection. While maintaining high precision, it greatly reduces the expensive computing cost and realizes real-time reasoning.

[0045] In some embodiments, the global feature extraction branch module T is a second feature extraction network with a self-attention mechanism that can provide long-range contextual information. The second feature extraction network is friendly to big data, and the CNN branch module C has excellent detection effects on small data sets. The combination of the two architectures can train an efficient and accurate defect detection model, fully leveraging their respective advantages and jointly improving the model's ability to understand data. It can handle defect detection tasks in both data-rich and data-deficient industries, thereby solving the problem of industrial image defect detection in different vertical industries with both sufficient and insufficient data. It has few restrictions and is easy to use, facilitating large-scale promotion and deployment in different scenarios.

[0046] CNN branch module C can extract features from images and use the extracted features to predict defects. Figure 2 In some embodiments, the CNN branch module C includes a first feature extraction network for extracting features and a decoder D, and the decoder D is responsible for decoding the features and outputting prediction results. Specifically, the decoder D decodes the feature l extracted by the first feature extraction network and outputs the prediction result c. The prediction result c is calculated with the corresponding defect annotation y for loss, and the global feature b is calculated with the feature l for loss, and the two jointly guide the training of the network model. If the global feature extraction branch module T is a second feature extraction network with a self-attention mechanism, the decoder D decodes the feature b from the second feature extraction network and the feature l from the first feature extraction network, and outputs the prediction results t and c respectively. For the overall structure of the network F, the prediction results t and c are calculated with the corresponding defect annotation y for loss, and the feature b is calculated with the feature l for loss, completing the training of the model. For the second feature extraction network and the first feature extraction network, the decoder D is weight-shared or the same decoder D is used.

[0047] Please refer to Figure 2 Some embodiments also include a guiding module S, which is a semantic information alignment module for aligning the feature representations between the second feature extraction network and the CNN, two different network architectures, during the training process, so as to facilitate the transfer of global semantic information to the CNN.

[0048] Among them, the combined design of the second feature extraction network and CNN, the structure of the guidance module S and decoder D, and the design of the loss function in the training phase are the key to the entire network model.

[0049] The defect detection method of the present invention uses the defect detection model (i.e., CNN branch module C) obtained by training the above architecture to perform defect detection. The following is a detailed description of its training method. Please refer to Figure 4 In some embodiments, the training method includes steps 100 to 400.

[0050] Step 100: Obtain training sample images and corresponding defect annotations.

[0051] It can be understood that defect annotation is used to mark whether there are defects in the training sample images, and can also mark the location and / or category of the defects.

[0052] Step 200: extracting features from the training sample image using a defect detection model to obtain a first feature map, and performing defect prediction using the first feature map to obtain a first defect prediction result of the training sample image.

[0053] like Figure 2 As shown, in some embodiments, the defect detection model includes a first feature extraction network for extracting features and a decoder D, then step 200 includes: extracting features from the training sample image through the first feature extraction network to obtain a first feature map, and using the first feature map, performing dimensionality upgrading and information recovery through the decoder D to generate a first defect prediction result.

[0054] The first feature extraction network and decoder D can adopt the existing CNN-based image feature extraction network and decoder, for example, the first feature extraction network can adopt AlexNet, ResNet-18, etc.

[0055] Step 300: Perform global feature extraction on the training sample image to obtain a second feature map.

[0056] Global feature extraction can be performed using an existing network model capable of global feature extraction. In some embodiments, a second feature extraction network with a self-attention mechanism performs global feature extraction on the training sample image to obtain a second feature map. The second feature extraction network can be a network based on the Transformer architecture, such as ViT (Vision Transformer).

[0057] The second feature extraction network can better capture the global information in the image through its own self-attention mechanism, learn more complex feature representations, is friendly to big data, and does not suffer from the "data saturation" phenomenon. By integrating global information into module C, it can greatly enhance the accuracy and generalization ability of the model and solve the "data saturation" problem.

[0058] It should be noted that due to the self-attention mechanism adopted by the second feature extraction network, its computational complexity is quadratically related to the image resolution. If it is used directly for defect detection, the inference time will increase, the real-time performance will be poor, and the delay will be extremely large when processing high-resolution industrial images. In the present invention, by combining it with CNN, a hybrid architecture of the second feature extraction network plus CNN (for example, a hybrid architecture of Transformer plus CNN) is proposed, which achieves complementary advantages and makes up for the shortcomings of both.

[0059] Step 400: Adjust the weight of the defect detection model according to the preset loss function Loss.

[0060] like Figure 1 As shown, the loss function Loss is determined based on at least the difference between the first feature map and the second feature map, and the difference between the first defect prediction result and the defect annotation, so that the global information is passed to the defect detection model. The model can learn how to extract rich global context information and detailed spatial detail information, and perform defect prediction.

[0061] The combination of the two architectures in the present invention does not simply superimpose or connect the two architectures in series, or simply splice or fuse the first feature map and the second feature map. Instead, a special loss function is designed to introduce the difference between the first feature map and the second feature map, so that the model learns to extract both global context information and spatial detail information, and then uses the CNN model for detection.

[0062] In some embodiments, after performing global feature extraction on a training sample image to obtain a second feature map, defect prediction is performed using the second feature map to obtain a second defect prediction result for the training sample image; a loss function is further determined based on the difference between the second defect prediction result and the defect annotation; and weight adjustment of the defect detection model is performed based on a preset loss function, including: weight adjustment of both the defect detection model and the second feature extraction network based on the preset loss function. This enables the second feature extraction network to learn to extract features that are more effective for defect detection.

[0063] In some embodiments, defect prediction is performed using the second feature map to obtain a second defect prediction result of the training sample image, including: using the second feature map, performing dimensionality increase and information recovery through a decoder D to generate the second defect prediction result.

[0064] In some embodiments, the defect detection model extracts N first feature maps of different scales, and the global feature extraction also extracts N second feature maps of different scales, where N is an integer not less than 2, the N first feature maps are sorted by scale, the N second feature maps are sorted by scale, and the first feature maps and second feature maps with the same sequence number are paired. The difference between the first feature map and the second feature map described above includes the difference between each first feature map and the second feature map paired with it. The N first feature maps of different scales can be output by different feature extraction layers (referred to as the first feature extraction layer) in the first feature extraction network, such as the four feature extraction layers Layer1, Layer2, Layer3, and Layer4 composed of residual blocks in ResNet-18. The N second feature maps of different scales can be output by different feature extraction layers (referred to as the second feature extraction layer) in the second feature extraction network, such as the three layers of Block in ViT, namely Block1, Block2, and Block3. It can be understood that the higher the number of layers, that is, the deeper the feature extraction layer, the smaller the scale of the feature map output and the larger the number of channels.

[0065] In a specific implementation, the first feature extraction network includes at least N layers of first feature extraction layers, and the N first feature maps are respectively the first feature maps output by the N layers of first feature extraction layers. The second feature extraction network includes at least N layers of second feature extraction layers, and the N second feature maps are respectively the second feature maps output by the N layers of second feature extraction layers. The difference between the first feature map and the second feature map includes the difference between each first feature map and the second feature map paired with it, wherein the pairing of the first feature map and the second feature map means that the N layers of first feature extraction layers are sorted by the number of layers, and the N layers of second feature extraction layers are sorted by the number of layers (from high to low or from high to low), and the position of the first feature extraction layer that outputs the first feature map in the N layers of first feature extraction layers is the same as the position of the second feature extraction layer that outputs the second feature map in the N layers of second feature extraction layers. At this time, during training, the decoder D can process the first feature map output by the last first feature extraction layer in the N layers of first feature extraction layers, and process the second feature map output by the last second feature extraction layer in the N layers of second feature extraction layers, while during inference, it processes the feature map output by the last first feature extraction layer in the N layers of first feature extraction layers.

[0066] For example Figure 3In the example, taking the first feature extraction network as ResNet-18 and the second feature extraction network as ViT, the first feature extraction layers Layer2, Layer3 and Layer4 output the first feature maps l2, l3 and l4 respectively, and the second feature extraction layers Block1, Block2 and Block3 output the second feature maps b1, b2 and b3 respectively. Sorted from low to high by the number of layers, the first feature extraction layers Layer2, Layer3 and Layer4 are ranked 1st, 2nd and 3rd respectively, and the second feature extraction layers Block1, Block2 and Block3 are ranked 1st, 2nd and 3rd respectively. Therefore, l2 is paired with b1, l3 is paired with b2, and l4 is paired with b3.

[0067] Since the feature maps extracted by the shallow feature extraction layer have high resolution and small receptive field, the integration of global information in these / this layers cannot gain any benefit to the overall network. Based on the considerations of both effect and computing resources, in some embodiments of the present invention, the N first feature maps are respectively taken as the first feature maps output by the last N layers of the first feature extraction layer, for example Figure 3 In some embodiments, the N second feature maps are the second feature maps output by the last N second feature extraction layers, for example Figure 3 b1, b2 and b3 in .

[0068] In the above embodiment, the effect can be enhanced by transferring and guiding multiple layers of feature maps.

[0069] As described above, some embodiments also include a guidance module S, which specifically transfers the global semantic information extracted by the global feature extraction branch module T to the defect detection model for alignment and fusion. This enables the model to extract rich global contextual information and detailed spatial information, enhancing the model's expressiveness and generalization capabilities. The specific processing within the guidance module S is described below.

[0070] In some embodiments, the paired first feature map and the second feature map are further transformed in shape to unify the number of channels and resolution of the two, so as to measure the difference between the paired first feature map and the second feature map based on the first feature map and the second feature map after the unified number of channels and resolution. That is, the loss function Loss will be determined based on the paired first feature map and the second feature map after the unified number of channels and resolution.

[0071] Those skilled in the art can select specific shape transformation operations based on the actual shapes of the first and second feature maps to unify the number of channels and resolution. For example, using a ResNet-18 network as the first feature extraction network and a ViT network as the second feature extraction network, the second feature map can be subjected to positional encoding removal, dimensionality conversion, and reshape operations. A 1×1 convolution can then be performed on the first feature map to match the number of channels with the second feature map, and scaling can be performed to match the resolution with the second feature map. Unifying the number of channels and resolution facilitates comparison between the paired first and second feature maps.

[0072] Please refer to Figure 5 In some embodiments, the shape-transformed paired first and second feature maps are further input into the first and second convolutional layers, respectively, for convolution processing. The difference between the paired first and second feature maps is measured based on the convolution-processed first and second feature maps sl and sb. That is, the loss function Loss is determined based on the convolution-processed paired first and second feature maps sl and sb. When the weights of the defect detection model and the second feature extraction network are adjusted together according to the loss function Loss, the weights of the first and second convolutional layers are also adjusted together. The first and second convolutional layers can be convolutional layers with a 3×3 convolution kernel.

[0073] Considering that the feature representations extracted from the two different architectures differ significantly, the applicant believes that directly aligning the two different types of features may still lead to network learning difficulties, affecting the detection performance of the defect detection model. Therefore, in the above embodiment, after unifying the number of channels and resolution, the first feature map and the second feature map each need to be processed by a convolutional layer to achieve feature alignment under the control of the loss function. The role of the convolutional layer is to enable the network to adaptively learn linear processing of different architectural features during the learning process, thereby guiding the global information of the global feature extraction branch module T to the defect detection model and improving the performance of the model.

[0074] Please refer to Figure 6 In some embodiments, the decoder D includes a feature fusion module FS and a third 1×1 convolutional layer, on which the feature fusion module FS and the third 1×1 convolutional layer are constructed. Figure 7 As shown, using the first feature map, the decoder D performs dimension increase and information recovery to generate a first defect prediction result, including:

[0075] Step 210: The feature fusion module FS is used to fuse the inter-channel information and enhance the spatial information of the first feature map to obtain a first fused feature map. If the first feature map includes the first feature map output by the N first feature extraction layers of the first feature extraction network, the feature fusion module FS is used to fuse the inter-channel information and enhance the spatial information of the first feature map output by the last first feature extraction layer to obtain the first fused feature map.

[0076] Step 220: Input the first fused feature map into the third 1×1 convolutional layer for processing to convert the number of channels to the same number as the preset classification categories, and then upsample (Upsample) to the resolution of the training sample image to obtain a first multi-class score map. The value of each channel at each pixel position in the first multi-class score map represents the probability that the pixel at the corresponding position in the training sample image belongs to each classification category, where one classification category represents the background and at least one classification category is used to represent defects. If it is necessary to distinguish the categories of defects, multiple different defect classification categories can be set;

[0077] Step 230: For each pixel in the first multi-class score map, the classification category corresponding to the channel with the largest value is taken as the final classification category to obtain a first defect prediction map. The first defect prediction result includes the first defect prediction map. In this case, the background can be represented by a low grayscale value (such as 0), and the defect can be represented by a high grayscale value (such as 255). Different grayscale values ​​can also be used to represent different categories of defects.

[0078] Similarly, if Figure 8 As shown, using the second feature map, the decoder D performs dimension increase and information recovery to generate a second defect prediction result, including:

[0079] Step 310: The feature fusion module FS is used to fuse the inter-channel information and enhance the spatial information of the second feature map to obtain a second fused feature map. If the second feature map includes the second feature maps output by the N second feature extraction layers of the second feature extraction network, the feature fusion module FS is used to fuse the inter-channel information and enhance the spatial information of the second feature map output by the last second feature extraction layer to obtain the second fused feature map.

[0080] Step 320: Input the second fused feature map into the third 1×1 convolutional layer for processing to convert the number of channels to the same as the number of classification categories, and then upsample to the resolution of the training sample image to obtain a second multi-class score map. The value of each channel at each pixel position in the second multi-class score map represents the probability that the pixel at the corresponding position in the training sample image belongs to each classification category.

[0081] Step 330: For each pixel of the second multi-class score map, the classification category corresponding to the channel with the largest value is taken as the final classification category to obtain a second defect prediction map. The second defect prediction result includes the second defect prediction map.

[0082] The feature fusion module (FS) can include multiple 1×1 convolutional layers and channel pooling to fuse inter-channel information in feature maps, as well as multiple convolutional layers with kernel sizes of at least 3×3 to enhance spatial information in feature maps. By fusing inter-channel information and enhancing spatial information in feature maps, global and local features can be effectively integrated, improving the detection capabilities of the defect detection model.

[0083] Please refer to Figure 6 In some embodiments, the feature fusion module FS includes multiple feature fusion submodules connected in series ( Figure 6 3 are taken as an example). The feature fusion module FS is used to fuse the inter-channel information and enhance the spatial information of the first feature map to obtain a first fused feature map, including: inputting the first feature map into the feature fusion module FS, and sequentially processing the first feature map through multiple feature fusion sub-modules Fusion to obtain the first fused feature map; and the feature fusion module FS is used to fuse the inter-channel information and enhance the spatial information of the second feature map to obtain a second fused feature map, including: inputting the second feature map into the feature fusion module FS, and sequentially processing the second feature map through multiple feature fusion sub-modules Fusion to obtain the second fused feature map.

[0084] Please refer to Figure 9 In some embodiments, the feature fusion submodule Fusion includes a channel fusion branch CF and a spatial information enhancement branch SF. On this basis, Figure 10 As shown, the processing of each feature fusion submodule Fusion includes:

[0085] Step 1: Input the input feature map input into the channel fusion branch CF and the spatial information enhancement branch SF for processing, and obtain the channel fusion feature map and the spatial enhancement feature map; wherein, when processing the first feature map, the input feature map input of the first feature fusion submodule Fusion is the first feature map, and when processing the second feature map, the input feature map input of the first feature fusion submodule Fusion is the second feature map, and the input feature map input of the remaining feature fusion submodules Fusion is the output feature map of the previous feature fusion submodule Fusion;

[0086] Step 2: Concatenate the channel fusion feature map and the spatial enhancement feature map to obtain a concatenated feature map.

[0087] Step 3: Reduce the number of channels of the concatenated feature map and increase the resolution to obtain the output feature map output.

[0088] The first fused feature map and the second fused feature map are output feature maps of the last feature fusion submodule Fusion when processing the first feature map and the second feature map, respectively.

[0089] In this embodiment, separate branches are designed to fuse the inter-channel information and enhance the spatial information of the feature map, and then the outputs of the two branches are spliced ​​together, making the network structure simple and efficient.

[0090] like Figure 9 As shown, in some embodiments, the feature fusion submodule further includes a seventh 1×1 convolutional layer. Step 30 reduces the number of channels of the concatenated feature map and increases the resolution to obtain an output feature map, including: inputting the concatenated feature map into the seventh 1×1 convolutional layer for processing to reduce the number of channels, and then upsampling to increase the resolution. In one example, the number of channels is reduced by half, and the resolution is increased to twice the original value.

[0091] In some embodiments, the channel fusion branch CF includes a fourth 1×1 convolutional layer, and the spatial information enhancement branch SF includes a fifth 1×1 convolutional layer and a sixth convolutional layer connected in series. The fourth 1×1 convolutional layer and the fifth 1×1 convolutional layer can reduce the number of channels of the input feature map. In one instance, the fourth 1×1 convolutional layer and the fifth 1×1 convolutional layer reduce the number of channels by half, and the sixth convolutional layer is a 3×3 convolutional layer.

[0092] Among them, the role of the fourth 1×1 convolutional layer in the channel fusion branch CF is to interact and integrate information across channels, promoting information fusion between channels; in the spatial information enhancement branch SF, the fifth 1×1 convolutional layer is first used to reduce the number of channels of the input feature map to reduce the number of parameters and complete the information refinement, and then the sixth convolutional layer is used to further enhance the spatial information of the features, enhancing the model's ability to extract key spatial features; the fifth 1×1 convolutional layer and the sixth convolutional layer are a simple but effective network layer combination that effectively reduces the number of parameters and computational complexity while improving the training efficiency and expression ability of the model. It can be seen that the feature fusion submodule in this embodiment is efficient and lightweight, taking into account both accuracy and speed.

[0093] Regarding the loss function Loss, in some embodiments, it is composed of three sub-loss functions. The first sub-loss function L s Determined according to the difference between the first feature map and the second feature map, that is, after obtaining the first feature map and the second feature map, according to the first sub-loss function L s Calculate the loss. If the first and second feature maps of multiple layers are extracted, the first sub-loss function L sDetermined according to the difference between each first feature map and its paired second feature map; and the second sub-loss function L t The third sub-loss function L is determined based on the difference between the second defect prediction result and the defect annotation. c Determined based on the difference between the first defect prediction result and the defect annotation.

[0094] In one embodiment, the expression of the loss function Loss is:

[0095] Loss = αL s +βL t +μL c ,

[0096] Among them, α, β and μ are preset weight coefficients respectively.

[0097] In one embodiment, the first sub-loss function L s The expression is:

[0098]

[0099] in

[0100]

[0101] m is the number of elements in the i-th first feature map after unifying the number of channels and resolution of the paired first and second feature maps (also the number of elements in the paired second feature map), sb i,j is the jth element of the i-th first feature map, sl i,j is the jth element of the second feature map paired with the i-th first feature map.

[0102] In one embodiment, the second sub-loss function L t The expression is:

[0103] L t =1-2×IoU2,

[0104] Where IoU2 represents the intersection-over-union ratio between the second defect prediction image and the defect annotation, which can be Where t represents the second defect prediction graph, ∩ represents finding the intersection, and ∪ represents finding the union.

[0105] In one embodiment, the third sub-loss function L c The expression is:

[0106] L c =1-2×IoU1,

[0107] Where IoU1 represents the intersection-over-union ratio between the first defect prediction image and the defect annotation, which can be Where c represents the first defect prediction graph.

[0108] Defect annotation can use a defect annotation map with the same resolution as the training sample image. Each pixel point in the defect annotation map indicates whether the corresponding pixel point in the training sample image is a defect. Specifically, different grayscale values ​​can be used to mark the background and defects.

[0109] Please refer to Figure 11 Based on the above training method, the defect detection method of the present invention includes the following steps:

[0110] Step 001: Obtain an image of the object to be detected;

[0111] Step 002: Input the image to be inspected into a pre-trained defect detection model to obtain a defect detection result for the image to be inspected. The defect detection model is built based on a convolutional neural network and is used to extract features from the image to be inspected and use the extracted features to predict defects. The defect detection model is trained using the training method described above. It is understood that the defect detection results obtained by using the defect detection model will include the same information as the information annotated in the defect annotation.

[0112] In one embodiment, the defect detection method of the present invention is used to detect defects on the object to be detected to obtain the defect detection results such as Figure 12 As shown, the left picture is the image to be detected of the object to be tested, and the right picture is the defect detection result. It can be seen that the defect detection method of the present invention can accurately detect subtle scratch defects on the object to be tested.

[0113] Based on the above defect detection method, the present invention also provides a defect detection system, please refer to Figure 13 In some embodiments, the device includes a device 10 for acquiring an image to be detected, a detection device 20 and a training device 30.

[0114] The device 10 for acquiring an image to be detected is used to acquire an image to be detected of the object to be detected.

[0115] The detection device 20 is used to input the image to be detected into a pre-trained defect detection model to obtain a defect detection result for the image to be detected, wherein the defect detection model is constructed based on a convolutional neural network, which is used to extract features of the image to be detected and use the extracted features to predict defects.

[0116] The training device 30 is used to train the defect detection model. Figure 13As shown, the training device 30 includes a sample image acquisition module 31, a prediction module 32, a global feature extraction module 33, and a weight adjustment module 34. The sample image acquisition module 31 is used to obtain training sample images and corresponding defect annotations; the prediction module 32 is used to extract features from the training sample images using a defect detection model to obtain a first feature map, and to use the first feature map to perform defect prediction to obtain a first defect prediction result for the training sample images; the global feature extraction module 33 is used to perform global feature extraction on the training sample images to obtain a second feature map; and the weight adjustment module 34 is used to adjust the weights of the defect detection model according to a preset loss function. The loss function is determined based on at least the difference between the first feature map and the second feature map, and the difference between the first defect prediction result and the defect annotation.

[0117] In some embodiments, the defect detection model includes a first feature extraction network and a decoder for extracting features. The prediction module 32 extracts features from the training sample image through the defect detection model to obtain a first feature map, and uses the first feature map to perform defect prediction. Obtaining the first defect prediction result of the training sample image includes: extracting features from the training sample image through the first feature extraction network to obtain a first feature map, and using the first feature map to perform dimensionality upgrading and information recovery through the decoder to generate a first defect prediction result.

[0118] In some embodiments, the global feature extraction module 33 performs global feature extraction on the training sample image to obtain the second feature map, including: performing global feature extraction on the training sample image through a second feature extraction network to obtain the second feature map; wherein the second feature extraction network has a self-attention mechanism.

[0119] In some embodiments, the prediction module 32 is further configured to use the second feature map to perform defect prediction, obtaining a second defect prediction result for the training sample image. The loss function is further determined based on the difference between the second defect prediction result and the defect annotation. The weight adjustment module 34 adjusts the weights of the defect detection model based on a preset loss function, including adjusting the weights of both the defect detection model and the second feature extraction network based on the preset loss function.

[0120] In some embodiments, the prediction module 32 uses the second feature map to perform defect prediction to obtain a second defect prediction result of the training sample image, including: using the second feature map, performing dimensionality increase and information recovery through a decoder to generate the second defect prediction result.

[0121] Regarding the extraction method of the first feature map and the second feature map, please refer to the above introduction and will not be repeated here.

[0122] Please refer to Figure 14In some embodiments, the training device 30 further includes a feature alignment module 35 for performing shape transformation on the paired first feature map and the second feature map to unify the number of channels and the resolution, so as to measure the difference between the paired first feature map and the second feature map based on the first feature map and the second feature map after unifying the number of channels and the resolution.

[0123] In some embodiments, the feature alignment module 35 is further configured to input the shape-transformed paired first and second feature maps into the first and second convolutional layers, respectively, for convolution processing, thereby measuring the difference between the paired first and second feature maps based on the convolutional first and second feature maps. When the weight adjustment module 34 adjusts the weights of the defect detection model and the second feature extraction network together according to a preset loss function, it also adjusts the weights of the first and second convolutional layers.

[0124] In some embodiments, the decoder includes a feature fusion module and a third 1×1 convolutional layer.

[0125] The prediction module 32 uses the first feature map and performs dimensionality upgrading and information recovery through the decoder to generate a first defect prediction result, including: fusing the inter-channel information and enhancing the spatial information of the first feature map through the feature fusion module to obtain a first fused feature map, wherein, if the first feature map includes the first feature map output by the N-layer first feature extraction layer of the first feature extraction network, then the first feature map output by the last layer of the first feature extraction layer is subjected to inter-channel information fusion and spatial information enhancement to obtain a first fused feature map through the feature fusion module; the first fused feature map is input into the third 1×1 convolutional layer for processing to transform the number of channels to the same as the number of preset classification categories, and then upsampled to the resolution of the training sample image to obtain a first multi-category score map; for each pixel of the first multi-category score map, the classification category corresponding to the channel with the largest value is taken as the final classification category to obtain a first defect prediction map, and the first defect prediction result includes the first defect prediction map.

[0126] The prediction module 32 uses the second feature map and performs dimensionality upgrading and information recovery through the decoder to generate a second defect prediction result, including: fusing the second feature map between channels and enhancing the spatial information through the feature fusion module to obtain a second fused feature map, wherein, if the second feature map includes the second feature map output by the N-layer second feature extraction layer of the second feature extraction network, then the second feature map output by the last layer of the second feature extraction layer is subjected to inter-channel information fusion and spatial information enhancement to obtain a second fused feature map through the feature fusion module; the second fused feature map is input into the third 1×1 convolutional layer for processing to transform the number of channels to the same as the number of classification categories, and then upsampled to the resolution of the training sample image to obtain a second multi-category score map; for each pixel of the second multi-category score map, the classification category corresponding to the channel with the largest value is taken as the final classification category to obtain a second defect prediction map, and the second defect prediction result includes the second defect prediction map.

[0127] In some embodiments, the feature fusion module includes a plurality of feature fusion submodules connected in series, and the feature fusion submodule includes a channel fusion branch and a spatial information enhancement branch.

[0128] The prediction module 32 performs inter-channel information fusion and spatial information enhancement on the first feature map through the feature fusion module to obtain a first fused feature map, including: inputting the first feature map into the feature fusion module, and sequentially processing the first fused feature map through multiple feature fusion sub-modules. The prediction module 32 performs inter-channel information fusion and spatial information enhancement on the second feature map through the feature fusion module to obtain a second fused feature map, including: inputting the second feature map into the feature fusion module, and sequentially processing the second feature map through multiple feature fusion sub-modules to obtain a second fused feature map.

[0129] For the processing process of each feature fusion sub-module, the specific structure of the channel fusion branch and the spatial information enhancement branch, and the specific expression of the loss function, please refer to the relevant introduction above and will not be repeated here.

[0130] The present invention further provides a computer program product, comprising a computer program and / or instructions, which implement the defect detection method of the present invention when executed by a processor.

[0131] The defect detection method proposed in this paper, based on a hybrid architecture combining global feature extraction and CNN, is adaptable to vertical defect detection tasks and can perform image training and inference in an end-to-end manner. Compared to existing technologies, it has the following advantages.

[0132] (1) Hybrid architecture to improve detection performance. Global semantic information is obtained through global feature extraction, while the CNN-based defect detection model focuses on extracting local spatial detail information. By aligning features, the feature extraction capability of the defect detection model is enhanced, and it is compatible with accurate detection of targets of different sizes, thereby improving detection accuracy and the generalization performance of the model. In some embodiments, the hybrid architecture design of a second feature extraction network with a self-attention mechanism plus a CNN is used to achieve complementary advantages, fully combine the advantages of the two architectures, and jointly improve the model's ability to understand data. It can cope with defect detection tasks in industry fields with sufficient data, as well as defect detection tasks in industry fields with insufficient data. It has a wide range of application scenarios and is easy to promote.

[0133] (2) Fast training and fitting speed, easy to train and use. The present invention adopts a combined design of global feature extraction and CNN. Compared with the simple use of more complex and deeper network structures, the training and fitting speed is fast, no complicated parameter adjustment is required, and it is easy to train and use, which reduces the technical threshold of users and saves costs.

[0134] (3) Solve the "data saturation" problem and have high sample capacity. For massive data, training and fitting are fast, the number of effective samples increases, and the performance of the defect detection model is improved accordingly, solving the "data saturation" problem.

[0135] (4) Good real-time performance. A hybrid architecture is used for training, and only CNN is used for inference after completion. The model has a simple structure, few parameters, fast inference speed, and excellent real-time performance. It achieves significant improvements between accuracy and speed, while greatly reducing hardware costs, improving manufacturers' production efficiency, and achieving cost reduction and efficiency improvement.

[0136] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer program. When all or part of the functions in the above embodiments are implemented by computer program, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented. In addition, when all or part of the functions in the above embodiments are implemented by computer program, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and saved in the memory of the local device by downloading or copying, or the system of the local device is updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be implemented.

[0137] The above examples are used to illustrate the present invention, which are only used to help understand the present invention and are not intended to limit the present invention. Those skilled in the art can make several simple deductions, modifications or substitutions based on the concept of the present invention.

Claims

1. A defect detection method, characterized in that: include: Acquire an image of the object to be detected; Inputting the image to be inspected into a pre-trained defect detection model to obtain a defect detection result for the image to be inspected, wherein the defect detection model is constructed based on a convolutional neural network and is used to extract features from the image to be inspected and predict defects using the extracted features; The training process of the defect detection model includes: Obtain training sample images and corresponding defect annotations; Extracting features from the training sample image using the defect detection model to obtain a first feature map, and performing defect prediction using the first feature map to obtain a first defect prediction result for the training sample image; Performing global feature extraction on the training sample image to obtain a second feature map; The defect detection model is weighted according to a preset loss function; wherein the loss function is determined at least based on the difference between the first feature map and the second feature map, and the difference between the first defect prediction result and the defect annotation.

2. The defect detection method according to claim 1, wherein: The global feature extraction of the training sample image to obtain the second feature map includes: performing global feature extraction on the training sample image through a second feature extraction network to obtain the second feature map; wherein the second feature extraction network has a self-attention mechanism.

3. The defect detection method according to claim 2, wherein: After performing global feature extraction on the training sample image to obtain a second feature map, defect prediction is performed using the second feature map to obtain a second defect prediction result for the training sample image; The loss function is further determined based on the difference between the second defect prediction result and the defect annotation; The weight adjustment of the defect detection model according to a preset loss function includes: weight adjustment of the defect detection model and the second feature extraction network together according to a preset loss function.

4. The defect detection method according to claim 3, wherein: The defect detection model includes a first feature extraction network and a decoder; The step of extracting features from the training sample image using the defect detection model to obtain a first feature map, and performing defect prediction using the first feature map to obtain a first defect prediction result for the training sample image includes: extracting features from the training sample image using the first feature extraction network to obtain the first feature map, and performing dimension upgrading and information recovery using the decoder using the first feature map to generate the first defect prediction result; Using the second feature map to perform defect prediction to obtain a second defect prediction result of the training sample image includes: using the second feature map, performing dimension upgrading and information recovery through the decoder to generate the second defect prediction result.

5. The defect detection method according to claim 2, wherein: The number of the first feature maps and the number of the second feature maps are both N, the first feature extraction network includes at least N layers of first feature extraction layers, the N first feature maps are respectively first feature maps output by the N layers of first feature extraction layers, the second feature extraction network includes at least N layers of second feature extraction layers, the N second feature maps are respectively second feature maps output by the N layers of second feature extraction layers, where N is an integer not less than 2; The difference between the first feature map and the second feature map includes the difference between each first feature map and the second feature map paired with it, wherein the pairing of the first feature map and the second feature map means that the N layers of first feature extraction layers are sorted by layer number, and the N layers of second feature extraction layers are sorted by layer number, and the position of the first feature extraction layer that outputs the first feature map in the N layers of first feature extraction layers is the same as the position of the second feature extraction layer that outputs the second feature map in the N layers of second feature extraction layers.

6. The defect detection method according to claim 5, wherein: The N first feature maps are respectively the first feature maps output by the last N first feature extraction layers, and the N second feature maps are respectively the second feature maps output by the last N second feature extraction layers.

7. The defect detection method according to claim 5, wherein: Also includes: A shape transformation is performed on the paired first feature map and the second feature map to unify the number of channels and the resolution, so as to measure a difference between the paired first feature map and the second feature map based on the first feature map and the second feature map after the number of channels and the resolution are unified.

8. The defect detection method according to claim 7, wherein: Also includes: Inputting the paired first feature map and the second feature map after the shape transformation into a first convolution layer and a second convolution layer respectively for convolution processing, so as to measure the difference between the paired first feature map and the second feature map based on the convolution processing of the first feature map and the second feature map; When the weights of the defect detection model and the second feature extraction network are adjusted together according to a preset loss function, the weights of the first convolutional layer and the second convolutional layer are also adjusted together.

9. The defect detection method according to claim 4 or 5, characterized in that: The decoder includes a feature fusion module and a third 1×1 convolutional layer; The utilizing the first feature map to perform dimension upgrading and information recovery by the decoder to generate the first defect prediction result includes: fusing inter-channel information and enhancing spatial information on the first feature map by the feature fusion module to obtain a first fused feature map, wherein, if the first feature map includes first feature maps output by N first feature extraction layers of the first feature extraction network, fusing inter-channel information and enhancing spatial information on the first feature map output by the last first feature extraction layer by the feature fusion module to obtain a first fused feature map; Inputting the first fused feature map into the third 1×1 convolutional layer for processing to convert the number of channels to the same number as the preset classification categories, and then upsampling to the resolution of the training sample image to obtain a first multi-class score map, wherein the value of each channel at each pixel position in the first multi-class score map respectively represents the probability that the pixel at the corresponding position in the training sample image belongs to each classification category, wherein at least one of the classification categories is used to represent a defect; For each pixel of the first multi-class score map, taking the classification category corresponding to the channel with the largest value as the final classification category, and obtaining a first defect prediction map, wherein the first defect prediction result includes the first defect prediction map; The step of utilizing the second feature map and performing dimension upgrading and information recovery by the decoder to generate the second defect prediction result includes: fusing inter-channel information and enhancing spatial information on the second feature map by the feature fusion module to obtain a second fused feature map, wherein, if the second feature map includes second feature maps output by N layers of second feature extraction layers of the second feature extraction network, then fusing inter-channel information and enhancing spatial information on the second feature map output by the last layer of the second feature extraction layer by the feature fusion module to obtain a second fused feature map; Inputting the second fused feature map into the third 1×1 convolutional layer for processing to convert the number of channels to the same as the number of classification categories, and then upsampling to the resolution of the training sample image to obtain a second multi-class score map, where the value of each channel at each pixel position in the second multi-class score map represents the probability that the pixel at the corresponding position in the training sample image belongs to each classification category; For each pixel of the second multi-class score map, the classification category corresponding to the channel with the largest value is taken as the final classification category to obtain a second defect prediction map. The second defect prediction result includes the second defect prediction map.

10. The defect detection method according to claim 9, wherein: The feature fusion module includes a plurality of feature fusion submodules connected in series, and the feature fusion submodule includes a channel fusion branch and a spatial information enhancement branch; The step of fusing inter-channel information and enhancing spatial information of the first feature map by the feature fusion module to obtain a first fused feature map includes: inputting the first feature map into the feature fusion module, and sequentially processing the first feature map by the plurality of feature fusion submodules to obtain a first fused feature map; The step of fusing inter-channel information and enhancing spatial information of the second feature map by the feature fusion module to obtain a second fused feature map includes: inputting the second feature map into the feature fusion module, and sequentially processing the second feature map by the plurality of feature fusion submodules to obtain a second fused feature map; The processing of each feature fusion submodule includes: The input feature map is respectively input into the channel fusion branch and the spatial information enhancement branch for processing to obtain a channel fusion feature map and a spatial enhancement feature map; wherein, when the first feature map is processed, the input feature map of the first feature fusion submodule is the first feature map, and when the second feature map is processed, the input feature map of the first feature fusion submodule is the second feature map, and the input feature maps of the remaining feature fusion submodules are the output feature maps of the previous feature fusion submodule; Splicing the channel fusion feature map and the spatial enhancement feature map to obtain a spliced ​​feature map; The number of channels of the spliced ​​feature map is reduced and the resolution is increased to obtain an output feature map; the first fused feature map and the second fused feature map are the output feature maps of the last feature fusion submodule when processing the first feature map and the second feature map, respectively.

11. The defect detection method according to claim 10, wherein: The channel fusion branch includes a fourth 1×1 convolutional layer, and the spatial information enhancement branch includes a fifth 1×1 convolutional layer and a sixth convolutional layer connected in series. The fourth 1×1 convolutional layer and the fifth 1×1 convolutional layer can reduce the number of channels of the input feature map.

12. The defect detection method according to claim 10, wherein: The feature fusion submodule also includes a seventh 1×1 convolutional layer; The reducing the number of channels of the spliced ​​feature map and increasing the resolution to obtain the output feature map includes: inputting the spliced ​​feature map into the seventh 1×1 convolutional layer for processing to reduce the number of channels, and then upsampling to increase the resolution.

13. The defect detection method according to claim 9, wherein: The expression of the loss function is: Loss=αL s +βL t +μL c , Among them L s is the first sub-loss function, determined according to the difference between each of the first feature maps and the second feature map paired with it, L t is the second sub-loss function, determined according to the difference between the second defect prediction result and the defect annotation, L c is the third sub-loss function, which is determined according to the difference between the first defect prediction result and the defect annotation, and α, β and μ are preset weight coefficients respectively.

14. The defect detection method according to claim 13, wherein: The expression of the first sub-loss function is: in m is the number of elements in the i-th first feature map after unifying the number of channels and resolution of the paired first feature map and the second feature map, sb i,j is the jth element of the first feature map of the i-th, sl i,j is the jth element of the second feature map paired with the i-th first feature map; The expression of the second sub-loss function is: L t =1-2×IoU2, Wherein IoU2 represents the intersection-over-union ratio of the second defect prediction image and the defect annotation; The expression of the third sub-loss function is: L c =1-2×IoU1, Wherein IoU1 represents the intersection-over-union ratio of the first defect prediction image and the defect annotation.

15. A defect detection system, characterized in that: include: The device for acquiring an image to be detected is used to acquire an image to be detected of the object to be detected; A detection device, configured to input the image to be detected into a pre-trained defect detection model to obtain a defect detection result for the image to be detected, wherein the defect detection model is constructed based on a convolutional neural network and is configured to extract features from the image to be detected and predict defects using the extracted features; A training device for training the defect detection model; the training device comprises: Sample image acquisition module, used to obtain training sample images and corresponding defect annotations; a prediction module, configured to extract features of the training sample image using the defect detection model to obtain a first feature map, and perform defect prediction using the first feature map to obtain a first defect prediction result of the training sample image; A global feature extraction module, configured to perform global feature extraction on the training sample image to obtain a second feature map; A weight adjustment module is used to adjust the weight of the defect detection model according to a preset loss function; wherein the loss function is determined based on at least the difference between the first feature map and the second feature map, and the difference between the first defect prediction result and the defect annotation.

16. A computer program product comprising a computer program and / or instructions, characterized in that When the computer program and / or instructions are executed by a processor, the defect detection method according to any one of claims 1 to 14 is implemented.

Images