Image defect segmentation method and system based on Swin Transformer
Through the Swin Transformer-based image defect segmentation method, combined with multi-scale feature extraction and multi-task learning, the problem of insufficient feature extraction and robustness in the prior art is solved, and more efficient image defect segmentation performance is achieved.
Patent Information
- Application Number
- CN202411354540.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-09-27
AI Technical Summary
The existing image defect segmentation technology has limitations in feature extraction capabilities, multi-task processing capabilities and robustness, making it difficult to effectively capture multi-scale features and process images in complex backgrounds.
The image defect segmentation method based on Swin Transformer is adopted, and multi-scale feature extraction and multi-task learning are combined with conditional random field and dynamic weight adjustment mechanism to optimize the segmentation results and improve the segmentation accuracy and robustness.
It significantly improves the overall performance of image defect segmentation, enhances the model's ability to capture multi-scale features and handles complex backgrounds, and improves the accuracy and robustness of segmentation results.
Smart Images

Figure CN118887242B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the technical field of image defect detection, and in particular to an image defect segmentation method and system based on Swin Transformer. Background Art
[0002] In many fields such as industrial production and medical health, image defect segmentation technology is widely used in scenarios such as product surface quality inspection and pathological tissue section analysis. These application scenarios require the segmentation algorithm to not only accurately identify and segment the defective parts in the image, but also be able to further classify the defect type and accurately mark the location of the defect.
[0003] Most of the current mainstream image defect segmentation technologies are based on traditional convolutional neural networks (CNNs). These models have been able to perform basic segmentation tasks well with the support of a large amount of training data. However, as the complexity and diversity of application scenarios increase, traditional CNN models begin to expose some shortcomings.
[0004] The following are the shortcomings of the existing solutions:
[0005] Limited feature extraction capabilities: Traditional CNN models are limited by their fixed convolution kernel size and receptive field, making it difficult to effectively capture multi-scale feature information in images, especially for those small defects, the model has low sensitivity.
[0006] Lack of multi-task learning capabilities: In a single-task learning framework, the model can often only focus on the task of defect segmentation, while ignoring other important subtasks such as defect type classification and defect location localization, resulting in limited generalization and practicality of the model.
[0007] Insufficient robustness of the model: Due to the lack of an effective contextual information integration mechanism, existing models are easily disturbed when processing images with complex backgrounds, thus affecting the accuracy of the segmentation results.
[0008] In summary, the existing technical solutions have certain limitations in feature extraction capability, multi-task processing capability and robustness, which are the main problems that this application solution attempts to solve. By introducing an image defect segmentation method based on Swin Transformer, we aim to overcome the above-mentioned defects and improve the overall performance of image defect segmentation. Summary of the invention
[0009] The embodiment of the present application provides an image defect segmentation method and system based on Swin Transformer to solve the problem that the prior art has certain limitations in feature extraction capability, multi-task processing capability and robustness.
[0010] In a first aspect, an embodiment of the present application provides an image defect segmentation method based on Swin Transformer, comprising:
[0011] Obtain the image to be detected and input it into the deep learning model based on the Swin Transformer architecture;
[0012] Using the deep learning model to perform multi-scale feature extraction on the image to be detected, and generate a feature map containing image defect information;
[0013] Perform image defect segmentation according to the feature map to obtain a segmentation result;
[0014] In the process of image defect segmentation, multi-task learning is performed, and the multi-task learning includes at least one of the following sub-tasks or any combination of several of them: defect segmentation, defect type classification, and defect position location;
[0015] The image defect segmentation is performed according to the feature map to obtain the segmentation result, including: applying a conditional random field to the feature map to obtain a feature map optimized by the conditional random field, wherein the conditional random field is used to refine the segmentation boundary of the feature map and improve the segmentation accuracy; introducing a dynamic weight adjustment mechanism into the deep learning model to dynamically adjust the weight of the loss function corresponding to the deep learning model according to the importance of different subtasks; and inputting the feature map optimized by the conditional random field into a deep learning module introduced with the dynamic weight adjustment mechanism to output the segmentation result through the deep learning module;
[0016] The applying of the conditional random field to the feature map includes: using the energy function optimized by the conditional random field to refine the segmentation boundary of the feature map to obtain the feature map optimized by the conditional random field; wherein the energy function is composed of a data term, a smooth term, a context consistency term, a task coordination term, and a priori term, and the expression of the energy function includes:
[0017]
[0018] in, represents the energy function, represents the segmentation result, are the weight coefficients of the smooth term, contextual consistency term, task coordination term, and prior term, respectively. is a data item, is a smooth term, is the contextual consistency term, It is a task coordination item. is a priori term.
[0019] Optionally, before acquiring the image to be detected and inputting it into the deep learning model based on the Swin Transformer architecture, the method further includes:
[0020] Build a deep learning model based on the Swin Transformer architecture;
[0021] The construction of a deep learning model based on the Swin Transformer architecture includes:
[0022] Initialize the deep learning framework;
[0023] Define the Swin Transformer module that includes the self-attention mechanism and the feedforward network;
[0024] Applying a multi-scale attention mechanism in the Swin Transformer module to capture feature information at different resolutions;
[0025] Constructing a network structure consisting of multiple Swin Transformer modules in the deep learning framework, and adding an input layer and an output layer to the network structure;
[0026] The network structure is trained by preprocessing data, defining a loss function and an optimizer to complete the training of the deep learning model.
[0027] Optionally, the using the deep learning model to perform multi-scale feature extraction on the image to be detected and generate a feature map containing image defect information includes:
[0028] Deploy Swin Transformer modules at different resolution levels of the deep learning model, and the Swin Transformer modules at each resolution level are used to capture feature information of a specific scale;
[0029] A feature fusion strategy is adopted at each resolution level, wherein the feature fusion strategy includes at least a skip connection, a pyramid pooling module or a dense connection to promote the complementarity and fusion of the feature information between different levels;
[0030] Integrate feature information at different scales to generate image feature representation;
[0031] Based on the image feature representation, feature information of different scales is mapped into a feature space to generate a feature map containing image defect information.
[0032] Optionally, the data item Defined as:
[0033]
[0034] in, Based on parameters The deep learning model is used to analyze the pixels Probability distribution of predictions , pixels The real label The conditional probability of , which is used to measure the deep learning model's The degree of match between the predicted probability distribution and the conditional probability of the true label, It's pixels The real label, Based on parameters The deep learning model is used to analyze the pixels The predicted probability distribution, is the regularization coefficient, which is used to control the regularization term The strength of is the jth regularization term, used to restrict the distribution of predicted labels; is the weight coefficient of the supervisory signal term, which is used to control the supervisory signal term The strength of is the kth supervisory signal item, which is used to constrain the prediction results from different angles; is the weight coefficient of the attention item, controlling the attention item The strength of is the mth attention item, which is used to enhance the model’s attention to the key areas; Represents the pixel index in the image; represents the index of the regularization term; Represents the index of the supervision signal item; Represents the index of the attention item.
[0035] Optionally, the smooth item Defined as:
[0036]
[0037] in, Representing pixels With pixels The connection relationship between Represents pixels With pixels The smoothing weight between is a binary potential function used to penalize pixels With pixels Inconsistent label assignments between Used to measure pixels With pixels Label consistency between ; Represents pixels With non-neighborhood pixels The long-range connection between is the long-range connection weight, is a long-range potential function used to process pixels With non-neighborhood pixels The relationship between Used to measure pixels With non-neighborhood pixels Consistency of labels between is the weight coefficient of the regularization term, which is used to control the regularization term The degree of influence, It is A regularization term is used to increase the smoothness of the model; is the weight coefficient of the high-order potential function, which is used to control the high-order potential function The degree of influence is the pth high-order potential function, which is used to further enhance the nonlinear fitting ability of the model; is a neighborhood set containing pixels Directly adjacent pixels pairing; is a non-neighborhood set, containing pixels Pixels that are not directly adjacent but have a certain spatial relationship pairing; is the index of the regularized term; is the index of the higher-order potential function.
[0038] Optionally, the context consistency item Defined as:
[0039]
[0040] in, Represents the context area, is the weight coefficient of the context consistency function, is a context consistency function, , Represents pixels With pixels The smoothing weight between Used to measure pixels in the context area With pixels Label consistency between ; stands for Global Guidance Area, is the weight coefficient of the global guidance term, is a global guidance term used to guide the consistency of local segmentation, , Represents pixels The neighborhood set of Used to measure the pixels in the global guidance area With pixels Label consistency between ; is the visual attention mechanism area, is the weight coefficient of the visual attention mechanism term, is the visual attention mechanism term, used to enhance attention to visually salient areas, , Represents the weight coefficient of the visually salient area, Used to measure pixels Attention scores on visually salient regions.
[0041] Optionally, the task coordination item Defined as:
[0042]
[0043] in, Represents the subtask type, is the weight coefficient of the subtask, is a function that measures the completion of the subtask and is used to determine the difference between the predicted label and the true label. , Represents pixels The predicted label of Represents a subtask Medium Pixels The true label of represents the balance factor between tasks, is the weight coefficient of the balance factor, is the balance factor function, which is used to adjust the relative importance of different tasks. , represents the average loss of all subtasks;
[0044] The prior Defined as:
[0045]
[0046] in, represents the type of prior knowledge, is the weight coefficient of prior knowledge, It is an item of prior knowledge.
[0047] In a second aspect, an embodiment of the present application provides an image defect segmentation system based on Swin Transformer, comprising:
[0048] The input module is used to obtain the image to be detected and input it into the deep learning model based on the Swin Transformer architecture;
[0049] An extraction and generation module, used to perform multi-scale feature extraction on the image to be detected using the deep learning model, and generate a feature map containing image defect information;
[0050] A segmentation module, used to perform image defect segmentation according to the feature map to obtain a segmentation result;
[0051] In the process of image defect segmentation, multi-task learning is performed, and the multi-task learning includes at least one of the following sub-tasks or any combination of several of them: defect segmentation, defect type classification, and defect position location;
[0052] The segmentation module is specifically used to apply a conditional random field to the feature map to obtain a feature map optimized by the conditional random field, wherein the conditional random field is used to refine the segmentation boundary of the feature map and improve the segmentation accuracy; a dynamic weight adjustment mechanism is introduced into the deep learning model to dynamically adjust the weight of the loss function corresponding to the deep learning model according to the importance of different subtasks; the feature map optimized by the conditional random field is input into the deep learning module introduced with the dynamic weight adjustment mechanism, so as to output the segmentation result through the deep learning module;
[0053] The applying of the conditional random field to the feature map includes: using the energy function optimized by the conditional random field to refine the segmentation boundary of the feature map to obtain the feature map optimized by the conditional random field; wherein the energy function is composed of a data term, a smooth term, a context consistency term, a task coordination term, and a priori term, and the expression of the energy function includes:
[0054]
[0055] in, represents the energy function, represents the segmentation result, are the weight coefficients of the smooth term, contextual consistency term, task coordination term, and prior term, respectively. is a data item, is a smooth term, is the contextual consistency term, It is a task coordination item. is a priori term.
[0056] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any of the Swin Transformer-based image defect segmentation methods described in the first aspect.
[0057] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the SwinTransformer-based image defect segmentation method described in any one of the first aspects.
[0058] In an embodiment of the present application, an image to be detected is obtained and input into a deep learning model based on the Swin Transformer architecture; the deep learning model is used to extract multi-scale features of the image to be detected, and a feature map containing image defect information is generated; image defect segmentation is performed according to the feature map to obtain a segmentation result; wherein, in the process of image defect segmentation, multi-task learning is performed, and the multi-task learning includes at least one of the following subtasks or any combination of several of them: defect segmentation, defect type classification, and defect location positioning. The technical solution provided by the present application improves the overall performance of image defect segmentation.
[0059] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0061] Figure 1 A flowchart of an image defect segmentation method based on Swin Transformer provided in an embodiment of the present application;
[0062] Figure 2 A schematic diagram of the structure of an image defect segmentation system based on Swin Transformer provided in an embodiment of the present application;
[0063] Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0064] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0065] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.
[0066] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0067] Figure 1 A flowchart of an image defect segmentation method based on Swin Transformer is provided for an embodiment of the present application, such as Figure 1 As shown, the method includes:
[0068] 101. Obtain the image to be detected and input it into the deep learning model based on the Swin Transformer architecture;
[0069] In this step, obtaining the image to be inspected refers to obtaining the image data required for defect detection from the image acquisition device or other sources. The deep learning model based on the Swin Transformer architecture is a traditional convolutional neural network (CNN) improved by Transformer technology, which is particularly suitable for processing data with long-distance dependencies, such as the relationship between pixels in an image. Swin Transformer implements the local attention mechanism through the window mechanism, and allows information interaction between different windows through the shifted window multi-head self-attention (SW-MSA) module, so that the model has both local perception capabilities and can capture global context information.
[0070] In the embodiment of the present application, we first prepare an image that needs to be defect-detected, for example, an image of a ceramic tile surface taken by an industrial camera on a ceramic production line. Assume that the resolution of the image is 1024x1024 pixels and is stored in RGB format. Next, we pass this image as input to a pre-trained deep learning model based on the SwinTransformer architecture.
[0071] In actual operation, this step usually involves image preprocessing, such as normalization, resizing, etc. For example, if the input size accepted by the model is fixed at 224x224 pixels, we need to first use libraries such as OpenCV or PIL to resize the original image to this size. In addition, the image pixel values need to be normalized, that is, the pixel values of each channel are converted to between 0 and 1.
[0072] Specifically, assuming we have a tile image called tile_image.jpg, we can load and resize it using Python's PIL library, and then normalize it using numpy:
[0073] from PIL import Image
[0074] import numpy as np
[0075] # Load the image
[0076] img = Image.open('tile_image.jpg')
[0077] # Resize the image
[0078] img_resized = img.resize((224, 224))
[0079] # Convert to numpy array
[0080] img_array = np.array(img_resized)
[0081] # Normalization
[0082] img_normalized = img_array / 255.0
[0083] After the above preprocessing, we can input the processed image img_normalized into our Swin Transformer model for the next step of feature extraction and defect segmentation. It should be noted here that "Swin Transformer" refers to "Shifting Window Transformer", which is a special Transformer architecture used for image processing tasks. Its core idea is to perform self-attention mechanism in local areas by sliding windows, while keeping the flow of global information by alternating between windows.
[0084] The above is a complete example of obtaining an image to be inspected and inputting it into a deep learning model based on the Swin Transformer architecture. Subsequent steps will be to perform multi-scale feature extraction, defect information generation, and final defect segmentation based on this input image.
[0085] 102. Perform multi-scale feature extraction on the image to be detected using the deep learning model, and generate a feature map containing image defect information;
[0086] Optionally, the step 102 of using the deep learning model to perform multi-scale feature extraction on the image to be detected and generate a feature map containing image defect information includes: deploying Swin Transformer modules at different resolution levels of the deep learning model, and the Swin Transformer modules at each resolution level are used to capture feature information of a specific scale; adopting a feature fusion strategy at each resolution level, and the feature fusion strategy includes at least a skip connection, a pyramid pooling module or a dense connection to promote the complementarity and fusion of the feature information between different levels; integrating feature information of different scales to generate an image feature representation; based on the image feature representation, mapping feature information of different scales into a feature space to generate a feature map containing image defect information.
[0087] In this step, the deep learning model is used to extract multi-scale features from the image to be detected, and a feature map containing image defect information is generated. Specifically, this step includes the following aspects:
[0088] The Swin Transformer module is deployed at different resolution levels of the deep learning model: the Swin Transformer module implements the local attention mechanism through the window mechanism, and allows information interaction between different windows through the Shifted Window Multi-head Self-Attention (SW-MSA) module, thereby capturing feature information of a specific scale.
[0089] A feature fusion strategy is adopted at each resolution level: the feature fusion strategy includes but is not limited to Skip Connection, Pyramid Pooling Module (PPM) or Dense Connection to promote the complementarity and fusion of feature information between different levels.
[0090] Integrate feature information at different scales: Integrate feature information at different resolution levels to form a complete image feature representation.
[0091] Based on the image feature representation, feature information of different scales is mapped into a feature space to generate a feature map containing image defect information.
[0092] In the embodiment of this application, we use a deep learning model based on the Swin Transformer architecture to extract multi-scale features from images and generate a feature map containing image defect information. The specific implementation steps are as follows:
[0093] Deploy the Swin Transformer module:
[0094] Deploy Swin Transformer modules at different resolution levels. Suppose we have four different resolution levels, namely , the Swin Transformer module at each resolution level is used to capture feature information of a specific scale.
[0095] Feature fusion strategy:
[0096] A feature fusion strategy is used at each resolution level, such as skip connection, pyramid pooling module or dense connection. Here we take skip connection as an example:
[0097] Skip Connection: directly connect high-level features to low-level features to retain more detailed information.
[0098] Suppose we have three feature maps , which come from different resolution levels. We use skip connections to and Connect them to form a new feature map .
[0099] F_concat=tf.concat([F_1,F_3],axis=3)
[0100] Integrate feature information at different scales:
[0101] The feature information at different resolution levels is combined to form a complete image feature representation. Suppose we have four feature maps , we combine them to form a feature representation .
[0102] F_combined=tf.concat([F_1,F_2,F_3,F_4],axis=3)
[0103] Mapped into a feature space:
[0104] Based on the image feature representation, feature information of different scales is mapped into a feature space to generate a feature map containing image defect information. Suppose we have a convolutional layer , which represents the feature Mapped into a feature space.
[0105] def generate_feature_map(F_combined):
[0106] # Use convolutional layers to map feature representations into a feature space
[0107] feature_map = tf.keras.layers.Conv2D(filters=64, kernel_size=3,padding='same', activation='relu')(F_combined)
[0108] return feature_map
[0109] feature_map = generate_feature_map(F_combined)
[0110] In the embodiment of the present application, we deploy Swin Transformer modules at different resolution levels and adopt feature fusion strategies such as jump connections to integrate feature information of different scales to form a complete image feature representation. Finally, the feature representation is mapped to a feature space through a convolutional layer to generate a feature map containing image defect information. This method can make full use of feature information of different scales and improve the accuracy and robustness of image defect segmentation.
[0111] Through the above steps, we can effectively capture the feature information in the image from different scales, enhance the complementarity of features through feature fusion strategies, and finally generate a feature map containing image defect information.
[0112] 103. Perform image defect segmentation according to the feature map to obtain a segmentation result;
[0113] In the process of image defect segmentation, multi-task learning is performed, and the multi-task learning includes at least one of the following sub-tasks or any combination of several of them: defect segmentation, defect type classification, and defect position positioning.
[0114] Optionally, the step 103 of performing image defect segmentation according to the feature map to obtain a segmentation result includes: applying a conditional random field to the feature map to obtain a feature map optimized by the conditional random field, wherein the conditional random field is used to refine the segmentation boundary of the feature map and improve the segmentation accuracy; introducing a dynamic weight adjustment mechanism into the deep learning model to dynamically adjust the weight of the loss function corresponding to the deep learning model according to the importance of different subtasks; and inputting the feature map optimized by the conditional random field into a deep learning module that introduces a dynamic weight adjustment mechanism to output the segmentation result through the deep learning module.
[0115] Optionally, the applying of the conditional random field to the feature map in step 103 includes: using an energy function optimized by the conditional random field to refine the segmentation boundary of the feature map to obtain a feature map optimized by the conditional random field;
[0116] The energy function is composed of a data term, a smooth term, a context consistency term, a task coordination term, and a priori term, and the expression of the energy function includes:
[0117]
[0118] in, represents the energy function, represents the segmentation result, are the weight coefficients of the smooth term, contextual consistency term, task coordination term, and prior term, respectively. is a data item, is a smooth term, is the contextual consistency term, It is a task coordination item. is a priori term.
[0119] Among them, the data item Defined as:
[0120]
[0121] in, Based on parameters The deep learning model is used to analyze the pixels Probability distribution of predictions , pixels The real label The conditional probability of , which is used to measure the deep learning model's The degree of match between the predicted probability distribution and the conditional probability of the true label, It's pixels The real label, Based on parameters The deep learning model is used to analyze the pixels The predicted probability distribution, is the regularization coefficient, which is used to control the regularization term The strength of is the jth regularization term, used to restrict the distribution of predicted labels; is the weight coefficient of the supervisory signal term, which is used to control the supervisory signal term The strength of is the kth supervisory signal item, which is used to constrain the prediction results from different angles; is the weight coefficient of the attention item, controlling the attention item The strength of is the mth attention item, which is used to enhance the model’s attention to the key areas; Represents the pixel index in the image; represents the index of the regularization term; Represents the index of the supervision signal item; Represents the index of the attention item.
[0122] The smooth term Defined as:
[0123]
[0124] in, Representing pixels With pixels The connection relationship between Represents pixels With pixels The smoothing weight between is a binary potential function used to penalize pixels With pixels Inconsistent label assignments between Used to measure pixels With pixels Label consistency between ; Represents pixels With non-neighborhood pixels The long-range connection between is the long-range connection weight, is a long-range potential function used to process pixels With non-neighborhood pixels The relationship between Used to measure pixels With non-neighborhood pixels Consistency of labels between is the weight coefficient of the regularization term, which is used to control the regularization term The degree of influence It is A regularization term is used to increase the smoothness of the model; is the weight coefficient of the high-order potential function, which is used to control the high-order potential function The degree of influence is the pth high-order potential function, which is used to further enhance the nonlinear fitting ability of the model; is a neighborhood set containing pixels Directly adjacent pixels pairing; is a non-neighborhood set, containing pixels Pixels that are not directly adjacent but have a certain spatial relationship pairing; is the index of the regularized term; is the index of the higher-order potential function.
[0125] Among them, the context consistency term Defined as:
[0126]
[0127] in, Represents the context area, is the weight coefficient of the context consistency function, is a context consistency function, , Represents pixels With pixels The smoothing weight between Used to measure pixels in the context area With pixels Label consistency between ; stands for Global Guidance Area, is the weight coefficient of the global guidance term, is a global guidance term used to guide the consistency of local segmentation, , Represents pixels The neighborhood set of Used to measure the pixels in the global guidance area With pixels Label consistency between ; is the visual attention mechanism area, is the weight coefficient of the visual attention mechanism term, is the visual attention mechanism term, used to enhance attention to visually salient areas, , Represents the weight coefficient of the visually salient area, Used to measure pixels Attention scores on visually salient regions.
[0128] The task coordination item Defined as:
[0129]
[0130] in, Represents the subtask type, is the weight coefficient of the subtask, is a function that measures the completion of the subtask and is used to determine the difference between the predicted label and the true label. , Represents pixels The predicted label of Represents a subtask Medium Pixels The true label of represents the balance factor between tasks, is the weight coefficient of the balance factor, is the balance factor function, which is used to adjust the relative importance of different tasks. , represents the average loss of all subtasks;
[0131] The prior Defined as:
[0132]
[0133] in, represents the type of prior knowledge, is the weight coefficient of prior knowledge, It is an item of prior knowledge.
[0134] In this step, we will perform image defect segmentation based on the feature map and perform multi-task learning during the segmentation process. The specific steps include:
[0135] Apply Conditional Random Field (CRF) to feature maps: Optimize feature maps through conditional random fields to refine segmentation boundaries and improve segmentation accuracy.
[0136] Introduce a dynamic weight adjustment mechanism: dynamically adjust the weight of the loss function corresponding to the deep learning model according to the importance of different subtasks.
[0137] Output the segmentation result through the deep learning module: The feature map optimized by the conditional random field is input into the deep learning module that introduces the dynamic weight adjustment mechanism to output the final segmentation result.
[0138] In the embodiments of the present application, we will introduce in detail how to perform image defect segmentation based on the feature map, optimize the feature map through the conditional random field, and finally output the segmentation result.
[0139] First, we apply a conditional random field (CRF) to the feature map to refine the segmentation boundaries and improve segmentation accuracy. It includes several components:
[0140] Data Item : Measures the difference between the predicted label and the true label.
[0141] Smooth Item : Ensure label consistency between adjacent pixels.
[0142] Contextual consistency item : Ensure label consistency within the context area.
[0143] Task coordination items : Ensure collaborative work between multiple tasks.
[0144] Prior : Introduce domain expert knowledge or statistical characteristics of historical data.
[0145] The expression of the energy function is:
[0146]
[0147] Data Item Defined as:
[0148]
[0149] in, Based on parameters The deep learning model is used to analyze the pixels Probability distribution of predictions , pixels The real label The conditional probability of , which is used to measure the deep learning model's The degree of match between the predicted probability distribution and the conditional probability of the true label, It's pixels The real label, Based on parameters The deep learning model is used to analyze the pixels The predicted probability distribution, is the regularization coefficient, which is used to control the regularization term The strength of is the jth regularization term, used to restrict the distribution of predicted labels; is the weight coefficient of the supervisory signal term, which is used to control the supervisory signal term The strength of is the kth supervisory signal item, which is used to constrain the prediction results from different angles; is the weight coefficient of the attention item, controlling the attention item The strength of is the mth attention item, which is used to enhance the model’s attention to the key areas; Represents the pixel index in the image; represents the index of the regularization term; Represents the index of the supervision signal item; Represents the index of the attention item.
[0150] Smooth Item Defined as:
[0151]
[0152] in, Representing pixels With pixels The connection relationship between Represents pixels With pixels The smoothing weight between is a binary potential function used to penalize pixels With pixels Inconsistent label assignments between Used to measure pixels With pixels Label consistency between ; Represents pixels With non-neighborhood pixels The long-range connection between is the long-range connection weight, is a long-range potential function used to process pixels With non-neighborhood pixels The relationship between Used to measure pixels With non-neighborhood pixels Consistency of labels between is the weight coefficient of the regularization term, which is used to control the regularization term The degree of influence It is A regularization term is used to increase the smoothness of the model; is the weight coefficient of the high-order potential function, which is used to control the high-order potential function The degree of influence is the pth high-order potential function, which is used to further enhance the nonlinear fitting ability of the model; is a neighborhood set containing pixels Directly adjacent pixels pairing; is a non-neighborhood set, containing pixels Pixels that are not directly adjacent but have a certain spatial relationship pairing; is the index of the regularized term; is the index of the higher-order potential function.
[0153] Contextual consistency item Defined as:
[0154]
[0155] in, Represents the context area, is the weight coefficient of the context consistency function, is a context consistency function, , Represents pixels With pixels The smoothing weight between Used to measure pixels in the context area With pixels Label consistency between ; stands for Global Guidance Area, is the weight coefficient of the global guidance term, is a global guidance term used to guide the consistency of local segmentation, , Represents pixels The neighborhood set of Used to measure the pixels in the global guidance area With pixels Label consistency between ; is the visual attention mechanism area, is the weight coefficient of the visual attention mechanism term, is the visual attention mechanism term, used to enhance attention to visually salient areas, , Represents the weight coefficient of the visually salient area, Used to measure pixels Attention scores on visually salient regions.
[0156] The task coordination item Defined as:
[0157]
[0158] in, Represents the subtask type, is the weight coefficient of the subtask, is a function that measures the completion of the subtask and is used to determine the difference between the predicted label and the true label. , Represents pixels The predicted label of Represents a subtask Medium Pixels The true label of represents the balance factor between tasks, is the weight coefficient of the balance factor, is the balance factor function, which is used to adjust the relative importance of different tasks. , represents the average loss of all subtasks;
[0159] The prior Defined as:
[0160]
[0161] in, represents the type of prior knowledge, is the weight coefficient of prior knowledge, It is an item of prior knowledge.
[0162] Assume that we have obtained a feature map , now we need to optimize it. We set the weight coefficient of the energy function to .
[0163] Apply a conditional random field:
[0164] Hypothesis feature map The size is , we will apply conditional random fields to optimize the feature map. The specific steps are as follows:
[0165] Initialize segmentation results : Assume that the initial segmentation results are randomly generated labels.
[0166] Optimize segmentation results: Continuously update segmentation results through iterative optimization algorithms (such as Loopy Belief Propagation) , until convergence.
[0167] Introducing a dynamic weight adjustment mechanism:
[0168] In the deep learning model, we introduce a dynamic weight adjustment mechanism to dynamically adjust the weight of the loss function according to the importance of different subtasks. Assuming that the current tasks are defect segmentation, defect type classification, and defect location, we set the weight to .
[0169] Output segmentation results:
[0170] The feature map optimized by the conditional random field Input into the deep learning module that introduces a dynamic weight adjustment mechanism, and output the final segmentation result .
[0171] Through the above steps, we can effectively optimize the feature map and generate high-quality segmentation results under the multi-task learning framework. This method can ensure the accuracy and robustness of the segmentation results while taking into account the needs of multiple subtasks.
[0172] Optionally, before step 101 of "obtaining an image to be detected and inputting it into a deep learning model based on the Swin Transformer architecture", the method further includes: constructing a deep learning model based on the Swin Transformer architecture;
[0173] Among them, the construction of a deep learning model based on the Swin Transformer architecture includes: initializing the deep learning framework; defining a Swin Transformer module including a self-attention mechanism and a feedforward network; applying a multi-scale attention mechanism in the Swin Transformer module to capture feature information at different resolutions; constructing a network structure composed of multiple Swin Transformer modules in the deep learning framework, and adding an input layer and an output layer to the network structure; training the network structure by preprocessing data, defining a loss function and an optimizer to complete the training of the deep learning model.
[0174] In this step, we will build a deep learning model based on the Swin Transformer architecture for image defect detection. The specific steps include:
[0175] Initialize the deep learning framework;
[0176] Define the Swin Transformer module that includes the self-attention mechanism and the feedforward network;
[0177] Apply multi-scale attention mechanism in Swin Transformer module to capture feature information at different resolutions;
[0178] Build a network structure consisting of multiple Swin Transformer modules and add input and output layers to the network structure;
[0179] The deep learning model is trained by preprocessing data, defining loss functions and optimizers to train the network structure.
[0180] In the embodiments of this application, we will introduce in detail how to build a deep learning model based on the Swin Transformer architecture and enable it to effectively perform image defect detection through training.
[0181] First, we need to initialize the deep learning framework. Here we choose to use PyTorch as our development framework because it provides a flexible API to build and train complex neural network models. The initialization process usually includes importing necessary libraries, setting up devices (CPU or GPU), and configuring basic hyperparameters.
[0182] The following is a specific code example:
[0183] import torch
[0184] import torch.nn as nn
[0185] from torch.utils.data import DataLoader
[0186] from torchvision import datasets, transforms
[0187] device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
[0188] Next, define a Swin Transformer module. Swin Transformer is a Transformer-based architecture that captures local and global dependencies through the local self-attention mechanism within the window and the shifted window mutual attention mechanism between windows. In addition, each module also contains a feed-forward network (FFN).
[0189] class SwinTransformerBlock(nn.Module):
[0190] def init__(self, dim, num_heads, window_size=7, shift_size=0, mlp_ratio=4., qkv_bias=False, qk_scale=None, drop=0., attn_drop=0., drop_path=0., act_layer=nn.GELU, norm_layer=nn.LayerNorm):
[0191] super().__init__()
[0192] self.dim = dim
[0193] self.num_heads = num_heads
[0194] self.window_size = window_size
[0195] self.shift_size = shift_size
[0196] self.mlp_ratio = mlp_ratio
[0197] assert 0<= self.shift_size<self.window_size, "shift_size must in 0-window_size"
[0198] # Define a patch embedding network
[0199] self.norm1 = norm_layer(dim)
[0200] self.attn = WindowAttention(
[0201] dim,window_size=(self.window_size,self.window_size), num_heads=num_heads,
[0202] qkv_bias=qkv_bias, qk_scale=qk_scale, attn_drop=attn_drop, proj_drop=drop)
[0203] self.drop_path = DropPath(drop_path) if drop_path>0. else nn.Identity()
[0204] self.norm2 = norm_layer(dim)
[0205] mlp_hidden_dim = int(dim * mlp_ratio)
[0206] self.mlp = Mlp(in_features=dim, hidden_features=mlp_hidden_dim, act_layer=act_layer, drop=drop)
[0207] def forward(self, x, attn_mask):
[0208] H, W = x.shape[1], x.shape[2]
[0209] x = self.norm1(x)
[0210] # calculate attention mask for SW-MSA
[0211] attn_mask = self.calculate_mask(H, W)
[0212] # cyclic shift
[0213] shifted_x = torch.roll(x, shifts=(-self.shift_size, -self.shift_size), dims=(1, 2))
[0214] # partition windows
[0215] x_windows = window_partition(shifted_x, self.window_size)
[0216] x_windows = x_windows.view(-1, self.window_size * self.window_size,self.dim)
[0217] # W-MSA / SW-MSA
[0218] attn_windows = self.attn(x_windows, mask=attn_mask)
[0219] # merge windows
[0220] attn_windows = attn_windows.view(-1, self.window_size, self.window_size, self.dim)
[0221] shifted_x = window_reverse(attn_windows, self.window_size, H, W)
[0222] # reverse cyclic shift
[0223] x = torch.roll(shifted_x, shifts=(self.shift_size, self.shift_size),dims=(1, 2))
[0224] #FFN
[0225] x = x + self.drop_path(self.mlp(self.norm2(x)))
[0226] return x
[0227] In order to capture feature information at different resolutions, we will apply a multi-scale attention mechanism in the Swin Transformer module. This means that at different levels of the network, we will have windows of different sizes for attention calculations to capture features of different scales.
[0228] Then, we build a network structure consisting of multiple Swin Transformer modules and add input and output layers. The input layer accepts the raw image data, while the output layer is responsible for producing the final segmentation result.
[0229] The following is a specific code example:
[0230] class SwinTransformer(nn.Module):
[0231] def __init__(self, img_size=224, patch_size=4, in_chans=3, num_classes=1000, embed_dim=96, depths=[2, 2, 6, 2],
[0232] num_heads=[3, 6, 12, 24], window_size=7, mlp_ratio=4., qkv_bias=True,qk_scale=None,
[0233] drop_rate=0., attn_drop_rate=0., drop_path_rate=0.1, norm_layer=nn.LayerNorm, ape=False,
[0234] patch_norm=True, use_checkpoint=False, **kwargs):
[0235] super().__init__()
[0236] # Split image into non-overlapping patches
[0237] self.patch_embed = PatchEmbed(
[0238] img_size=img_size, patch_size=patch_size, in_chans=in_chans, embed_dim=embed_dim,
[0239] norm_layer=norm_layer if patch_norm else None)
[0240] num_patches = self.patch_embed.num_patches
[0241] patches_resolution = self.patch_embed.patches_resolution
[0242] self.patches_resolution = patches_resolution
[0243] # Absolute Position Embedding
[0244] self.absolute_pos_embed = nn.Parameter(torch.zeros(1, num_patches,embed_dim))
[0245] self.pos_drop = nn.Dropout(p=drop_rate)
[0246] # stochastic depth
[0247] dpr = [x.item() for x in torch.linspace(0, drop_path_rate, sum(depths))] # stochastic depth decay rule
[0248] # build layers
[0249] self.layers = nn.ModuleList()
[0250] for i_layer in range(len(depths)):
[0251] layer = BasicLayer(dim=int(embed_dim * 2 ** i_layer),
[0252] input_resolution=(patches_resolution[0] / / (2 ** i_layer),
[0253] patches_resolution[1] / / (2 ** i_layer)),
[0254] depth=depths[i_layer],
[0255] num_heads=num_heads[i_layer],
[0256] window_size=window_size,
[0257] mlp_ratio=mlp_ratio,
[0258] qkv_bias=qkv_bias, qk_scale=qk_scale,
[0259] drop=drop_rate, attn_drop=attn_drop_rate,
[0260] drop_path=dpr[sum(depths[:i_layer]):sum(depths[:i_layer + 1])],
[0261] norm_layer=norm_layer,
[0262] downsample=PatchMerging if (i_layer<len(depths) - 1) else None,
[0263] use_checkpoint=use_checkpoint)
[0264] self.layers.append(layer)
[0265] self.norm = norm_layer(self.num_features)
[0266] self.avgpool = nn.AdaptiveAvgPool1d(1)
[0267] self.head = nn.Linear(self.num_features, num_classes) if num_classes>0 else nn.Identity()
[0268] self.apply(self._init_weights)
[0269] def _init_weights(self, m):
[0270] if isinstance(m, nn.Linear):
[0271] trunc_normal_(m.weight, std=.02)
[0272] if isinstance(m, nn.Linear) and m.bias is not None:
[0273] nn.init.constant_(m.bias, 0)
[0274] elif isinstance(m, nn.LayerNorm):
[0275] nn.init.constant_(m.bias, 0)
[0276] nn.init.constant_(m.weight, 1.0)
[0277] @torch.jit.ignore
[0278] def no_weight_decay(self):
[0279] return {'absolute_pos_embed'}
[0280] @torch.jit.ignore
[0281] def no_weight_decay_keywords(self):
[0282] return {'relative_position_bias_table'}
[0283] def forward_features(self, x):
[0284] x = self.patch_embed(x)
[0285] x = x + self.absolute_pos_embed
[0286] x = self.pos_drop(x)
[0287] for layer in self.layers:
[0288] x = layer(x)
[0289] x = self.norm(x) # B L C
[0290] x = self.avgpool(x.transpose(1, 2)) # B C 1
[0291] x = torch.flatten(x, 1)
[0292] return x
[0293] def forward(self, x):
[0294] x = self.forward_features(x)
[0295] x = self.head(x)
[0296] return x
[0297] Finally, we train the network structure by preprocessing the data, defining the loss function and the optimizer. For example, we can use Cross-Entropy Loss as the loss function and choose Adam as the optimizer.
[0298] The following is a specific code example:
[0299] # Data preprocessing
[0300] transform = transforms.Compose([
[0301] transforms.Resize((img_size, img_size)),
[0302] transforms.ToTensor(),
[0303] transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224,0.225]),])
[0304] train_dataset=datasets.ImageFolder(train_dir,transform=transform)
[0305] train_loader = DataLoader(train_dataset, batch_size=batch_size,shuffle=True)
[0306] # Define the model
[0307] model = SwinTransformer().to(device)
[0308] # Define loss function and optimizer
[0309] criterion = nn.CrossEntropyLoss()
[0310] optimizer=torch.optim.Adam(model.parameters(), lr=learning_rate)
[0311] # Train the model
[0312] num_epochs = 10
[0313] for epoch in range(num_epochs):
[0314] model.train()
[0315] for images, labels in train_loader:
[0316] images, labels = images.to(device), labels.to(device)
[0317] optimizer.zero_grad()
[0318] outputs = model(images)
[0319] loss = criterion(outputs, labels)
[0320] loss.backward()
[0321] optimizer.step()
[0322] Through the above steps, we have completed the construction and training process of a deep learning model based on the Swin Transformer architecture. This model can be used to handle image defect detection tasks. By adjusting the network structure and training strategy, we can further improve the performance of the model.
[0323] Figure 2 A structural diagram of an image defect segmentation system based on Swin Transformer is provided for an embodiment of the present application. Figure 2 As shown, the system includes:
[0324] An acquisition input module 21 is used to acquire the image to be detected and input it into a deep learning model based on the Swin Transformer architecture;
[0325] An extraction and generation module 22, used to perform multi-scale feature extraction on the image to be detected using the deep learning model, and generate a feature map containing image defect information;
[0326] A segmentation module 23, used to perform image defect segmentation according to the feature map to obtain a segmentation result;
[0327] In the process of image defect segmentation, multi-task learning is performed, and the multi-task learning includes at least one of the following sub-tasks or any combination of several of them: defect segmentation, defect type classification, and defect position location;
[0328] The segmentation module 23 is specifically used to apply a conditional random field to the feature map to obtain a feature map optimized by the conditional random field, wherein the conditional random field is used to refine the segmentation boundary of the feature map and improve the segmentation accuracy; introduce a dynamic weight adjustment mechanism into the deep learning model to dynamically adjust the weight of the loss function corresponding to the deep learning model according to the importance of different subtasks; input the feature map optimized by the conditional random field into the deep learning module introduced with the dynamic weight adjustment mechanism, so as to output the segmentation result through the deep learning module;
[0329] The applying of the conditional random field to the feature map includes: using the energy function optimized by the conditional random field to refine the segmentation boundary of the feature map to obtain the feature map optimized by the conditional random field; wherein the energy function is composed of a data term, a smooth term, a context consistency term, a task coordination term, and a priori term, and the expression of the energy function includes:
[0330]
[0331] in, represents the energy function, represents the segmentation result, are the weight coefficients of the smooth term, contextual consistency term, task coordination term, and prior term, respectively. is a data item, is a smooth term, is the contextual consistency term, It is a task coordination item. is a priori term.
[0332] Figure 2 The image defect segmentation system based on Swin Transformer can perform Figure 1 The implementation principle and technical effect of the Swin Transformer-based image defect segmentation method described in the illustrated embodiment will not be described in detail. The specific manner in which each module and unit performs operations in the Swin Transformer-based image defect segmentation system in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.
[0333] In one possible design, Figure 2 The image defect segmentation system based on Swin Transformer in the embodiment shown can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;
[0334] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .
[0335] The processing component 32 is used to: obtain the image to be detected and input it into a deep learning model based on the Swin Transformer architecture; use the deep learning model to perform multi-scale feature extraction on the image to be detected, and generate a feature map containing image defect information; perform image defect segmentation according to the feature map to obtain a segmentation result; wherein, in the process of image defect segmentation, multi-task learning is performed, and the multi-task learning includes at least one of the following sub-tasks or any combination of several of them: defect segmentation, defect type classification, and defect position location.
[0336] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.
[0337] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0338] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.
[0339] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.
[0340] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.
[0341] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.
[0342] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 The image defect segmentation method based on Swin Transformer in the illustrated embodiment.
[0343] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0344] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.
[0345] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0346] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. An image defect segmentation method based on Swin Transformer, characterized in that: include: Obtain the image to be detected and input it into the deep learning model based on the Swin Transformer architecture; Using the deep learning model to perform multi-scale feature extraction on the image to be detected, and generate a feature map containing image defect information; Perform image defect segmentation according to the feature map to obtain a segmentation result; In the process of image defect segmentation, multi-task learning is performed, and the multi-task learning includes at least one of the following sub-tasks or any combination of several of them: defect segmentation, defect type classification, and defect position location; The image defect segmentation is performed according to the feature map to obtain the segmentation result, including: applying a conditional random field to the feature map to obtain a feature map optimized by the conditional random field, wherein the conditional random field is used to refine the segmentation boundary of the feature map and improve the segmentation accuracy; introducing a dynamic weight adjustment mechanism into the deep learning model to dynamically adjust the weight of the loss function corresponding to the deep learning model according to the importance of different subtasks; and inputting the feature map optimized by the conditional random field into a deep learning module introduced with the dynamic weight adjustment mechanism to output the segmentation result through the deep learning module; The applying of the conditional random field to the feature map includes: using the energy function optimized by the conditional random field to refine the segmentation boundary of the feature map to obtain the feature map optimized by the conditional random field; wherein the energy function is composed of a data term, a smooth term, a context consistency term, a task coordination term, and a priori term, and the expression of the energy function includes: E(u)=E data (u)+λ1E smooth (u)+λ2E context (u)+λ3E task (u)+λ4E prior (the) Among them, E(u) represents the energy function, u represents the segmentation result, λ1, λ2, λ3, and λ4 are the weight coefficients of the smoothness term, context consistency term, task coordination term, and prior term, respectively. data is a data item, E smooth is the smooth term, E context is the contextual consistency term, E task is the task coordination item, E prior is a priori term; Among them, the data item E data Defined as: Among them, P(y i |f(x i ; θ)) is the probability distribution f(x) predicted by the deep learning model based on parameter θ for pixel i i ;θ), the true label y of pixel i i The conditional probability of y is used to measure the match between the probability distribution predicted by the deep learning model for pixel i and the conditional probability of the true label, i is the true label of pixel i, f(x i ; θ) is the probability distribution of pixel i predicted by the deep learning model based on parameter θ, α j is the regularization coefficient, which is used to control the regularization term D j (u) strength, D j (u) is the jth regularization term, which is used to restrict the distribution of predicted labels; η k is the weight coefficient of the supervisory signal term, which is used to control the supervisory signal term S k (u) strength, S k (u) is the kth supervisory signal term, which is used to constrain the prediction results from different angles; μ m is the weight coefficient of the attention item, controlling the attention item A m (u) strength, A m (u) is the mth attention item, which is used to enhance the model’s attention to the key area; i represents the pixel index in the image; j represents the index of the regularization item; k represents the index of the supervision signal item; m represents the index of the attention item; Among them, the smooth term E smooth Defined as: Among them, (i, j) represents the connection relationship between pixel i and pixel j, w ij represents the smoothing weight between pixel i and pixel j, ψ unary is a binary potential function that penalizes inconsistent label assignments between pixel i and pixel j, ψ unary (u i ,u j ) is used to measure the label consistency between pixel i and pixel j; (i, k) represents the long-range connection relationship between pixel i and non-neighborhood pixel k, β k is the long-range connection weight, φ k is a long-range potential function used to deal with the relationship between pixel i and non-neighborhood pixel k, φ k (u i ,u k ) is used to measure the label consistency between pixel i and non-neighborhood pixel k; ω l is the weight coefficient of the regularization term, which is used to control the regularization term R l (u) The degree of influence, R l (u) is the lth regularization term, which is used to increase the smoothness of the model; ρ p is the weight coefficient of the high-order potential function, which is used to control the high-order potential function H p The influence degree of (u), H p (u) is the pth high-order potential function, which is used to further enhance the nonlinear fitting ability of the model; N is the neighborhood set, which contains the pairs of pixels j that are directly adjacent to pixel i; K is the non-neighborhood set, which contains the pairs of pixels k that are not directly adjacent to pixel i but have a certain spatial relationship; l is the index of the regularization term; p is the index of the high-order potential function; Among them, the context consistency term E context Defined as: Where C represents the context area, γ C is the weight coefficient of the context consistency function, χ C is a context consistency function, χ C (u)=∑ i∈C ∑ j∈C w ij ψ context (u i ,u j ), w ij represents the smoothing weight between pixel i and pixel j, ψ context (u i ,u j ) is used to measure the label consistency between pixel i and pixel j in the context region; G represents the global guidance region, δ G is the weight coefficient of the global guidance term, L G (u) is the global guidance term, which is used to guide the consistency of local segmentation. represents the neighborhood set of pixel l, ψ global (u i ,u j ) is used to measure the label consistency between pixel i and pixel j in the global guidance region; V is the visual attention mechanism region, ν V is the weight coefficient of the visual attention mechanism term, T V (u) is the visual attention mechanism term, which is used to enhance the attention to the visually salient area, T V (u)=∑ i∈V ω i (1-attention(u i )),ω i represents the weight coefficient of the visually salient area, attention(u i ) is used to measure the attention score of pixel i in the visually salient area; Among them, the task coordination item E task Defined as: Among them, T represents the subtask type, τ T is the weight coefficient of the subtask, ζ T (u) is a function that measures the completion of the subtask and is used to determine the difference between the predicted label and the true label. u i represents the predicted label of pixel i, represents the true label of pixel i in subtask T; B represents the balance factor between tasks, β B is the weight coefficient of the balance factor, F B (u) is the balance factor function, which is used to adjust the relative importance of different tasks. represents the average loss of all subtasks; The prior term E prior Defined as: Among them, P represents the type of prior knowledge, π P is the weight coefficient of prior knowledge, Ψ P (u) is a prior knowledge item.
2. The method according to claim 1, characterized in that Before obtaining the image to be detected and inputting it into the deep learning model based on the Swin Transformer architecture, it also includes: Build a deep learning model based on the Swin Transformer architecture; The construction of a deep learning model based on the Swin Transformer architecture includes: Initialize the deep learning framework; Define the Swin Transformer module that includes the self-attention mechanism and the feedforward network; Applying a multi-scale attention mechanism in the Swin Transformer module to capture feature information at different resolutions; Constructing a network structure consisting of multiple Swin Transformer modules in the deep learning framework, and adding an input layer and an output layer to the network structure; The network structure is trained by preprocessing data, defining a loss function and an optimizer to complete the training of the deep learning model.
3. The method according to claim 1, characterized in that The using the deep learning model to extract multi-scale features from the image to be detected and generating a feature map containing image defect information includes: Deploy Swin Transformer modules at different resolution levels of the deep learning model, and the Swin Transformer modules at each resolution level are used to capture feature information of a specific scale; A feature fusion strategy is adopted at each resolution level, wherein the feature fusion strategy includes at least a skip connection, a pyramid pooling module or a dense connection to promote the complementarity and fusion of the feature information between different levels; Integrate feature information at different scales to generate image feature representation; Based on the image feature representation, feature information of different scales is mapped into a feature space to generate a feature map containing image defect information.
4. An image defect segmentation system based on Swin Transformer, characterized in that: include: The input module is used to obtain the image to be detected and input it into the deep learning model based on the Swin Transformer architecture; An extraction and generation module, used to perform multi-scale feature extraction on the image to be detected using the deep learning model, and generate a feature map containing image defect information; A segmentation module, used to perform image defect segmentation according to the feature map to obtain a segmentation result; In the process of image defect segmentation, multi-task learning is performed, and the multi-task learning includes at least one of the following sub-tasks or any combination of several of them: defect segmentation, defect type classification, and defect position location; The segmentation module is specifically used to apply a conditional random field to the feature map to obtain a feature map optimized by the conditional random field, wherein the conditional random field is used to refine the segmentation boundary of the feature map and improve the segmentation accuracy; a dynamic weight adjustment mechanism is introduced into the deep learning model to dynamically adjust the weight of the loss function corresponding to the deep learning model according to the importance of different subtasks; the feature map optimized by the conditional random field is input into the deep learning module introduced with the dynamic weight adjustment mechanism, so as to output the segmentation result through the deep learning module; The applying of the conditional random field to the feature map includes: using the energy function optimized by the conditional random field to refine the segmentation boundary of the feature map to obtain the feature map optimized by the conditional random field; wherein the energy function is composed of a data term, a smooth term, a context consistency term, a task coordination term, and a priori term, and the expression of the energy function includes: E(u)=E data (u)+λ1E smooth (u)+λ2E context (u)+λ3E task (u)+λ4E prior (the) Among them, E(u) represents the energy function, u represents the segmentation result, λ1, λ2, λ3, and λ4 are the weight coefficients of the smoothness term, context consistency term, task coordination term, and prior term, respectively. data is a data item, E smooth is the smooth term, E context is the contextual consistency term, E task is the task coordination item, E prior is a priori term; Among them, the data item E data Defined as: Among them, P(y i |f(x i ; θ)) is the probability distribution f(x) predicted by the deep learning model based on parameter θ for pixel i i ;θ), the true label y of pixel i i The conditional probability of y is used to measure the match between the probability distribution predicted by the deep learning model for pixel i and the conditional probability of the true label, i is the true label of pixel i, f(x i ; θ) is the probability distribution of pixel i predicted by the deep learning model based on parameter θ, α j is the regularization coefficient, which is used to control the regularization term D j (u) strength, D j (u) is the jth regularization term, which is used to restrict the distribution of predicted labels; η k is the weight coefficient of the supervisory signal term, which is used to control the supervisory signal term S k (u) strength, S k (u) is the kth supervisory signal term, which is used to constrain the prediction results from different angles; μ m is the weight coefficient of the attention item, controlling the attention item A m (u) strength, A m (u) is the mth attention item, which is used to enhance the model’s attention to the key area; i represents the pixel index in the image; j represents the index of the regularization item; k represents the index of the supervision signal item; m represents the index of the attention item; Among them, the smooth term E smooth Defined as: Among them, (i, j) represents the connection relationship between pixel i and pixel j, w ij represents the smoothing weight between pixel i and pixel j, ψ unary is a binary potential function that penalizes inconsistent label assignments between pixel i and pixel j, ψ unary (u i ,u j ) is used to measure the label consistency between pixel i and pixel j; (i, k) represents the long-range connection relationship between pixel i and non-neighborhood pixel k, β k is the long-range connection weight, φ k is a long-range potential function used to deal with the relationship between pixel i and non-neighborhood pixel k, φ k (u i ,u k ) is used to measure the label consistency between pixel i and non-neighborhood pixel k; ω l is the weight coefficient of the regularization term, which is used to control the regularization term R l (u) The degree of influence, R l (u) is the lth regularization term, which is used to increase the smoothness of the model; ρ p is the weight coefficient of the high-order potential function, which is used to control the high-order potential function H p The influence degree of (u), H p (u) is the pth high-order potential function, which is used to further enhance the nonlinear fitting ability of the model; N is the neighborhood set, which contains the pairs of pixels j that are directly adjacent to pixel i; K is the non-neighborhood set, which contains the pairs of pixels k that are not directly adjacent to pixel i but have a certain spatial relationship; l is the index of the regularization term; p is the index of the high-order potential function; Among them, the context consistency term E context Defined as: Where C represents the context area, γ C is the weight coefficient of the context consistency function, χ C is a context consistency function, χ C (u)=∑ i∈C ∑ j∈C w ij ψ context (u i ,u j ), w ij represents the smoothing weight between pixel i and pixel j, ψ context (u i ,u j ) is used to measure the label consistency between pixel i and pixel j in the context region; G represents the global guidance region, δ G is the weight coefficient of the global guidance term, L G (u) is the global guidance term, which is used to guide the consistency of local segmentation. represents the neighborhood set of pixel l, ψ global (u i ,u j ) is used to measure the label consistency between pixel i and pixel j in the global guidance region; V is the visual attention mechanism region, ν V is the weight coefficient of the visual attention mechanism term, T V (u) is the visual attention mechanism term, which is used to enhance the attention to the visually salient area, T V (u)=∑ i∈V ω i (1-attention(u i )),ω i represents the weight coefficient of the visually salient area, attention(u i ) is used to measure the attention score of pixel i in the visually salient area; Among them, the task coordination item E task Defined as: Among them, T represents the subtask type, τ T is the weight coefficient of the subtask, ζ T (u) is a function that measures the completion of the subtask and is used to determine the difference between the predicted label and the true label. u i represents the predicted label of pixel i, represents the true label of pixel i in subtask T; B represents the balance factor between tasks, β B is the weight coefficient of the balance factor, F B (u) is the balance factor function, which is used to adjust the relative importance of different tasks. represents the average loss of all subtasks; The prior term E prior Defined as: Among them, P represents the type of prior knowledge, π P is the weight coefficient of prior knowledge, Ψ P (u) is a prior knowledge item.
5. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the SwinTransformer-based image defect segmentation method according to any one of claims 1 to 3.
6. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the image defect segmentation method based on Swin Transformer described in any one of claims 1 to 3 is implemented.
Citation Information
Patent Citations
Small sample image segmentation method based on guide network and full-connection conditional random field
CN113223011A
High-resolution remote sensing image vegetation extraction method based on sensitive feature focusing perception
CN113609889A
