A method for segmenting defects in copper strip images based on space-time domain

Abstract

This invention relates to a spatiotemporal domain-based defect segmentation method for copper strip images, belonging to the field of copper strip defect segmentation technology. The method includes: acquiring a copper strip image; inputting the copper strip image into a pre-constructed defect segmentation model; and finally outputting the segmentation result of the model. The model construction includes: acquiring a time-frequency domain dual-branch attention network model; incorporating the dual-backbone approach and adding a DCT frequency domain enhancement branch based on physical mechanisms; fusing feature outputs from different scales of the dual backbones using a weighted fusion approach; utilizing a triple attention mechanism in the neck pyramid feature fusion stage to obtain the constructed model; the frequency domain enhancement branch uses frequency domain information to compensate for the loss of spatial domain information, the weighted fusion approach allows the model to adaptively fuse important information, and the triple attention mechanism supplements details for high-resolution layers and provides semantics for low-resolution layers, improving the accuracy of defect segmentation.

Description

Technical Field

[0001] This invention relates to a method for segmenting defects in copper strip images based on the spatiotemporal domain, belonging to the field of copper strip defect segmentation technology. Background Technology

[0002] With the continuous improvement of the level of intelligent manufacturing, intelligent segmentation technology for surface defects of metal materials is playing an increasingly important role in industrial quality control. As an important basic material in key fields such as electronics, energy, and aerospace, copper strips have surface defects (such as pits, scratches, black spots, etc.) that directly affect the quality of subsequent processes and product performance. At present, the segmentation methods for surface defects of copper strips mainly rely on traditional image processing technology or manual experience rules. However, such methods have poor adaptability to complex texture backgrounds and weak robustness, making it difficult to meet the requirements of high-precision and automated segmentation.

[0003] In recent years, with the rapid development of deep learning and convolutional neural networks (CNNs), end-to-end image segmentation models have been widely used in industrial visual segmentation. Although the feature extraction capability in the spatial domain has been significantly improved, relying solely on spatial convolution still has certain limitations, making it difficult to fully capture frequency domain features such as periodic textures, global structures, and high-frequency changes in images. Therefore, how to effectively integrate spatial and frequency information to improve the model's ability to identify minute defects in complex backgrounds has become an important direction in current defect segmentation technology research.

[0004] In the task of segmenting defects on the surface of copper strips, the complexity of segmentation algorithms increases significantly due to the presence of various types of defects, such as pits, scratches, and black spots. Existing traditional segmentation methods can usually only handle single-type defects. For the segmentation of multi-category defects, multiple algorithms or rule logics are often required, which increases the computational burden and makes it difficult to ensure the overall segmentation effect. With the increase in the number and diversity of defects, the feature interference faced by the model increases significantly, further increasing the difficulty of the segmentation task. Currently, in the process of segmenting copper strip defects, there are common problems such as the incoordination of spatial and frequency information modeling, insufficient feature fusion, and weak multi-scale structural response capability, which seriously restrict the accuracy and stability of the segmentation model. Summary of the Invention

[0005] To address the challenges of complex background interference and multi-scale defect distribution in copper strip defect segmentation, this invention provides a spatiotemporal domain-based copper strip image defect segmentation method to improve segmentation accuracy and precision.

[0006] The technical solution of this invention is: a method for defect segmentation of copper strip images based on the spatiotemporal domain, the method comprising:

[0007] Step 1: Construct a time-frequency domain dual-branch attention network as a defect segmentation model:

[0008] The time-frequency domain dual-branch attention network includes combining physical mechanisms, adding a DCT frequency domain enhancement branch to the CNN, using frequency domain information to supplement spatial domain information, converting grayscale information into frequency domain information through discrete cosine transform, and weighting the channel information of each layer with weights obtained by average pooling to obtain a dual-branch backbone.

[0009] In the first fusion phase, a weighted fusion approach is used to weight and fuse the information of the main branches of the two branches, and to adaptively fuse the information.

[0010] In the second fusion stage, a triple attention mechanism is added to supplement the details of the high-resolution layer and provide semantics for the low-resolution layer. Attention weights are calculated by capturing the interaction between the three dimensions of channel, height and width through a three-branch structure.

[0011] Step 2: Train the constructed time-frequency domain dual-branch attention network;

[0012] Step 3: Input the image to be segmented into the time-frequency domain dual-branch attention network obtained through training, and obtain the segmentation result image of the image to be segmented. The result includes defect mask, defect detection box and confidence score.

[0013] Further, step two includes: acquiring images of defective copper strips produced in the industry using an industrial camera; using the label-studio image segmentation dataset creation tool to create labeled images with segmentation masks from the original images; dividing the dataset into training set, validation set, and test set; then using the dataset to train the network; using a cosine annealing training strategy; employing AMP mixed precision training; obtaining the optimal weight parameters of the model; and thus obtaining the trained time-frequency domain dual-branch attention network.

[0014] In step three, the image to be segmented is obtained by taking pictures of the copper sheets on the production line using a camera.

[0015] Furthermore, the implementation process of the DCT frequency domain enhancement branch includes the following:

[0016] The input image is subjected to a two-dimensional discrete cosine transform, which transforms the feature map of each channel from the spatial domain to the frequency domain.

[0017] Frequency domain feature normalization and compression;

[0018] Then, channel-dimensional average pooling is performed to obtain the frequency domain response intensity of each channel;

[0019] Channel attention weights are generated through a fully connected layer or a non-linear mapping.

[0020] Apply frequency domain weights to the original spatial domain feature map;

[0021] The enhanced spatial domain features are fused with the original spatial features or the backbone network to form the final output features.

[0022] Furthermore, in step one, after the DCT frequency domain enhancement branch maps and transforms the input feature map to the frequency domain, it constructs channel attention weights using the energy distribution in the frequency domain, weights the input feature map, and enhances the frequency-sensitive key channel response. The formula for the DCT frequency domain enhancement branch processing is expressed as follows:

[0023] Let the feature map input vector be: ;

[0024] For each input channel, a two-dimensional DCT transform is performed as follows:

[0025] ;

[0026] ;

[0027] The flattened frequency domain feature map is as follows: ;

[0028] LayerNorm is standardized as follows:

[0029] ;

[0030] The channel vector is generated as follows: ;

[0031] Channel attention generation MLP is as follows:

[0032] ;

[0033] The final channel weighting is: ;

[0034] Where B is the training batch size, C is the number of channels, and H×W is the feature map space size. It is a 4D tensor, which is the feature map input to the module; For the vertical and horizontal indices of the frequency space in DCT, For pixel position index in the spatial domain, This represents the spatial location of the feature map of the c-th channel in the b-th sample. Pixel value at that location, For the b-th sample, the c-th channel is at the frequency position DCT coefficients, These are the DCT normalization coefficients. , These are the row normalization factor and the column normalization factor, respectively. This indicates how the two normalization factors are determined. H represents the feature map height, which is the number of rows, and W represents the feature map width, which is the number of columns. k represents two ranges, 0-H or 0-W, depending on which direction the DCT transformation is performed. For the DCT flattening vector of the b-th sample and the c-th channel, This means transforming a two-dimensional feature map into a one-dimensional vector, which is actually a one-dimensional array. Each number in the array is the value after the previous two-dimensional DCT transformation.

[0035] yes Normalized vector, yes The mean, yes standard deviation It is an extremely small number. This represents the value after the two-dimensional cosine transform at position i. Because the two-dimensional cosine transform will generate C two-dimensional cosine feature maps, each of which is flattened into a one-dimensional array of length H*W, where i represents the position in the array.

[0036] It is the mean spectral value of the c-th channel in the b-th sample. It is the feature map vector of all channels of the b-th sample. This represents the result of standardizing each number in the H*W one-dimensional array, where i still represents the position of the number in the array;

[0037] It is the product of the normalized frequency coefficients in a two-dimensional DCT. These are the weight parameters of the first fully connected layer. These are the weight parameters of the second fully connected layer. , () is the Sigmoid activation function. It is the channel attention coefficient vector generated by the b-th sample. It is a function that reshapes a one-dimensional vector into a 3D tensor. It is It can be multiplied one channel at a time. These are the channel weight coefficients for the b-th sample. These are all the corresponding channels of the b-th sample in the original input feature map. It is the result after frequency domain weighting.

[0038] Furthermore, in step one, the weighted fusion concept is expressed as follows:

[0039] Concatenated input: ;

[0040] 3×3 depthwise separable convolution:

[0041] ;

[0042] 1×1 convolution: ;

[0043] Global average pooling: ;

[0044] 1×1 convolution dimensionality reduction: ;

[0045] Sigmoid activation ;

[0046] Weight fusion: ;

[0047] Relu6 activation: ;

[0048] In the formula, where, These are two fused feature maps, each with C channels and a size of H×W. yes Feature map after concatenation function; It is a 3×3 depthwise separable convolution. It's a 1×1 convolution operation, using GAP global average pooling. It is the Sigmoid activation function. These are the weighting parameters, and R is the final weighted fusion output. This represents the feature map generated after depthwise separable convolution. represent This feature map is generated after passing through a 1x1 ordinary convolution. yes The feature map generated after global average pooling. yes The feature map is generated after dimensionality reduction through a 1x1 convolution.

[0049] Further, in step one, the triple attention mechanism consists of three branches, each responsible for capturing the interaction features between the spatial dimension H or W and the number of channels C; attention weights are generated by permuting the input tensor in each branch, followed by Z-pooling and a convolutional layer of size k1×k1; these weights are generated through a sigmoid activation layer, then applied to the permuted input tensor, and then transformed back to the original input shape. The calculation formula for the triple attention mechanism is expressed as:

[0050] enter:

[0051] CW direction:

[0052] HC direction:

[0053] HW direction:

[0054]

[0055] In the formula, It is the input feature map. It is a dimensional order transformation function. It is the output feature map after CH dimension swapping. It is a channel weighting function. This is the feature map output after weighting by the CW dimension. This is the output after HC dimension swapping. It is the output feature map after CW dimension swapping. It is the feature map output after HC dimension weighting. This is the output after WC dimension swapping. It is the output after weighting by the HW dimension. It is an average value output after weighting in three different directions.

[0056] Furthermore, in step one, the processing steps of the defect segmentation model include:

[0057] For copper strip defect images, dual-pillar feature extraction, weighted fusion, secondary pyramid fusion, attention mechanism, and finally multi-scale segmentation are performed; specifically including the following:

[0058] After weighting by 2D DCT transformation, the output is a frequency feature map. The original image is convolved by CNN to obtain a spatial domain feature map. The two feature maps are concatenated to form a weighting coefficient. The two feature maps are then fused, followed by a second fusion process, which involves upsampling and downsampling fusion, combined with an attention mechanism, and finally multi-scale segmentation.

[0059] The present invention also provides a copper strip image defect segmentation system based on the spatiotemporal domain, comprising:

[0060] The image acquisition module is used to acquire images of copper strips on the production line using an industrial camera;

[0061] The copper strip defect segmentation module is used to input the copper strip image into a pre-built defect segmentation model with optimal training weights and output the copper strip defect segmentation result.

[0062] The model building module, used for building the defect segmentation model, includes:

[0063] Obtain a time-frequency domain dual-branch attention network;

[0064] The time-frequency domain dual-branch attention network includes combining physical mechanisms, adding a DCT frequency domain enhancement branch to the CNN, using frequency domain information to supplement spatial domain information, converting grayscale information into frequency domain information through discrete cosine transform, and weighting the channel information of each layer with weights obtained by average pooling to obtain a dual-branch backbone.

[0065] In the first fusion phase, a weighted fusion approach is used to weight and fuse the information of the main branches of the two branches, and to adaptively fuse the information.

[0066] In the second fusion stage, a triple attention mechanism is added to supplement the details of the high-resolution layer and provide semantics for the low-resolution layer. Attention weights are calculated by capturing the interaction between the three dimensions of channel, height and width through a three-branch structure.

[0067] Training module: Used to train the constructed time-frequency domain dual-branch attention network.

[0068] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aforementioned method for segmenting copper strip images based on the spatiotemporal domain.

[0069] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned method for segmenting copper strip images based on the spatiotemporal domain.

[0070] The beneficial effects of this invention are:

[0071] 1. This invention improves the accuracy of defect recognition: By introducing two-dimensional discrete cosine transform (2D-DCT) to extract image frequency domain features, this invention can effectively capture high-frequency details and periodic texture information on the surface of copper strips, thereby making up for the lack of fine-grained defect recognition ability of traditional convolutional neural networks in the spatial domain and significantly improving the overall segmentation accuracy.

[0072] 2. This invention achieves deep fusion of the frequency domain and spatial domain through the cooperation of two main trunks. By splicing the frequency feature map and the spatial domain feature map to form weighted coefficients and performing a fusion operation, the model's ability to comprehensively perceive multi-source information is enhanced, the robustness of segmentation of small defects under complex background interference is improved, and the proportion of effective features is increased.

[0073] 3. This invention further introduces upsampling and downsampling structures for secondary feature fusion, and uses attention mechanisms to achieve explicit enhancement of important features, enabling communication and interaction between features of different dimensions, effectively improving the model's response and segmentation capabilities for defect regions of different scales, and adapting to the segmentation needs of multiple categories of defects;

[0074] 4. To address the issue of multiple concurrent defects that may exist in copper strip images, this invention constructs a segmentation structure that combines discriminative power and generalization ability. It can achieve unified processing of multiple types of defects without increasing additional annotation costs. It is suitable for real-time defect segmentation systems in embedded devices or industrial production lines, meeting the practical needs of high-efficiency, large-batch image segmentation. Attached Figure Description

[0075] Figure 1 The diagram shown is a flowchart of one embodiment of the defect segmentation method for copper strips of the present invention.

[0076] Figure 2 The diagram shown is a structural schematic of the defect segmentation model of the present invention in one embodiment;

[0077] Figure 3 This is a schematic diagram of the structure of the DCT frequency enhancement module of the present invention in one embodiment;

[0078] Figure 4 This is a schematic diagram of the fusion module of the present invention in one embodiment;

[0079] Figure 5 This is a schematic diagram of the attention mechanism module of the present invention in one embodiment;

[0080] Figure 6 This is a schematic diagram of the defect segmentation results of the copper strip of the present invention in one embodiment;

[0081] Figure 7 This is a schematic diagram of the defect segmentation result of the copper strip of the present invention in another embodiment. Detailed Implementation

[0082] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0083] The term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three cases: A alone, A and B at the same time, and B alone. In addition, the character " / " generally indicates that the related objects before and after it are in an "or" relationship.

[0084] Example 1: As Figures 1-7 As shown, a method for defect segmentation of copper strip images based on the spatiotemporal domain is proposed, the method comprising:

[0085] Step 1: Construct a time-frequency domain dual-branch attention network as a defect segmentation model:

[0086] To obtain more accurate image segmentation, the time-frequency domain dual-branch attention network incorporates a DCT frequency domain enhancement branch into the CNN, based on physical mechanisms, such as... Figure 3 As shown, frequency domain information is used to supplement spatial domain information, grayscale information is transformed into frequency domain information through discrete cosine transform, and the channel information of each layer is weighted by the weights obtained by average pooling to obtain a dual-branch backbone.

[0087] In the first fusion phase, a weighted fusion approach is used to weight and fuse the information of the main branches of the two branches, and to adaptively fuse the information.

[0088] In the second fusion stage, a triple attention mechanism is added to supplement the details of the high-resolution layer and provide semantics for the low-resolution layer. Attention weights are calculated by capturing the interaction between the three dimensions of channel, height and width through a three-branch structure.

[0089] In step one, the attention mechanism is integrated into the dual backbone of feature extraction. A triple attention mechanism is introduced before the backbone feature extraction to strengthen the backbone’s feature extraction strength. This makes up for the problem that the backbone of traditional segmentation networks has a weak ability to extract information. It can cooperate more efficiently with the neck information fusion process. Finally, it enters the segmentation head and outputs the segmentation result image.

[0090] Before entering the segmentation head, the feature map enters the attention mechanism to further enhance the features. First, the three-branch structure captures the interaction between the three dimensions of channel, height and width to calculate the attention weights. Then, the input tensor is rotated and residual transformation is applied to establish the dependencies between different dimensions. The core is to emphasize the importance of capturing cross-dimensional dependencies when calculating attention weights.

[0091] CNN (Convolutional Neural Network) is a local filter in the spatial domain, mainly focusing on large-scale textures and edges. It is easy to lose information about small defects. DCT (Derivative Transform) transforms the image into the frequency domain, which is more efficient for recognizing some stereo features. When combined, CNN is responsible for learning the spatial structure, and DCT is responsible for supplementing the frequency domain features, making it easier to capture small defects.

[0092] Furthermore, in step one, the implementation of the DCT frequency domain enhancement branch is as follows:

[0093] The input image is subjected to a two-dimensional discrete cosine transform, which transforms the feature map of each channel from the spatial domain to the frequency domain.

[0094] Frequency domain feature normalization and compression;

[0095] Then, channel-dimensional average pooling is performed to obtain the frequency domain response intensity of each channel;

[0096] Channel attention weights are generated through a fully connected layer or a non-linear mapping.

[0097] Frequency domain weights Acting on the original spatial domain feature map ;

[0098] The enhanced spatial domain features are fused with the original spatial features or backbone network (weighted addition or splicing) to form the final output features;

[0099] By automatically learning important channels through frequency domain statistics and dynamically weighting them, redundant or useless features can be effectively suppressed, and information utilization can be improved.

[0100] High-frequency information in images often contains details but also noise. DCT provides the ability to suppress unwanted frequency band noise from a frequency perspective, thereby improving the robustness of the model.

[0101] Traditional convolution can only extract image information in the spatial domain and lacks the ability to model frequency characteristics. It is prone to inaccurate recognition, especially in areas with blurred edges and low contrast. By extracting the frequency domain features of the image through DCT branches, the model's ability to perceive detailed textures, boundary contours, and high-frequency defects is effectively enhanced, thus improving the problem of inaccurate recognition of blurred areas by traditional convolution.

[0102] Spatial convolution is sensitive to random noise and local artifacts in images, and is prone to introducing erroneous responses. It performs poorly, especially in industrial images with complex surface textures such as metal and plastic. Frequency domain features are naturally insensitive to low-frequency background changes and local noise, and can stably extract structural features in complex industrial environments, thereby improving the overall segmentation accuracy and stability.

[0103] Furthermore, in step one, DCT is a method for transforming an image from the spatial domain to the frequency domain. After the transformation, low-frequency contour information and high-frequency texture and edge information can be separated. The DCT frequency domain enhancement branch maps and transforms the input feature map to the frequency domain, and then uses the energy distribution in the frequency domain to construct channel attention weights to weight the input feature map, thereby enhancing the frequency-sensitive key channel responses. The formula for the DCT frequency domain enhancement branch processing is expressed as follows:

[0104] Let the feature map input vector be: ;

[0105] For each input channel, a two-dimensional DCT transform is performed as follows:

[0106] ;

[0107] ;

[0108] The flattened frequency domain feature map is as follows: ;

[0109] LayerNorm is standardized as follows:

[0110] ;

[0111] The channel vector is generated as follows: ;

[0112] Channel attention generation MLP is as follows:

[0113] ;

[0114] The final channel weighting is: ;

[0115] Where B is the training batch size, C is the number of channels, and H×W is the feature map space size. It is a 4D tensor, which is the feature map input to the module; For the vertical and horizontal indices of the frequency space in DCT, For pixel position index in the spatial domain, This represents the spatial location of the feature map of the c-th channel in the b-th sample. Pixel value at that location, For the b-th sample, the c-th channel is at the frequency position DCT coefficients, These are the DCT normalization coefficients. , These are the row normalization factor and the column normalization factor, respectively. This indicates how the two normalization factors are determined. H represents the feature map height, which is the number of rows, and W represents the feature map width, which is the number of columns. k represents two ranges, 0-H or 0-W, depending on which direction the DCT transformation is performed. For the DCT flattening vector of the b-th sample and the c-th channel, This means transforming a two-dimensional feature map into a one-dimensional vector, which is actually a one-dimensional array. Each number in the array is the value after the previous two-dimensional DCT transformation.

[0116] yes Normalized vector, yes The mean, yes standard deviation It is an extremely small number. This represents the value after the two-dimensional cosine transform at position i. Because the two-dimensional cosine transform will generate C two-dimensional cosine feature maps, each of which is flattened into a one-dimensional array of length H*W, where i represents the position in the array.

[0117] It is the mean spectral value of the c-th channel in the b-th sample. It is the feature map vector of all channels of the b-th sample. This represents the result of standardizing each number in the H*W one-dimensional array, where i still represents the position of the number in the array;

[0118] It is the product of the normalized frequency coefficients in a two-dimensional DCT. These are the weight parameters of the first fully connected layer. These are the weight parameters of the second fully connected layer. , () is the Sigmoid activation function. It is the channel attention coefficient vector generated by the b-th sample. It is a function that reshapes a one-dimensional vector into a 3D tensor. It is It can be multiplied one channel at a time. These are the channel weight coefficients for the b-th sample. These are all the corresponding channels of the b-th sample in the original input feature map. It is the result after frequency domain weighting.

[0119] Furthermore, the fusion in the first stage is a weighted fusion, with the structure as follows: Figure 4As shown, in step one, the weighted fusion concept is expressed by the following formula:

[0120] Concatenated input: ;

[0121] 3×3 depthwise separable convolution:

[0122] ;

[0123] This invention employs a lightweight design to: It requires fewer parameters and is faster than regular convolution;

[0124] 1×1 convolution: ;

[0125] Global average pooling: ;

[0126] 1×1 convolution dimensionality reduction: ;

[0127] Sigmoid activation ;

[0128] Weight fusion: ;

[0129] Relu6 activation: ;

[0130] In the formula, where, These are two fused feature maps, each with C channels and a size of H×W. yes Feature map after concatenation function; It is a 3×3 depthwise separable convolution. It's a 1×1 convolution operation, using GAP global average pooling. It is the Sigmoid activation function. These are the weighting parameters, and R is the final weighted fusion output. This represents the feature map generated after depthwise separable convolution. represent This feature map is generated after passing through a 1x1 ordinary convolution. yes The feature map generated after global average pooling. yes The feature map is generated after dimensionality reduction through a 1x1 convolution.

[0131] Further, in step one, the triple attention mechanism consists of three branches, each responsible for capturing the interaction features between the spatial dimension H or W and the number of channels C. Attention weights are generated by permuting the input tensor in each branch, followed by Z-pooling and a convolutional layer of size k1×k1. These weights are generated through a sigmoid activation layer, applied to the permuted input tensor, and then transformed back to the original input shape. The structural diagram of the triple attention module corresponding to the triple attention mechanism is shown below. Figure 5 As shown, the calculation formula is expressed as follows:

[0132] enter:

[0133] CW direction:

[0134] IC direction:

[0135] HW direction:

[0136]

[0137] In the formula, It is the input feature map. It is a dimensional order transformation function. It is the output feature map after CH dimension swapping. It is a channel weighting function. This is the feature map output after weighting by the CW dimension. This is the output after HC dimension swapping. It is the output feature map after CW dimension swapping. It is the feature map output after HC dimension weighting. This is the output after WC dimension swapping. It is the output after weighting by the HW dimension. It is an average value output after weighting in three different directions.

[0138] The proposed triple attention module utilizes a three-branch structure to capture cross-dimensional interactions of input data, thereby calculating attention weights. This method can construct interdependencies between input channels or spatial locations with low computational cost. Triple attention consists of three branches, each responsible for capturing the interaction features between spatial dimension H or W and channel dimension C. Attention weights are generated by permuting the input tensor in each branch, followed by Z-pooling and a k×k convolutional layer. These weights are generated through a sigmoid activation layer, applied to the permuted input tensor, and then transformed back to the original input shape.

[0139] Using dual-backbone fusion and attention-weighted mode to process target segmentation tasks can effectively balance accuracy and speed, making it more suitable for real-world application needs.

[0140] Furthermore, in step one, the processing steps of the defect segmentation model include:

[0141] For copper strip defect images, dual-pillar feature extraction, weighted fusion, secondary pyramid fusion, attention mechanism, and finally multi-scale segmentation are performed; specifically including the following:

[0142] After weighting by 2D DCT transformation, the output is a frequency feature map. The original image is convolved by CNN to obtain a spatial domain feature map. The two feature maps are concatenated to form a weighting coefficient. The two feature maps are then fused, followed by a second fusion process, which involves upsampling and downsampling fusion, combined with an attention mechanism, and finally multi-scale segmentation.

[0143] Step 2: Train the constructed time-frequency domain dual-branch attention network. Step 2 includes: acquiring images of defective copper strips produced in the industry using an industrial camera; using the label-studio image segmentation dataset creation tool to create labeled images with segmentation masks from the original images; dividing the dataset into training, validation, and test sets in an 8:1:1 ratio; then training the network using the dataset; using a cosine annealing training strategy and AMP mixed precision training to obtain the optimal weight parameters of the model, thereby obtaining the trained time-frequency domain dual-branch attention network.

[0144] The training set is the sample used during model training. The model performs forward propagation, backward propagation, and parameter updates on this set in order to learn the mapping relationship between input images and labels.

[0145] The validation set is not involved in updating the model weights. The model performance is evaluated on the validation set after each training batch in order to update the model's hyperparameters and control overfitting.

[0146] The test set is a collection of images used to test the performance of various parameters of the model, with the aim of testing the model's performance.

[0147] In industrial defect segmentation, samples are scarce and the proportion of outliers is low. An 8:1:1 allocation ensures that the model has enough positive samples for training, while maintaining the stability and reliability of the evaluation process, which is particularly crucial for the practical application of industrial models.

[0148] Then, the initial model is trained, the trained model is tested on the test set, the optimal training parameters are selected, and the optimal trained model is obtained.

[0149] The defect segmentation model includes a Backbone module, a Fusion module, a Neck module, an Attention module, and a Head module. The structure diagram of the defect segmentation model is shown below. Figure 2 As shown;

[0150] in:

[0151] The backbone module includes the following DCT frequency branches: First ConvModule, Second ConvModule, First dct_channel_block_2d Module, Third ConvModule, Second dct_channel_block_2d Module, Fourth ConvModule, Third dct_channel_block_2d Module, Fifth ConvModule, Fourth dct_channel_block_2d Module, First SPPF, First C2PSA Module; and the CNN branch: Sixth ConvModule, Seventh ConvModule, First CNN Module, Eighth ConvModule, Second CNN Module, Ninth ConvModule, Third CNN Module, Tenth ConvModule, Fourth CNN Module, Second SPPF, Second C2PSA Module, First FusionModule, Second FusionModule, Third FusionModule.

[0152] The Neck module includes the first Upsample module, the first Concat module, the fifth CNN module, the second Upsample module, the Concat module, the sixth CNN module, the first TripletAttention module, the eleventh ConvModule module, the second Concat module, the seventh CNN module, the second TripletAttention module, the twelfth ConvModule module, the third Concat module, the third TripletAttention module, and the eighth CNN module.

[0153] The Head module includes three subheadings, each of which includes the thirteenth ConvModule, the fourteenth ConvModule, the first Conv2d module, the first DwConv module, the second DwConv module, and the second Conv2d module.

[0154] The copper strip image is first input into the Backbone module, which sequentially performs feature extraction, first feature fusion, and feature stitching on the copper strip image to obtain the stitched features.

[0155] Then, after passing through the Neck module, a second feature fusion is performed. According to the preset size, the stitched features are upsampled to obtain a high-resolution feature map. The high-resolution feature map is then stitched together along the feature channel direction and fused by convolution to obtain a convolutionally fused feature map. The convolutionally fused feature map includes multi-scale fused features of semantic and spatial information.

[0156] Finally, the Head module sequentially performs two-dimensional convolution, depthwise separable convolution, and normalization on the convolutionally fused feature map to output the segmentation result of the copper strip defect.

[0157] In the Conv module, features are extracted and nonlinearity is added to help the network learn complex features, accelerate training, and stabilize the network. In the CNN module, deep features are extracted through two or more convolutional layers, and the input feature map is directly added to the output feature map through residual connections to alleviate the gradient vanishing problem. Features extracted from multiple branches are fused to adapt to features at different scales. In the dct_channel_block_2d module, the feature map is converted into the frequency domain, and after feature extraction, a weighted feature map is formed. The frequency domain information is combined to supplement the spatial domain information. In the SPPF module, pooling kernels of different sizes (such as 1x1, 3x3, 5x5) are used to pool the input feature map to extract local and global features. The pooled multi-scale features are concatenated to form a richer feature representation. In C2PSA, there is good multi-scale semantic fusion capability. It can also establish global contextual dependencies at different scales, spatial and channel dimensions, improve the model's segmentation and partitioning performance of targets in complex scenes, and is particularly suitable for fine-grained target recognition and tasks with significant background interference.

[0158] In the Upsample module, the low-resolution feature map (e.g., a feature map of size H×W×C, where H and W are height and width, and C is the number of channels) from the Backbone module is upsampled to the target size, for example, from H×W to 2H×2W, generating a high-resolution feature map. The upsampling operation can be nearest-neighbor interpolation: directly copying each pixel value to the surrounding area to enlarge the feature map size; or bilinear interpolation: calculating the weighted average of surrounding pixels to fill the new pixel; or transposed convolution: applying a learnable convolution kernel to generate a finer, higher-resolution feature map from the input feature map.

[0159] The Concat module combines two or more feature maps (typically from different branches or scales, such as high-resolution shallow features and low-resolution deep features). The feature map sizes need to be consistent in spatial dimensions. If the sizes differ, they are usually adjusted to the same size using upsampling or downsampling. The maps are then concatenated along the channel dimension, merging multiple feature maps into a larger one. For example, two feature maps F1 and F2 have C1 and C2 channels respectively. The concatenated feature map has C1+C2 channels. A convolution operation is performed on the concatenated feature map to fuse information and compress the number of channels. Activation functions (such as ReLU) and normalization (such as BatchNorm) can also be used to enhance feature representation, generating a fused feature map that contains both high-resolution spatial detail and low-resolution semantic information.

[0160] In the Conv2d module, the fused feature map is used to extract features from the input feature map and perform further processing using a two-dimensional convolution kernel. Convolutional kernel sizes are typically set to 1×1 or 3×3 for different tasks. 1×1 convolutions change the number of channels (e.g., dimensionality reduction or adjustment to the number of classes or prediction dimensions); 3×3 convolutions further enhance local feature representations, with a stride typically set to 1 to maintain the spatial resolution of the feature map. Padding (e.g., SAME) is usually used to ensure the output feature map size remains constant. Activation functions (e.g., ReLU, LeakyReLU, or Sigmoid) are applied to introduce non-linearity, enabling the network to learn complex feature relationships. In object segmentation tasks, the activation function may be adjusted depending on the task. Sigmoid is used for confidence and class probability outputs (ranging from 0 to 1); no activation is used for direct regression predictions (e.g., bounding box coordinates). Normalizing the convolutional output reduces gradient vanishing or exploding problems, accelerates network training, and improves convergence stability. The output of the Conv2d module is a multi-scale feature map, with each feature map used to predict the object segmentation result, including: the center coordinates (x, y) of the bounding box; the width and height (w, h) of the bounding box; the target confidence score; and the target class probability.

[0161] The DwConv module decomposes the standard convolution into two steps: first, it uses a 3×3 Depthwise convolution to extract spatial features independently for each input channel; then, it uses a 1×1 Pointwise convolution to perform linear combination and information fusion along the channel dimension, thereby achieving decoupled modeling of space and channels. Typically, a stride of 1 is set and SAME padding is used to keep the feature map size constant. Normalization and non-linear activation functions (such as ReLU or LeakyReLU) are applied after the convolution to enhance feature representation. The DwConv module is a lightweight convolutional structure designed to significantly reduce the number of model parameters and computational overhead while maintaining feature extraction capabilities. The DwConv module is widely used in lightweight network structures, such as MobileNet and YOLO series, for efficient inference on edge devices or in real-time tasks, with advantages of fewer parameters, higher speed, and greater adaptability.

[0162] Step 3: Use a D700 camera to take pictures of the copper sheets on the production line and input the image to be segmented into the time-frequency domain dual-branch attention network obtained through training to obtain the segmentation result image of the image to be segmented. The result includes defect mask, defect detection box and confidence score.

[0163] The copper strip image is input into a pre-built optimal training weight model, which outputs the copper strip defect segmentation results. The segmentation results include black spots, pits, oil droplets, scratches, and abrasions. The defect characteristics of black spots, pits, oil droplets, scratches, and abrasions are as follows: Figure 6 As shown.

[0164] Example 2:

[0165] This embodiment also provides a copper strip image defect segmentation system based on the spatiotemporal domain, including:

[0166] The image acquisition module is used to acquire images of copper strips on the production line using an industrial camera;

[0167] The copper strip defect segmentation module is used to input the copper strip image into a pre-built defect segmentation model with optimal training weights, and output the copper strip defect segmentation results. The segmentation results include black spots, pits, oil drops, scratches, and abrasions. The segmentation details of the copper strip black spots, pits, oil drops, scratches, and abrasions are as follows: Figure 7 As shown;

[0168] The model building module, used for building the defect segmentation model, includes:

[0169] Obtain a time-frequency domain dual-branch attention network;

[0170] The time-frequency domain dual-branch attention network includes combining physical mechanisms, adding a DCT frequency domain enhancement branch to the CNN, using frequency domain information to supplement spatial domain information, converting grayscale information into frequency domain information through discrete cosine transform, and weighting the channel information of each layer with weights obtained by average pooling to obtain a dual-branch backbone.

[0171] In the first fusion phase, a weighted fusion approach is used to weight and fuse the information of the main branches of the two branches, and to adaptively fuse the information.

[0172] In the second fusion stage, a triple attention mechanism is added to supplement the details of the high-resolution layer and provide semantics for the low-resolution layer. Attention weights are calculated by capturing the interaction between the three dimensions of channel, height and width through a three-branch structure.

[0173] Training module: Used to train the constructed time-frequency domain dual-branch attention network.

[0174] Example 3:

[0175] This embodiment also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the module of the spatiotemporal domain-based copper strip image defect segmentation method.

[0176] This embodiment introduces a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the copper strip defect segmentation method described in Embodiment 1.

[0177] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer programs can be provided.

[0179] Instructions are sent to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which are executable by the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0180] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0182] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for defect segmentation of copper strip images based on the spatiotemporal domain, characterized in that: The method includes: Step 1: Construct a time-frequency domain dual-branch attention network as a defect segmentation model: The time-frequency domain dual-branch attention network includes combining physical mechanisms, adding a DCT frequency domain enhancement branch to the CNN, using frequency domain information to supplement spatial domain information, converting grayscale information into frequency domain information through discrete cosine transform, and weighting the channel information of each layer with weights obtained by average pooling to obtain a dual-branch backbone. In the first fusion phase, a weighted fusion approach is used to weight and fuse the information of the main branches of the two branches, and to adaptively fuse the information. In the second fusion stage, a triple attention mechanism is added to supplement the details of the high-resolution layer and provide semantics for the low-resolution layer. Attention weights are calculated by capturing the interaction between the three dimensions of channel, height and width through a three-branch structure. The implementation process of the DCT frequency domain enhancement branch includes the following: The input image is subjected to a two-dimensional discrete cosine transform, which transforms the feature map of each channel from the spatial domain to the frequency domain. Frequency domain feature normalization and compression; Then, channel-dimensional average pooling is performed to obtain the frequency domain response intensity of each channel; Channel attention weights are generated through a fully connected layer or a non-linear mapping. Apply frequency domain weights to the original spatial domain feature map; The enhanced spatial domain features are fused with the original spatial features or the backbone network to form the final output features; In step one, the triple attention mechanism consists of three branches, each responsible for capturing the interaction features between the spatial dimension H or W and the number of channels C. Attention weights are generated by permuting the input tensor in each branch, followed by Z-pooling and a k1×k1 convolutional layer. These weights are generated through a sigmoid activation layer, applied to the permuted input tensor, and then transformed back to the original input shape. The calculation formula for the triple attention mechanism is expressed as follows: enter: ; CW direction: ; HC direction: ; HW direction: ; ； In the formula, It is the input feature map. It is a dimensional order transformation function. It is the output feature map after CH dimension swapping. It is a channel weighting function. This is the feature map output after weighting by the CW dimension. This is the output after HC dimension swapping. It is the output feature map after CW dimension swapping. It is the feature map output after HC dimension weighting. This is the output after WC dimension swapping. It is the output after weighting by the HW dimension. It is an average value output after weighting in three different directions; Step 2: Train the constructed time-frequency domain dual-branch attention network; Step 3: Input the image to be segmented into the time-frequency domain dual-branch attention network obtained through training, and obtain the segmentation result image of the image to be segmented. The result includes defect mask, defect detection box and confidence score.

2. The method for segmenting copper strip images based on the spatiotemporal domain according to claim 1, characterized in that: Step two includes: acquiring images of defective copper strips produced in the industry using an industrial camera; using the label-studio image segmentation dataset creation tool to create labeled images with segmentation masks from the original images; dividing the dataset into training, validation, and test sets; then using the dataset to train the network; using a cosine annealing training strategy; employing AMP mixed precision training to obtain the optimal weight parameters of the model; and finally obtaining a trained time-frequency domain dual-branch attention network. In step three, the image to be segmented is obtained by taking pictures of the copper sheets on the production line using a camera.

3. The method for segmenting copper strip images based on the spatiotemporal domain according to claim 1, characterized in that: In step one, after the DCT frequency domain enhancement branch maps and transforms the input feature map to the frequency domain, it constructs channel attention weights using the energy distribution in the frequency domain to weight the input feature map, thereby enhancing the frequency-sensitive key channel response. The formula for the DCT frequency domain enhancement branch processing is expressed as follows: Let the feature map input vector be: ; For each input channel, a two-dimensional DCT transform is performed as follows: ； ； The flattened frequency domain feature map is as follows: ; LayerNorm is standardized as follows: ； The channel vector is generated as follows: ; Channel attention generation MLP is as follows: ； The final channel weighting is: ; Where B is the training batch size, C is the number of channels, and H×W is the feature map space size. It is a 4D tensor, which is the feature map input to the module; For the vertical and horizontal indices of the frequency space in DCT, For pixel position index in the spatial domain, This represents the spatial location of the feature map of the c-th channel in the b-th sample. Pixel value at that location, For the b-th sample, the c-th channel is at the frequency position DCT coefficients, These are the DCT normalization coefficients. , These are the row normalization factor and the column normalization factor, respectively. This indicates how the two normalization factors are determined. H represents the feature map height, which is the number of rows, and W represents the feature map width, which is the number of columns. k represents two ranges, 0-H or 0-W, depending on which direction the DCT transformation is performed. For the DCT flattening vector of the b-th sample and the c-th channel, This means transforming a two-dimensional feature map into a one-dimensional vector, which is actually a one-dimensional array. Each number in the array is the value after the previous two-dimensional DCT transformation. yes Normalized vector, yes The mean, yes standard deviation It is an extremely small number. This represents the value after the two-dimensional cosine transform at position i. Because the two-dimensional cosine transform will generate C two-dimensional cosine feature maps, each of which is flattened into a one-dimensional array of length H*W, where i represents the position in the array. It is the mean spectral value of the c-th channel in the b-th sample. It is the feature map vector of all channels of the b-th sample. This represents the result of standardizing each number in the H*W one-dimensional array, where i still represents the position of the number in the array; It is the product of the normalized frequency coefficients in a two-dimensional DCT. These are the weight parameters of the first fully connected layer. These are the weight parameters of the second fully connected layer. , () is the Sigmoid activation function. It is the channel attention coefficient vector generated by the b-th sample. It is a function that reshapes a one-dimensional vector into a 3D tensor. It is It can be multiplied one channel at a time. These are the channel weight coefficients for the b-th sample. These are all the corresponding channels of the b-th sample in the original input feature map. It is the result after frequency domain weighting.

4. The method for segmenting copper strip images based on the spatiotemporal domain according to claim 1, characterized in that: In step one, the weighted fusion concept is expressed by the following formula: Concatenated input: ; 3×3 depthwise separable convolution: ； 1×1 convolution: ; Global average pooling: ; 1×1 convolution dimensionality reduction: ; Sigmoid activation ; Weight fusion: ; Relu6 activation: ; In the formula, where, These are two fused feature maps, each with C channels and a size of H×W. yes Feature map after concatenation function; It is a 3×3 depthwise separable convolution. It's a 1×1 convolution operation, using GAP global average pooling. It is the Sigmoid activation function. These are the weighting parameters, and R is the final weighted fusion output. This represents the feature map generated after depthwise separable convolution. represent This feature map is generated after passing through a 1x1 ordinary convolution. yes The feature map generated after global average pooling. yes The feature map is generated after dimensionality reduction through a 1x1 convolution.

5. The method for segmenting copper strip images based on the spatiotemporal domain according to claim 1, characterized in that: In step one, the processing steps of the defect segmentation model include: For copper strip defect images, dual-pillar feature extraction, weighted fusion, secondary pyramid fusion, attention mechanism, and finally multi-scale segmentation are performed; specifically including the following: After weighting by 2D DCT transformation, the output is a frequency feature map. The original image is convolved by CNN to obtain a spatial domain feature map. The two feature maps are concatenated to form a weighting coefficient. The two feature maps are then fused, followed by a second fusion process, which involves upsampling and downsampling fusion, combined with an attention mechanism, and finally multi-scale segmentation.

6. A copper strip image defect segmentation system based on spatiotemporal domain, characterized in that, include: The image acquisition module is used to acquire images of copper strips on the production line using an industrial camera; The copper strip defect segmentation module is used to input the copper strip image into a pre-built defect segmentation model with optimal training weights and output the copper strip defect segmentation result. The model building module, used for building the defect segmentation model, includes: Obtain a time-frequency domain dual-branch attention network; The time-frequency domain dual-branch attention network includes combining physical mechanisms, adding a DCT frequency domain enhancement branch to the CNN, using frequency domain information to supplement spatial domain information, converting grayscale information into frequency domain information through discrete cosine transform, and weighting the channel information of each layer with weights obtained by average pooling to obtain a dual-branch backbone. In the first fusion phase, a weighted fusion approach is used to weight and fuse the information of the main branches of the two branches, and to adaptively fuse the information. In the second fusion stage, a triple attention mechanism is added to supplement the details of the high-resolution layer and provide semantics for the low-resolution layer. Attention weights are calculated by capturing the interaction between the three dimensions of channel, height and width through a three-branch structure. Training module: Used to train the constructed time-frequency domain dual-branch attention network.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a module of the spatiotemporal domain-based copper strip image defect segmentation method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a spatiotemporal domain-based copper strip image defect segmentation method as described in any one of claims 1 to 5.

Images