Copper strip surface defect detection method based on bimodal fusion

Abstract

The invention relates to a copper strip surface defect detection method based on bimodal fusion, and belongs to the field of copper strip surface defect detection. Comprising the following steps: inputting a near-infrared and visible light bimodal copper strip picture into a pre-constructed bimodal surface defect detection model, then carrying out copper strip image defect detection, and outputting a copper strip defect type, a segmentation mask and confidence; the construction of the bimodal surface defect detection model comprises the following steps: obtaining a yov8 network model; the NGAM attention is utilized to optimize a double-backbone network of the yolov8 network model, and an optimized yolov8 double-backbone network is obtained; and a near-infrared and visible light attention fusion module NVAF is added behind the optimized yov8 double-backbone network to obtain a constructed bimodal surface defect detection model. According to the invention, the NVAF module and the NGAM attention are combined to effectively fuse bimodal information when a copper strip surface defect detection task is processed, so that the average precision of copper strip surface defect detection is improved.

Description

TECHNICAL FIELD

[0001] The application relates to a copper strip surface defect detection method based on bimodal fusion and belongs to the technical field of copper strip surface defect detection. BACKGROUND

[0002] The production process of copper strips can cause different types of surface defects such as black spots, concave and convex pits, holes, oil droplets, peeling, scratches, scratches, stains and the like on the surface of the copper strips; these surface defects not only seriously affect the corrosion resistance, wear resistance, fatigue strength and other properties of the products, but also can have a disastrous impact on the finished copper products. Therefore, accurately and quickly detecting the surface defects of copper strips and copper plates is of great significance to improving product quality. The traditional surface defect detection method mainly relies on manual visual inspection, which has many drawbacks. On the one hand, manual detection is inefficient and cannot meet the rhythm of modern large-scale production. Manual detection cannot conduct comprehensive and detailed inspection on each piece of steel, and is prone to missed detection. On the other hand, the accuracy of manual detection is greatly affected by subjective factors, and the experience, vision and fatigue degree of different detection personnel can lead to inconsistent detection results, making it difficult to ensure the stability and reliability of the detection quality. In addition, copper strips have a problem of reflection, which leads to low detection accuracy during the detection stage.

[0003] With the continuous progress of artificial intelligence and deep learning technology, the field of machine vision has also undergone tremendous changes. In recent years, a large number of deep learning algorithms (such as convolutional neural networks and recurrent neural networks) have been widely used in object detection. Compared with traditional machine vision methods, deep learning-based algorithms have shown significant advantages in accuracy, operation speed and promotion ability. In the future, with the further development of deep learning and artificial intelligence technology, machine vision technology will achieve greater innovation and breakthrough. For example, copper strip surface defects have diverse shapes, different scales and complex backgrounds, and are easily disturbed by factors such as light and noise, resulting in the need to improve the accuracy and robustness of surface defect detection. In addition, some detection algorithms have high computational complexity, which makes it difficult to meet the real-time detection requirements and limits their widespread application in industrial production sites. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the application provides a copper strip surface defect detection method based on bimodal fusion. The application improves the problem of low detection accuracy caused by reflection by suppressing environmental light interference and penetrating surface contaminants with near-infrared light. The application can effectively improve the detection accuracy of copper strip surface defects.

[0005] The technical scheme of the application is: a copper strip surface defect detection method based on bimodal fusion, the method comprising:

[0006] The near-infrared and visible light dual-mode copper strip pictures collected from the SD200 binocular camera are input into a pre-constructed dual-mode surface defect detection model, then copper strip image defect detection is performed, and the type, segmentation mask and confidence of the copper strip defect are output;

[0007] The construction of the dual-mode surface defect detection model includes:

[0008] The yolov8 network model is obtained;

[0009] The dual-main network of the yolov8 network model is optimized by using the NGAM attention, to obtain an optimized yolov8 dual-main network; the NGAM attention is obtained by introducing a residual coefficient after the GAM channel attention;

[0010] A near-infrared and visible light attention fusion module NVAF is added after the optimized yolov8 dual-main network to obtain the constructed dual-mode surface defect detection model; the NVAF fuses dual-mode features through attention mechanisms, captures shared features, dynamic pyramid convolution and cross-modal combination.

[0011] Further, the output type of the copper strip defect includes black spots, concave and convex pits, holes, oil droplets, peeling, scratches, scratches, and stains.

[0012] Further, the NGAM attention is used for feature extraction, and the residual coefficient is introduced after the GAM channel attention to balance the weights of the channel attention and the spatial attention; the NGAM attention includes the residual coefficient, the channel attention and the spatial attention;

[0013] The residual coefficient: a residual connection with a coefficient is introduced after the channel attention;

[0014] The channel attention: the feature map is dimensionally permuted and reshaped;

[0015] The spatial attention: the feature map is convolved, normalized and mapped.

[0016] Further, the NGAM attention is represented as:

[0017]

[0018] Spatial attention:

[0019]

[0020]

[0021]

[0022] Channel attention:

[0023]

[0024]

[0025]

[0026]

[0027]

[0028] wherein, represents an output of NGAM attention; , represents a weighting coefficient; represents an output of spatial attention; represents an output of channel attention; represents a calculation output of spatial attention; represents an input feature map; represents convolution; BN represents Batch Normalization; represents an activation function; represents a spatial attention output; represents a sigmoid activation function; represents dimension permutation; is a feature map obtained after dimension reshaping, i.e., reshape processing, of ; represents channel attention calculation; represents ; represents ; represents element-wise multiplication;

[0029] permute: dimension permutation operation, used to change the dimension order of a tensor, without changing the element content, only adjusting the dimension arrangement; X : input feature map;

[0030] (0, 2, 3, 1): indicates that the input feature map X is dimensionally permuted, and the original dimension order is adjusted from (0, 1, 2, 3) to (0, 2, 3, 1); the original dimension of the feature map X in deep learning is (B, C, H, W) (0: B, 1: C, 2: H, 3: W); after permutation, the dimension becomes (B, H, W, C) (0: B, 2: H, 3: W, 1: C), that is, the channel number C is placed at the end, which is convenient for subsequent processing of spatial dimensions; B is the batch size, C is the channel number, H and W are spatial height and width; reshape: dimension reshaping operation, which re-divides the dimensions of the tensor while keeping the total number of elements unchanged; : indicates a real number space with dimension C, that is, a vector with length C, and each element is a real number;

[0031] : indicates a real number space with dimension , r is the reduction ratio, which is used to reduce the computational amount of channel attention; : indicates a linear transformation, which maps a vector with dimension C to a vector with dimension C / r, which is a "compression" process;

[0032] : indicates another linear transformation, which maps a vector with dimension C / r back to a vector with dimension C, which is a "dilation" process.

[0033] Further, the processing flow of the near-infrared and visible light attention fusion module NVAF includes:

[0034] First, the input features are fused in a dual-modal manner: the visible light features and the near-infrared features are concatenated along the channel dimension, the concatenated features are convolved, and then adaptive pooling is applied to retain channel information and compress spatial dimensions;

[0035] Then, the channel attention weight is generated: the relationship between channels is learned through one-dimensional convolution, and the channel attention weight is generated by applying Sigmoid activation;

[0036] Finally, cross-modal modulation is performed: the near-infrared information is used to modulate the visible light features, and the visible light information is used to modulate the near-infrared features.

[0037] Further, the near-infrared and visible light attention fusion module NVAF, i.e., the NVAF module, is represented as:

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046] In the formula, , represents the characteristic map of visible light and near-infrared light; represents the convolution of ; represents element-wise multiplication, represents the Sigmoid activation function; represents global average pooling; , represents the visible light and near-infrared attention module output; : NVAF module visible light output feature, : NVAF module near-infrared output feature; represents a depth separable convolution; represents an activation function; represents a point convolution; represents a depth convolution, and Concat is a dimension along the channel dimension Concat is concatenated into one dimension; h, w is the height and width of the feature map, and k is the size of the convolution kernel; Attention = , Attention = , = , = .

[0047] Further, the processing steps of the dual-modality surface defect detection model include:

[0048] The copper strip image is sequentially subjected to feature extraction, near-infrared and visible light dual-modality image fusion, feature fusion, and feature splicing to obtain spliced features;

[0049] According to a predetermined size, the spliced features are subjected to an upsampling operation to obtain a high-resolution feature map; the high-resolution feature map is sequentially subjected to feature channel direction splicing and convolution fusion to obtain a convolution fused feature map; the convolution fused feature map includes multi-scale fusion features of semantic information and spatial information;

[0050] The feature map after the convolution fusion is sequentially subjected to two-dimensional convolution and normalization to obtain a surface defect detection result of the copper strip.

[0051] The application further provides a copper strip surface defect detection system based on dual-modal fusion, which comprises a module for executing the copper strip surface defect detection method based on dual-modal fusion.

[0052] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to realize the module of the copper strip surface defect detection method based on dual-modal fusion.

[0053] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to realize the copper strip surface defect detection method based on dual-modal fusion.

[0054] 1. The application combines the NGAM attention mechanism and the NVAF module with the yolov8 double-trunk network to construct a dual-modal surface defect detection model; the main principle of the NVAF module is to capture the unique features of visible light and near-infrared light through attention, and then fuse through extraction of cross-modal shared features, convolution pyramid and cross-modal interaction; the introduction of this fusion method effectively solves the problem of surface defect missed detection caused by copper strip reflection, and simultaneously realizes the synchronous detection of multiple types of surface defect targets;

[0055] 2. The application also designs an improved NGAM attention mechanism to optimize the yolov8 double-trunk network to balance the weights of channel attention and spatial attention; channel attention focuses on "what" is important, spatial attention focuses on "where" is important, and the combination of the two is more comprehensive; at the same time, residual connection and batch normalization are helpful for the training of deep network, so that the yolov8 double-trunk extraction feature is more perfect; the NVAF module and the NGAM attention mechanism can effectively solve the difficult detection and missed detection problems caused by copper strip reflection when processing the copper strip surface defect detection task. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 Fig. 1 is a flowchart of the copper strip surface defect detection method in an embodiment of the application;

[0057] Fig. 2 is a structural schematic diagram of the surface defect detection model in an embodiment of the application

[0058] Fig. 3 is a structural schematic diagram of the NGAM attention mechanism in an embodiment of the application;

[0059] FIG. 4 is a schematic diagram of the structure of channel attention in the NGAM attention mechanism in an embodiment of the present application;

[0060] FIG. 5 is a schematic diagram of the structure of spatial attention in the NGAM attention mechanism in an embodiment of the present application;

[0061] FIG. 6 is a schematic diagram of the flow process of the near-infrared and visible light attention fusion module NVAF in an embodiment of the present application;

[0062] FIG. 7 is a schematic diagram of the surface defect detection result of the copper strip in another embodiment of the present application. DETAILED DESCRIPTION

[0063] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solutions of the present application, and are not limitations of the technical solutions of the present application. In the case of no conflict, the technical features in the embodiments of the present application and the embodiments can be combined with each other.

[0064] The term "and / or" is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " generally represents that the front and rear associated objects are in an "or" relationship.

[0065] Embodiment 1: As shown in the figure, a copper strip surface defect detection method based on dual-modal fusion, the method comprises: Figure 1

[0066] The near-infrared and visible light dual-modal copper strip pictures collected from the SD200 binocular camera are input into the pre-constructed dual-modal surface defect detection model, then the copper strip image defect detection is performed, and the type, segmentation mask and confidence of the copper strip defect are output;

[0067] The construction of the dual-modal surface defect detection model comprises:

[0068] The yolov8 network model is obtained;

[0069] The NGAM attention is used to optimize the double-trunk network of the yolov8 network model, and the optimized yolov8 double-trunk network is obtained; the residual coefficient is introduced after the GAM channel attention to obtain the NGAM attention;

[0070] ​After optimizing the yolov8 double stem network, a near-infrared and visible light attention fusion module NVAF is added to obtain a built double-mode surface defect detection model; the NVAF fuses double-mode features through attention mechanism, captures shared features, dynamic pyramid convolution and cross-modal combination.

[0071] Further, the training of the double-mode surface defect detection model comprises:

[0072] The near-infrared and visible light double-mode copper strip pictures collected from the SD200 binocular camera are input into the pre-constructed double-mode surface defect detection model to obtain the trained double-mode surface defect detection model.

[0073] Further, the output copper strip defect types include black spots, concave and convex pits, holes, oil droplets, peeling, scratches, scratches, and stains. The surface defect conditions of the black spots, concave and convex pits, holes, oil droplets, peeling, scratches, scratches, and stains are as shown in Figure 7 Figure 7 is the detection result of the method, and from the figure it can be seen that the copper strip defect types, segmentation mask and confidence of the output defect of the application.

[0074] Further, the NGAM attention is used for feature extraction, and a residual coefficient is introduced after the GAM channel attention, which is used to balance the weights of channel attention and spatial attention. Channel attention pays attention to "what" is important, spatial attention pays attention to "where" is important, and the combination of the two is more comprehensive. Residual connection and batch normalization help the training of deep network, so that the yolov8 double stem extracts more perfect features; the NGAM attention includes a residual coefficient, channel attention and spatial attention; the NGAM attention is as shown in Figure 3 , the channel attention is as shown in Figure 4 , and the spatial attention is as shown in Figure 5 .

[0075] The residual coefficient: a residual connection with a coefficient is introduced after the channel attention;

[0076] The channel attention: the feature map is dimensionally permuted and reshaped;

[0077] The spatial attention: the feature map is convolved, normalized and mapped.

[0078] The original NGAM attention is sequentially subjected to channel attention and spatial attention to obtain corresponding weights, which may cause the original features to be excessively enhanced or inhibited.

[0079] ​The NGAM attention introduces residual coefficients, allowing the network to adaptively balance new features and original features, rather than forcing a 1:1 fusion, which can prevent the attention module from excessively suppressing or enhancing original features, is more stable in deep networks, and reduces the risk of gradient vanishing or explosion.

[0080] Further, the NGAM attention is expressed as:

[0081]

[0082] Spatial attention:

[0083]

[0084]

[0085]

[0086] Channel attention:

[0087]

[0088]

[0089]

[0090]

[0091]

[0092] wherein, represents the output of the NGAM attention; , represents the weighting coefficient; represents the output of the spatial attention; represents the output of the channel attention; represents the calculation output of the spatial attention; represents the input feature map; represents convolution of; BN represents Batch Normalization; represents the activation function; represents the spatial attention output; represents the sigmoid activation function; represents the dimension permutation; is the feature map obtained after dimension reshaping, i.e., reshape, of ; represents channel attention calculation; represents ; represents ; denotes element-wise multiplication;

[0093] permute: dimension permutation operation, used to change the dimension order of the tensor, without changing the element content, only adjusting the dimension arrangement; X : input feature map;

[0094] (0,2,3,1): indicates that the dimension permutation is performed on the input feature map X, and the original dimension order is adjusted from (0,1,2,3) to (0,2,3,1); the original dimension of the feature map X in deep learning is (B, C, H, W) (0:B, 1:C, 2:H, 3:W); after permutation, the dimension becomes (B, H, W, C) (0:B, 2:H, 3:W, 1:C), that is, the channel number C is placed at the end, which is convenient for subsequent processing of spatial dimensions; B is the batch size, C is the channel number, H and W are spatial height and width; reshape: dimension reshaping operation, which redivides the dimension of the tensor while keeping the total number of elements unchanged; : represents a real number space with dimension C, that is, a vector with length C, and each element is a real number;

[0095] : represents a real number space with dimension r is the reduction ratio, used to reduce the computational amount of channel attention; : represents a linear transformation, which maps a vector with dimension C to a vector with dimension C / r, which is a "compression" process;

[0096] : represents another linear transformation, which maps a vector with dimension C / r back to a vector with dimension C, which is a "dilation" process.

[0097] The optimized yolov8 double-main network is added with the proposed near-infrared and visible light attention fusion module NVAF to obtain a constructed double-modality surface defect detection model; the NVAF fuses double-modality features through attention mechanism, captures shared features, dynamic pyramid convolution, and cross-modality combination; the processing flow of the NVAF is as shown in Figure 6 .

[0098] Further, the processing flow of the near-infrared and visible light attention fusion module NVAF includes:

[0099] First, the input features are fused in double-modality features: the visible light features and the near-infrared features are spliced along the channel dimension, the spliced features are subjected to convolution operation, and then adaptive pooling is applied to retain channel information and compress spatial dimension;

[0100] Then the channel attention weight generation is performed: the inter-channel relationship is learned through one-dimensional convolution, and the channel attention weight is generated by applying Sigmoid activation;

[0101] Finally, cross-modal modulation is performed: the near-infrared information is used to modulate the visible light feature, and the visible light information is used to modulate the near-infrared feature.

[0102] Further, the near-infrared and visible light attention fusion module NVAF, that is, the NVAF module, is represented as:

[0103]

[0104]

[0105]

[0106]

[0107]

[0108]

[0109]

[0110]

[0111] In the formula, , represents the feature map of visible light and near-infrared light; represents the convolution of ; represents element-wise multiplication, represents the Sigmoid activation function; represents global average pooling; , represents the output of the visible light and near-infrared attention module; : the visible light output feature of the NVAF module, : the near-infrared output feature of the NVAF module; represents a depth separable convolution; represents an activation function; represents a point convolution; represents a depth convolution, and Concat is a dimension along the channel dimension concatenated into one dimension; h, w are the height and width of the feature map, and k is the size of the convolution kernel; Attention = , Attention = , = , = .

[0112] The module can realize attention to single modal enhancement, extract cross-modal feature sharing features, and realize cross-modal modulation when fusing near-infrared and visible light images, so that visible light and near-infrared interact with each other, complement each other, and the feature fusion effect is better. It is more suitable for solving the problem of missed detection caused by reflection of copper strip under visible light. Further, the processing steps of the double modal surface defect detection model include:

[0113] The copper strip image is sequentially subjected to feature extraction, near-infrared and visible light double modal image fusion, feature fusion, and feature splicing to obtain spliced features;

[0114] According to the preset size, the spliced features are subjected to an upsampling operation to obtain a high-resolution feature map; the high-resolution feature map is sequentially subjected to feature channel direction splicing and convolution fusion to obtain a convolution fused feature map; the convolution fused feature map includes multi-scale fusion features of semantic information and spatial information;

[0115] The convolution fused feature map is sequentially subjected to two-dimensional convolution and normalization to obtain a surface defect detection result of the copper strip.

[0116] The embodiment also provides a copper strip surface defect detection system based on double modal fusion, which includes a module for executing the copper strip surface defect detection method based on double modal fusion.

[0117] The embodiment also provides an electronic device including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the module of the copper strip surface defect detection method based on double modal fusion.

[0118] The double modal surface defect detection model includes a yolov8 double stem network, a feature fusion module, and a segmentation head; the structure of the surface defect detection model is as shown in Figure 2 ;

[0119] The yolov8 double stem network includes two CBS modules, a C2f module, a CBS module, a C2f module, a CBS module, a C2f module, a CBS module, a C2f module, an NGAM attention mechanism module, and an SPPF module connected in sequence;

[0120] The feature fusion module includes a near-infrared and visible light attention fusion module NVAF;

[0121] The feature fusion module comprises an Unsample module, a Concat module, a C2f module, an Unsample module, a Concat module, a C2f module, a CBS module, a Concat module, a C2f module, a CBS module, a Concat module, and a C2f module.

[0122] The segmentation head has three general segment modules; each segment has three branches for target classification, target positioning, and mask parameters, and each branch is subjected to convolution and normalization to obtain defects finally;

[0123] The copper strip visible light and near-infrared image is first input into a double-main network with an improved GAM attention mechanism, i.e., NGAM, to sequentially perform feature extraction, feature fusion, and feature concatenation on the image to obtain a concatenated feature; the NGAM attention does not change the input and output feature maps.

[0124] The image is fused through a near-infrared and visible light attention fusion module NVAF; the NVAF accepts input from two feature maps with the same height, width, and channel number and outputs a single fused feature map.

[0125] Then, the concatenated feature is subjected to an upsampling operation according to a preset size through a feature fusion module to obtain a high-resolution feature map; the high-resolution feature map is sequentially subjected to feature channel direction concatenation and convolution fusion to obtain a convolution fused feature map; the convolution fused feature map comprises multi-scale fused features of semantic information and spatial information.

[0126] Finally, the multi-scale features are subjected to segmentation and detection through a segment head to output the type, segmentation mask, and confidence of the copper strip defect.

[0127] In the CBS module, features are extracted, nonlinearity is increased, the network is helped to learn complex features, and the network is accelerated and stabilized; the CBS module is composed of convolution, normalization, and an activation function. The input feature map is 640x640x3, and the output feature map is reduced by a factor of two according to the number of CBS. In the C2f module, deep features are extracted through two or more convolution layers, the input feature map is directly added to the output feature through residual connection to relieve the gradient vanishing problem, the features extracted by multiple different branches are fused to adapt to features of different scales, and different size pooling kernels (such as 1x1, 3x3, and 5x5) are used in the SPPF module to pool the input feature map to extract local and global features, and the multi-scale features after pooling are concatenated together to form richer feature representations. The input and output feature maps of C2f and SPPF are unchanged.

[0128] Upsample module: The size of low-resolution feature maps (e.g., a feature map with size H x W x C, where H and W are height and width, and C is the number of channels) from the backbone network is enlarged to the target size using an up-sampling operation, such as from H x W to 2H x 2W, to generate high-resolution feature maps. The up-sampling operation can be nearest-neighbor interpolation, which directly copies each pixel value to the surrounding area to enlarge the feature map size; or bilinear interpolation, which calculates the weighted average of surrounding pixels to fill in new pixels; or transposed convolution, which applies a learnable convolution kernel to generate a finer high-resolution feature map from the input feature map.

[0129] Concat module: Two or more feature maps (usually from different branches or different scales, such as high-resolution shallow features and low-resolution deep features) are concatenated along the channel dimension to merge multiple feature maps into a larger feature map. For example, two feature maps F1 and F2 have C1 and C2 channels, respectively, and the concatenated feature map has C1 + C2 channels. Convolution, activation function (e.g., ReLU), and normalization (e.g., BatchNorm) can be used to enhance feature expression and generate fused feature maps that contain both high-resolution spatial detail information and low-resolution semantic information.

[0130] Conv2d module: The fused feature map is processed using a two-dimensional convolution kernel for feature extraction and further processing. The convolution kernel size is usually set to 1 x 1 or 3 x 3 for different tasks, 1 x 1 convolution: changes the number of channels (e.g., dimension reduction or adjustment to the number of classes or prediction dimensions); 3 x 3 convolution: further enhances local feature representation, with a stride usually set to 1 to maintain the spatial resolution of the feature map; padding (e.g., SAME) is usually used to ensure that the output feature map size remains unchanged; an activation function (e.g., ReLU, Leaky ReLU, or Sigmoid) is applied to introduce nonlinearity, allowing the network to learn complex feature relationships; in segmentation and detection tasks, the activation function may be adjusted depending on the task, Sigmoid: for confidence and class probability output (range 0 to 1); no activation: for direct regression prediction (e.g., bounding box coordinates); normalization of the convolution output reduces the problem of gradient vanishing or gradient explosion, speeds up network training, and improves convergence stability.

[0131] Example 2:

[0132] The embodiment introduces a non-transitory computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement steps of the surface defect detection method of the copper strip in the embodiment 1.

[0133] Those skilled in the art will understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer usable program code.

[0134] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in the flowcharts and / or block diagrams.

[0135] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in the flowcharts and / or block diagrams.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in the flowcharts and / or block diagrams.

[0137] The specific embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the purpose of the present application.

Claims

1. A method for detecting surface defects of a copper strip based on bimodal fusion, characterized in that: The method comprises: inputting the near-infrared and visible light dual-mode copper strip pictures collected from the SD200 binocular camera into a pre-constructed dual-mode surface defect detection model, then performing copper strip image defect detection, and outputting the type, segmentation mask and confidence of the copper strip defect; The construction of the dual-mode surface defect detection model comprises: obtaining a yolov8 network model; optimizing the dual-main network of the yolov8 network model by using NGAM attention to obtain an optimized yolov8 dual-main network; the NGAM attention is obtained by introducing a residual coefficient after the GAM channel attention; adding a near-infrared and visible light attention fusion module NVAF to the optimized yolov8 dual-main network to obtain the constructed dual-mode surface defect detection model; the NVAF fuses dual-mode features through attention mechanism, captures shared features, dynamic pyramid convolution and cross-modal combination.

2. The copper strip surface defect detection method based on bimodal fusion according to claim 1, characterized in that: The output type of the copper strip defect includes black spots, concave and convex pits, holes, oil droplets, peeling, scratches, scratches and stains.

3. The copper strip surface defect detection method based on bimodal fusion according to claim 1, characterized in that: The NGAM attention is used for feature extraction, and the residual coefficient is introduced after the GAM channel attention to balance the weights of the channel attention and the spatial attention; The NGAM attention includes a residual coefficient, channel attention and spatial attention; The residual coefficient: introducing a residual connection with a coefficient after the channel attention; The channel attention: dimension permutation and reshaping of the feature map; The spatial attention: convolution, normalization and mapping of the feature map.

4. The copper strip surface defect detection method based on bimodal fusion of claim 3, characterized in that: The NGAM attention is expressed as: ； Spatial attention: ； ； ； Channel attention: ； ； ； ； ； In the formula, represents the output of NGAM attention; , represents a weighting coefficient; represents the output of spatial attention; represents the output of channel attention; represents the calculation output of spatial attention; represents an input feature map; represents convolution; BN represents Batch Normalization; represents an activation function; represents the spatial attention output; represents a sigmoid activation function; represents dimension permutation; is a feature map obtained after dimension reshaping, i.e., reshape processing, on ; represents channel attention calculation; represents ; represents ; represents element-wise multiplication; permute: dimension permutation operation, used to change the dimension order of the tensor, without changing the element content, only adjusting the dimension arrangement; X : input feature map; (0, 2, 3, 1): indicates that the dimension of the input feature map X is transposed, and the original dimension order is adjusted from (0, 1, 2, 3) to (0, 2, 3, 1); the original dimension of the feature map X in deep learning is (B, C, H, W) (0: B, 1: C, 2: H, 3: W); after transposition, the dimension becomes (B, H, W, C) (0: B, 2: H, 3: W, 1: C), that is, the channel number C is placed at the end, which is convenient for subsequent processing of spatial dimensions; B is the batch size, C is the channel number, H and W are the spatial height and width; reshape: dimension reshaping operation, which redivides the dimensions of the tensor while keeping the total number of elements unchanged; : indicates that the dimension is a real number space of C, that is, a vector of length C, and each element is a real number; : denotes a real space of dimension r is a reduction ratio, used to reduce the computational load of channel attention; : denotes a linear transformation that maps a vector of dimension C to a vector of dimension C / r, which is a "compression" process; : denotes another linear transformation that maps vectors of dimension C / r back to vectors of dimension C, is a "spreading" process.

5. The copper strip surface defect detection method based on bimodal fusion according to claim 1, characterized in that: The processing procedure of the near-infrared and visible light attention fusion module NVAF comprises: First, dual-mode feature fusion is performed on the input features: the visible light features and the near-infrared features are spliced along the channel dimension, the spliced features are subjected to convolution operation, and then adaptive pooling is applied to retain channel information and compress spatial dimension; Then, the channel attention weight generation is performed: the one-dimensional convolution channel is used to learn the relationship between channels, and the Sigmoid activation is applied to generate the channel attention weight; Finally, cross-modal modulation is performed: the near-infrared information is used to modulate the visible light features, and the visible light information is used to modulate the near-infrared features.

6. The copper strip surface defect detection method based on bimodal fusion according to claim 1, characterized in that: The near-infrared and visible light attention fusion module NVAF, i.e. the NVAF module, is expressed as: ； ； ； ； ； ； ； ； wherein, , represents the characteristic map of visible light and near-infrared light; represents the convolution of ; represents the element-wise multiplication, represents the Sigmoid activation function; represents the global average pooling; , represents the visible light and near-infrared attention module output; : NVAF module visible light output feature, : NVAF module near-infrared output feature; represents the depth separable convolution; represents the activation function; represents the point convolution; represents the depth convolution, and Concat is concatenating into one dimension along the channel dimension; h, w are the height and width of the feature map, and k is the size of the convolution kernel; , , .​​​​​ 7. The copper strip surface defect detection method based on bimodal fusion according to claim 1, characterized in that: The processing steps of the dual-mode surface defect detection model comprise: The copper strip image is sequentially subjected to feature extraction, near-infrared and visible light dual-mode image fusion, feature fusion and feature splicing to obtain the spliced features; The spliced features are subjected to upsampling operation according to the preset size to obtain a high-resolution feature map; the high-resolution feature map is sequentially subjected to feature channel direction splicing and convolution fusion to obtain a convolution fused feature map; the convolution fused feature map comprises multi-scale fusion features of semantic information and spatial information; The convolution fused feature map is sequentially subjected to two-dimensional convolution and normalization to obtain the surface defect detection result of the copper strip.

8. A dual-mode fusion-based copper strip surface defect detection system, characterized in that, The system comprises a module for performing a dual-mode fusion based copper strip surface defect detection method according to any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the modules of the double-mode fusion-based copper strip surface defect detection method according to any one of claims 1 to 7 when executing the program. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the double-mode fusion-based copper strip surface defect detection method according to any one of claims 1 to 7 when executed by the processor.

Images