A multi-class welding point defect classification method, system, device and medium based on Transformer and channel interaction

Abstract

The present application relates to a kind of multi-class weld defect classification method, system, equipment and medium based on the interaction of Transformer and channel, comprising: input image pre-processing, respectively to the input support image and query image division, embedding, position coding and class information embedding;Image block embedding sequence containing position and class information is transported Transformer encoder, and the embedding features of the image block of each channel containing class-related information and original are extracted;Class-related embedding vector and global visual context vector are used to adaptively adjust the embedding features of the image block of each channel;By calculating the similarity of support image and query image, judge whether they belong to the same class or defect type;The defect class of image is output through classification network;The present application effectively improves the precision of weld defect classification, especially in the scene of diverse defect types and small sample size, overcome the class confusion and small sample learning problem faced by traditional method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and deep learning, and particularly relates to a multi-class welding point defect classification method and system based on Transformer and channel interaction, a device and a medium. BACKGROUND

[0002] Welding point defect detection has important applications in modern manufacturing, especially in high-precision fields such as electronics, automobiles, and aviation. With the complexity of welding processes, welding point defects are diverse and varied, and traditional detection methods are difficult to meet the needs of high efficiency and accuracy. In particular, in defect classification, welding point defect features have strong similarity and small sample size, and deep learning models face the problems of data scarcity and class confusion.

[0003] In recent years, welding point defect detection based on deep learning has made some progress, but still faces two major challenges: one is the complexity of defect types, and local features can easily cause class confusion; the second is that the classification accuracy is limited under small sample learning conditions. To solve these problems, the Transformer model has gradually been applied to visual tasks with its strong global information modeling capability and channel interaction mechanism. The present application proposes a multi-class welding point defect classification method based on Transformer and channel interaction, which improves the accuracy of welding point defect classification through global context information modeling and fine-grained similarity measurement. SUMMARY

[0004] To overcome the shortcomings of the prior art, the present application provides a multi-class welding point defect classification method based on Transformer and channel interaction, which effectively improves the accuracy of welding point defect classification by introducing global visual context information and feature channel interaction mechanism. Especially in the scene of diverse defect types and small sample size, it overcomes the class confusion and small sample learning problems faced by traditional methods. At the same time, by constructing a fine-grained similarity measurement matrix, the model's ability to distinguish between different defect classes is further enhanced, which has strong robustness and broad application prospects.

[0005] To solve the above technical problems, the present application provides the following technical scheme: a multi-class welding point defect classification method based on Transformer and channel interaction, comprising the following steps:

[0006] S1, input image preprocessing, dividing, embedding, position coding and class information embedding of the input support image and query image respectively to generate image block embedding sequence containing position and class information ;

[0007] S2, constructing a class embedding extraction module to extract the image block embedding sequence containing position and class information The Transformer encoder extracts the embedded features containing category-related information and the original image blocks , enhance global context awareness;

[0008] S3, build feature channel interaction module, use category-related embedding vector With the global visual context vector Embed features for each channel image patch Make adaptive adjustments;

[0009] S4. Build a similarity measurement module to determine whether the support image and the query image belong to the same category or defect type by calculating the similarity between them;

[0010] S5. Classification output: Based on feature fusion and similarity scores, the defect category of the image is output through the classification network.

[0011] Furthermore, in step S1, image preprocessing specifically includes: dividing the image blocks, where the input image is divided into a plurality of small, non-overlapping image blocks, each of which contains a local area of ​​the image.

[0012] Furthermore, in step S1, an image block embedding sequence containing position and category information is generated. The specific process includes the following steps:

[0013] S11, image block embedding vector mapping, each image block is mapped to the feature space through linear projection to obtain the image block embedding vector ;

[0014] S12, then embed the vector into the image block Processing to obtain the initial image block embedding vector sequence :

[0015] ;

[0016] in, Indicates the number of features in the sequence;

[0017] S13, category information embedding, the global category information of the image is embedded through a learnable category embedding vector with global category information To represent, placed in the initial image block embedding vector sequence The starting position of the image block helps the model combine the global category information when processing the image block, and obtains the image block embedding sequence after the category embedding process. :

[0018] ;

[0019] S14, position encoding, since the Transformer architecture itself does not have spatial position information, the image block after category embedding processing is embedded into the sequence Adding a position encoding matrix , represents the position of each image block and is initialized to a set of learnable parameters to obtain an image block embedding sequence containing position and category information :

[0020] .

[0021] Furthermore, in step S2, the image blocks containing position and category information are embedded into the sequence The Transformer encoder extracts the embedded features containing category-related information and the original image blocks The specific process includes the following steps:

[0022] S21, the image block embedding sequence containing position and category information is sent to the layer normalization of the Transformer stage, which is composed of Transformer encoders are repeatedly stacked, and in the layer, In the Transformer encoder, layer normalization is used to normalize the input image patch embedding sequence;

[0023] S22, through multi-head self-attention and MLP processing, and then in the multi-head self-attention layer, use multiple different attention heads to calculate the sequence The attention output of , the attention output and the normalized output are connected together using a residual connection:

[0024] ;

[0025] in, is the output of the previous encoder, is the output of the current encoder layer after being processed by the multi-head self-attention layer;

[0026] S23, yes After layer normalization again, it is sent to the MLP layer, which contains two fully connected layers and a GELU activation function:

[0027] ;

[0028] in, Represents the output of the encoder of this layer;

[0029] S24: Aggregate and output category information to obtain the final output of the Lth layer:

[0030] ;

[0031] in, It is the final output of the Transformer encoder stack, which contains category-related information and features of each image block of the original image; It is a category-related embedding vector that represents the high-level semantic information of the entire image and accumulates information from all blocks of the image; It is the image block embedding sequence after feature extraction.

[0032] Furthermore, in step S3, the category-related embedding vector With the global visual context vector Embed features for each channel image patch Perform adaptive adjustment. The specific process includes the following steps:

[0033] S31, calculate the global visual context vector, the sequence obtained in the previous step , each image block is embedded with features Average to get the global visual context vector :

[0034] ;

[0035] S32. Generate category guide vector , embed the category-related vector With the global visual context vector Perform concatenation to form a new vector , input the concatenated vector into a two-layer MLP to generate a category guidance vector:

[0036] ;

[0037] in, represents the sigmoid activation function, and represents the weights of the two-layer MLP, and represents the bias of the two-layer MLP;

[0038] S33, adjust the image block features, and finally guide the category vector Add to each image block embedding feature , thereby guiding the features of each channel, and the final sequence is:

[0039] .

[0040] Furthermore, in step S4, the specific process includes the following steps:

[0041] S41, Mahalanobis distance calculation, the first i Image patch embedding sequence and the query image set j Image patch embedding sequence Input into the similarity measurement module and calculate the Mahalanobis distance between the two:

[0042] ;

[0043] in, Represents the similarity matrix i Rank j Elements of the column; Indicates the first i The average value of all feature embeddings in the image patch embedding sequence; Indicates the i The covariance of all feature embeddings in the image patch embedding sequence;

[0044] S42, similarity score and final output, after calculating the dense matrix S After that, it is flattened into a one-dimensional vector so that each element corresponds to the similarity of a support image and query image embedding pair. Subsequently, the flattened vector is input into the MLP and aggregated into a similarity score through multiple layers of nonlinear transformations. This score represents the overall similarity between the support image and the query image.

[0045] Further, in step S41, the first i The average value of all feature embeddings in the image patch embedding sequence and indicates the i The covariance of all feature embeddings in the image patch embedding sequence , which are calculated as follows:

[0046] ;

[0047] ;

[0048] in, Indicates the number of features in the sequence; Indicates the first Features.

[0049] Through the above technical solution, the present invention provides a multi-category solder joint defect classification method based on the interaction between Transformer and channels, which has at least the following beneficial effects:

[0050] The present invention realizes a multi-category solder joint defect classification method based on Transformer and channel interaction. First, by constructing a category embedding extraction module, a stacked Transformer structure is used to extract the embedded features containing category-related information and original image blocks, which can integrate the content and category relevance of the entire image and help the model better mine the global and local information of the image; secondly, by introducing global visual context information and feature channel interaction mechanism, the category-related embedding vector and the global visual context vector are used to embed the features of the image blocks of each channel. Adaptive adjustment is performed to effectively improve the accuracy of solder joint defect classification, especially in scenarios with diverse defect types and small sample sizes, overcoming the category confusion and small sample learning problems faced by traditional methods. At the same time, by constructing a fine-grained similarity matrix, the similarity between support image and query image pairs is quantified, making the classification of features with strong consistency more accurate, thereby further enhancing the model's ability to distinguish between different defect categories, with strong robustness and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0052] Figure 1 This is a flowchart of the main steps of the multi-category solder joint defect classification method based on Transformer and channel interaction proposed in this invention;

[0053] Figure 2 This is an algorithm flow chart of the multi-category solder joint defect classification method based on Transformer and channel interaction proposed in the present invention;

[0054] Figure 3 This is an overview diagram of the principle of the Transformer encoder proposed in this invention. DETAILED DESCRIPTION

[0055] To make the above-mentioned objectives, features, and advantages of the present invention more clearly understood, the present invention is further described below in detail with reference to the accompanying drawings and specific embodiments. This will enable a full understanding of how this application uses technical means to solve technical problems and achieve technical effects, and to implement the invention accordingly.

[0056] Those skilled in the art will appreciate that all or part of the steps in the above-mentioned embodiment methods can be accomplished by instructing the relevant hardware through a program. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] Please refer to Figure 1-Figure 3 , shows a specific implementation of this embodiment. This embodiment of the present invention implements a multi-category solder joint defect classification method based on Transformer and channel interaction. First, by constructing a category embedding extraction module, a stacked Transformer structure is used to extract embedded features containing category-related information and original image blocks. , which can integrate the content and category relevance of the entire image, helping the model to better mine the global and local information of the image; secondly, by introducing global visual context information and feature channel interaction mechanism, using category-related embedding vectors With the global visual context vector Embed features for each channel image patch Adaptive adjustment is performed to effectively improve the accuracy of solder joint defect classification, especially in scenarios with diverse defect types and small sample sizes, overcoming the category confusion and small sample learning problems faced by traditional methods. At the same time, by constructing a fine-grained similarity matrix, the similarity between support image and query image pairs is quantified, making the classification of features with strong consistency more accurate, thereby further enhancing the model's ability to distinguish between different defect categories, with strong robustness and broad application prospects.

[0058] Please refer to Figure 1 This embodiment proposes a multi-category solder joint defect classification method based on Transformer and channel interaction, which includes the following steps:

[0059] S1. Input image preprocessing: the input support image and query image are divided, embedded, position encoded and category information embedded respectively, and an image block embedding sequence containing position and category information is generated. ;

[0060] As a preferred implementation of step S1, in step S1, image preprocessing specifically includes: dividing the image blocks, the input image is divided into multiple small, non-overlapping image blocks, each image block contains a local area of ​​the image; this division method can not only improve the processing efficiency of local features, but also maintain the spatial structure information of the image, providing valuable local features for subsequent steps.

[0061] In step S1, the image block embedding sequence containing the position and category information is generated The specific process includes the following steps:

[0062] S11, image block embedding vector mapping, each image block is mapped to a feature space by linear projection to obtain an image block embedding vector This process converts the image block into a vector in a high-dimensional space, which can effectively represent the local features of the image block.

[0063] S12, then the image block embedding vector is processed to obtain an initial image block embedding vector sequence :

[0064] ;

[0065] wherein represents the number of features in the sequence;

[0066] S13, category information embedding, the global category information of the image is represented by a learnable category embedding vector with global category information placed at the starting position of the initial image block embedding vector sequence to help the model combine global category information when processing image blocks, obtaining an image block embedding sequence after category embedding processing :

[0067] ;

[0068] S14, position encoding, since the Transformer architecture itself does not have spatial position information, a position encoding matrix is added to the image block embedding sequence after category embedding processing for each image block to represent the position of each image block and is initialized as a set of learnable parameters, obtaining an image block embedding sequence containing position and category information :

[0069] ;

[0070] In this embodiment, by adding the position encoding to the embedding of each image block, the spatial position information of the image can be preserved, thereby integrating the content and category relevance of the entire image to help the model better mine the global and local information of the image. In this way, each image block embedding contains its position information to preserve spatial relationships in the self-attention mechanism.

[0071] S2, constructing a category embedding extraction module, sending the image block embedding sequence containing position and category information into the Transformer encoder The Transformer encoder extracts embedding features containing category-related information and the original individual image blocks , enhancing global context awareness

[0072] As a preferred embodiment of step S2, in step S2, the image block embedding sequence containing position and category information is sent into the layer normalization of the Transformer stage The Transformer encoder extracts embedding features containing category-related information and the original individual image blocks The specific process includes the following steps:

[0073] S21, sending the image block embedding sequence containing position and category information into the layer normalization of the Transformer stage, which is repeatedly stacked by The principle is shown in Figure 3 At the first layer, of the Transformer encoder, the input image block embedding sequence is normalized by using layer normalization

[0074] S22, processed by multi-head self-attention and MLP, and then in the multi-head self-attention layer, a plurality of different attention heads are used to calculate the attention output of the sequence The attention output and the normalized output are connected together by using a residual connection:

[0075] ;

[0076] Wherein, is the output of the previous layer encoder, is the output of the current encoder layer after processing by the multi-head self-attention layer

[0077] S23, the is sent into the MLP layer after being normalized again, and the MLP layer contains two fully connected layers and a GELU activation function

[0078] ;

[0079] Wherein, represents the output of the layer encoder

[0080] S24, category information aggregation and output, obtaining the final output of the Lth layer

[0081] ;

[0082] wherein, is the final output of the Transformer encoder stack, containing both class-related information and the features of individual image patches of the original image; is the class-related embedding vector, representing the high-level semantic information of the entire image, which accumulates information from all patches of the image; is the sequence of image patch embeddings after feature extraction.

[0083] In the present embodiment, as shown in Figure 2 , the class embedding extraction uses a stacked Transformer structure to extract embedding features containing class-related information and individual image patches of the original image The purpose of this step is to extract class-related information of the image through the Transformer encoder and effectively integrate it into the feature representation of the image patch, which can integrate the content and class-related information of the entire image, helping the model better mine global and local information of the image.

[0084] S3, constructing a feature channel interaction module, using the class-related embedding vector and the global visual context vector to adaptively adjust the image patch embedding features of each channel;

[0085] As a preferred embodiment of step S3, in step S3, the class-related embedding vector and the global visual context vector are used to adaptively adjust the image patch embedding features of each channel, and the specific process includes the following steps:

[0086] S31, calculating the global visual context vector, the sequence , obtained in the preceding step, is averaged to obtain the global visual context vector :

[0087] ;

[0088] S32, generating a class-guided vector , concatenating the class-related embedding vector and the global visual context vector to form a new vector , inputting the concatenated vector into a two-layer MLP to generate a class-guided vector:

[0089] ; ​​​

[0090] in, represents the sigmoid activation function, and represents the weights of the two-layer MLP, and represents the bias of the two-layer MLP;

[0091] S33, adjust the image block features, and finally the category guide vector Add to each image block embedding feature , thereby guiding the features of each channel, and the final sequence is:

[0092] .

[0093] In this embodiment, if Figure 2 As shown, feature channel interaction, by introducing global visual context information and feature channel interaction mechanism, uses category-related embedding vectors With the global visual context vector Embed features for each channel image patch Adaptive adjustment is performed by applying the global visual context vector on different channels of feature extraction, so that the model can more selectively enhance the visual features related to the category; thereby adjusting the features of each channel, the category guide vector It can adaptively adjust the image block embedding features according to the fine-grained features of different categories, enhance category-specific visual features, and thus improve the performance in small-sample multi-category tasks; it effectively improves the accuracy of solder joint defect classification, especially in scenarios with diverse defect types and small sample sizes, overcoming the category confusion and small-sample learning problems faced by traditional methods.

[0094] S4. Build a similarity measurement module to determine whether the support image and the query image belong to the same category or defect type by calculating the similarity between them;

[0095] As a preferred implementation of step S4, in step S4, the specific process includes the following steps:

[0096] S41, Mahalanobis distance calculation, the first i Image patch embedding sequence and the query image set j Image patch embedding sequence Input into the similarity measurement module and calculate the Mahalanobis distance between the two:

[0097] ;

[0098] in, Represents the similarity matrix i Rank j Elements of the column; Indicates the first i The average value of all feature embeddings in the image patch embedding sequence; Indicates the i The covariance of all feature embeddings in the image patch embedding sequence is calculated as follows:

[0099] ;

[0100] ;

[0101] in, Indicates the number of features in the sequence; Indicates the first Features

[0102] S42, similarity score and final output, such as Figure 2 As shown, when calculating the dense matrix S After that, it is flattened into a one-dimensional vector so that each element corresponds to the similarity of a support image and query image embedding pair. Subsequently, the flattened vector is input into the MLP and aggregated into a similarity score through multiple layers of nonlinear transformations. This score represents the overall similarity between the support image and the query image.

[0103] In this embodiment, in the solder joint defect classification task with strong inter-class similarity, the Mahalanobis distance can more accurately measure the differences between different categories. In particular, when the sample distributions between categories are relatively similar, the Mahalanobis distance can help avoid errors caused by feature irrelevance. At the same time, by constructing a fine-grained similarity matrix, the similarity between the support image and the query image pair is quantified, making the classification of features with strong consistency more accurate, thereby further enhancing the model's ability to distinguish between different defect categories, and having strong robustness and broad application prospects.

[0104] S5. Classification output: Based on feature fusion and similarity score, the obtained similarity score is used as the output of the classification task, and the defect category of the image is output through the classification network.

[0105] Specifically, this application also provides a multi-category solder joint defect classification system based on Transformer and channel interaction, including:

[0106] An image preprocessing module is used to detect the target by designing a positioning network model and a classification network model respectively, to obtain preliminary positioning and classification;

[0107] The dataset generation module is used to divide, embed, position encode and embed category information of the input support image and query image respectively, and generate an image block embedding sequence containing position and category information. ;

[0108] Category embedding extraction module, which is used to embed image blocks containing position and category information into a sequence The Transformer encoder extracts the embedded features containing category-related information and the original image blocks , enhance global context awareness;

[0109] Feature channel interaction module, which is used to use category-related embedding vectors With the global visual context vector Embed features for each channel image patch Make adaptive adjustments;

[0110] A similarity measurement module is used to determine whether the support image and the query image belong to the same category or defect type by calculating the similarity between them;

[0111] The classification output module is used to output the defect category of the image through a classification network based on feature fusion and similarity score.

[0112] Specifically, the present application also provides an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above-mentioned multi-category solder joint defect classification method based on Transformer and channel interaction.

[0113] Specifically, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned multi-category solder joint defect classification method based on Transformer and channel interaction.

[0114] A multi-class welding defect classification method based on Transformer and channel interaction is realized, aiming to improve the fine classification accuracy of welding defects and solve two main challenges in the welding defect classification task: one is the diversity of welding defect types, and the other is the consistency of welding defect features. In view of the characteristics of complex welding defect types and small sample size, a channel information interaction mechanism is adopted, a global visual context vector is introduced, and the information interaction between feature channels is strengthened, so as to effectively capture extensive context information and avoid confusion in the classification process of multi-class defect features. At the same time, a fine-grained similarity measurement matrix is constructed, the similarity between image blocks is calculated, the image features under the condition of small samples are maximally utilized, and the accuracy of classification is improved. The proposed algorithm includes three main modules: (1) a class embedding extraction module, which uses stacked Transformer blocks to capture global features of images; (2) a feature channel interaction module, which promotes joint representation of different feature channels; and (3) a similarity measurement matrix, which calculates the similarity between support images and query images through Mahalanobis distance for accurate classification. The proposed method has significant advantages in fine classification of welding defects and small sample learning scenarios.

[0115] In summary, the present application can realize multi-class welding defect classification based on Transformer and channel interaction, and effectively cope with the generalization problem under the condition of small samples.

[0116] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0117] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus, such as a computer-based system, a system including a processor or other system that can fetch and execute instructions from the instruction execution system, device or apparatus, or in conjunction with these instruction execution systems, devices or apparatus.

[0118] The above embodiments have been described in detail, and the principles and embodiments of the present application have been described by applying specific examples. The above examples are only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific embodiments and application scope will be changed, and the above description should not be understood as a limitation of the present application.

Claims

1. A multi-category solder joint defect classification method based on Transformer and channel interaction, characterized in that: The following steps are involved: S1. Input image preprocessing: segmentation, embedding, position encoding, and category information embedding of the input support image and query image, respectively, to generate an image block embedding sequence Z containing position and category information; S2. Construct a category embedding extraction module to embed the image blocks containing position and category information into the sequence Z and feed it to the Transformer encoder to extract the embedding features Z containing category-related information and the original image blocks. L , enhance global context awareness; S3, build feature channel interaction module, use category related embedding vector C class and the global visual context vector Z global Embed features for each channel image patch Perform adaptive adjustments, including: S31, calculate the global visual context vector, the sequence obtained in the previous step Each image block is embedded with features Average to get the global visual context vector Z global : S32, generate category guide vector γ L , embed the category-related vector C class and the global visual context vector Z global Perform concatenation to form a new vector [C class ,Z global ], the concatenated vector is input into a two-layer MLP to generate a category guide vector: γ L =σ(W2σ(W1[C class ;Z global ]+b1)+b2); Where σ represents the sigmoid activation function, W1 and W2 represent the weights of the two-layer MLP, and b1 and b2 represent the biases of the two-layer MLP; S33, adjust the image block features, and finally the category guide vector γ L Add to each image block embedding feature , thereby guiding the features of each channel, and the final sequence is: S4. Build a similarity measurement module to determine whether the support image and the query image belong to the same category or defect type by calculating the similarity between them; S5. Classification output: Based on feature fusion and similarity scores, the defect category of the image is output through the classification network.

2. The multi-category solder joint defect classification method based on Transformer and channel interaction according to claim 1 is characterized by: In step S1, image preprocessing specifically includes: dividing the image blocks, where the input image is divided into multiple small, non-overlapping image blocks, each of which contains a local area of ​​the image.

3. The multi-category solder joint defect classification method based on Transformer and channel interaction according to claim 1 is characterized by: In step S1, an image block embedding sequence Z containing position and category information is generated. The specific process includes the following steps: S11, image block embedding vector mapping, each image block is mapped to the feature space through linear projection to obtain the image block embedding vector z i ; S12, then embed the image block into vector z i Processing, get the initial image block embedding vector sequence Z int : WITH int =[z1,…,z i ,…,With M ]; Among them, M∈N * Indicates the number of features in the sequence; S13, category information embedding, the global category information of the image is represented by a learnable category embedding vector c with global category information, which is placed in the initial image block embedding vector sequence Z int The starting position of the image block helps the model combine the global category information when processing the image block, and obtains the image block embedding sequence Z after the category embedding process. c : WITH c =[c,z1,…,z i ,…,With M ]; S14, position encoding, since the Transformer architecture itself does not have spatial position information, the image block after category embedding processing is embedded into the sequence Z c Add the position encoding matrix P to represent the position of each image block and initialize it to a set of learnable parameters to obtain the image block embedding sequence Z containing position and category information: Z=Z c +P。 4. The multi-category solder joint defect classification method based on Transformer and channel interaction according to claim 1 is characterized by: In step S2, the image block embedding sequence Z containing position and category information is fed to the Transformer encoder to extract the embedding features Z containing category-related information and the original image blocks. L The specific process includes the following steps: S21, embed the image block containing the position and category information into the sequence Z and send it to the layer normalization of the Transformer stage, which is composed of L∈N * Transformer encoders are repeatedly stacked. In the lth layer, l = 1, 2, ..., L Transformer encoder, layer normalization is used to normalize the input image block embedding sequence; S22, through multi-head self-attention and MLP processing, then in the multi-head self-attention layer, use multiple different attention heads to calculate the attention output of sequence Z, and use residual connection to connect the attention output and normalized output together: Z l(MSHA) =LayerNorm(Z l-1 +MiltiHeadAttention(z l-1 )); Among them, Z l-1 is the output of the previous encoder, Z l(MSHA) is the output of the current encoder layer after being processed by the multi-head self-attention layer; S23, Z l(MSHA) After layer normalization again, it is sent to the MLP layer, which contains two fully connected layers and a GELU activation function: WITH l =LayerNorm(Z l(MSHA) +MLP(Z l(MSHA) )); Among them, Z l Represents the output of the encoder of this layer; S24: Aggregate and output category information to obtain the final output of the Lth layer: Among them, Z L It is the final output of the Transformer encoder stack, which contains category-related information and features of each image block of the original image; C class It is a category-related embedding vector that represents the high-level semantic information of the entire image and accumulates information from all blocks of the image; It is the image block embedding sequence after feature extraction.

5. The multi-category solder joint defect classification method based on Transformer and channel interaction according to claim 1 is characterized by: In step S4, the specific process includes the following steps: S41, Mahalanobis distance calculation, embed the i-th image block in the support image set obtained in the previous step into the sequence and the j-th image patch embedding sequence in the query image set Input into the similarity measurement module and calculate the Mahalanobis distance between the two: Among them, S ij Represents the element in row i and column j in the similarity matrix; μ i represents the average value of all feature embeddings in the embedding sequence of the u-th image block in the support image set; Σ u represents the covariance of all feature embeddings in the i-th image block embedding sequence; S42. Similarity score and final output. After calculating the dense matrix S, it is flattened into a one-dimensional vector so that each element corresponds to the similarity of a support image and query image embedding pair. Subsequently, the flattened vector is input into the MLP and aggregated into a similarity score through multiple layers of nonlinear transformations. This score represents the overall similarity between the support image and the query image.

6. The multi-category solder joint defect classification method based on Transformer and channel interaction according to claim 5 is characterized by: In step S41, μ represents the average value of all feature embeddings in the embedding sequence of the i-th image block in the support image set. i and represents the covariance Σ of all feature embeddings in the i-th image patch embedding sequence i , which are calculated as follows: Among them, M∈N * Indicates the number of features in the sequence; represents the kth feature in the sequence.

7. A multi-category solder joint defect classification system based on Transformer and channel interaction, characterized in that: include: An image preprocessing module is used to detect the target by designing a positioning network model and a classification network model respectively, to obtain preliminary positioning and classification; A data set generation module, which is used to divide, embed, position encode and embed category information of the input support image and query image respectively, and generate an image block embedding sequence Z containing position and category information; The category embedding extraction module is used to embed the image blocks containing position and category information into a sequence Z and feed it to the Transformer encoder to extract the embedding features Z containing category-related information and the original image blocks. L , enhance global context awareness; Feature channel interaction module, which is used to use the category-related embedding vector C class and the global visual context vector Z global Embed features for each channel image patch Perform adaptive adjustments, including: S31, calculate the global visual context vector, the sequence obtained in the previous step Each image block is embedded with features Average to get the global visual context vector Z global : S32, generate category guide vector γ L , embed the category-related vector C class and the global visual context vector Z global Perform concatenation to form a new vector [C class ,Z global ], the concatenated vector is input into a two-layer MLP to generate a category guide vector: γ L =σ(W2σ(W1[C class ;Z global ]+b1)+b2); Where σ represents the sigmoid activation function, W1 and W2 represent the weights of the two-layer MLP, and b1 and b2 represent the biases of the two-layer MLP; S33, adjust the image block features, and finally the category guide vector γ L Add to each image block embedding feature , thereby guiding the features of each channel, and the final sequence is: A similarity measurement module is used to determine whether the support image and the query image belong to the same category or defect type by calculating the similarity between them; The classification output module is used to output the defect category of the image through a classification network based on feature fusion and similarity score.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the multi-category solder joint defect classification method based on Transformer and channel interaction according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the multi-category solder joint defect classification method based on Transformer and channel interaction according to any one of claims 1 to 6 is implemented.

Images