Solder joint appearance image recognition method and system based on aggregation-calibration CNN

Abstract

The present invention discloses a method and system for identifying weld appearance images based on an aggregation-calibration CNN. First, the weld image is input into an aggregation-calibration module ACM for processing. The weld features of different scales are extracted by a multi-scale module MAB in the aggregation-calibration module. The extracted weld features are calibrated by an attention module CSAB in the aggregation-calibration module. The weld features are subjected to feature fusion processing to obtain weld appearance images. The processed weld outer tube images are classified. The method provided by the present invention can effectively distinguish the differences among similar welds and greatly enrich weld features through a multi-scale convolution aggregation module. The spatial information and channel information of the weld feature map can be calibrated through an attention module, so that the model pays more attention to effective weld features and ignores useless information. The accuracy of the classification of weld appearance images has been proven to be as high as 95.2%, which is very suitable for body-in-white production lines.

Description

Technical Field

[0001] The invention relates to the field of welding technology, in particular to a welding spot appearance image recognition method based on aggregation-calibration CNN. Background Art

[0002] With the continuous development of the automotive industry, the demand for intelligence and automation is increasing. As the most important welding method in the white body welding process of automobiles, resistance spot welding currently still relies on manual inspection of its appearance quality to eliminate unqualified products, resulting in low inspection efficiency and high error rate. If the appearance quality of the weld can be identified in real time, the corresponding process parameters can be controlled, and then the process parameters can be adjusted to control the appearance of the weld to obtain a better appearance; or after discovering the appearance quality problem of the weld, timely processing can be carried out to prevent unqualified products from entering the market.

[0003] With the rapid development of machine vision technology, the fast, accurate, and intelligent features of computer vision technology have gradually been widely used in various fields of modern industry. Especially for the automotive manufacturing industry, manufacturing quality inspection based on machine vision is one of the key technologies to improve the performance of production line manufacturing. Deep learning neural networks have a large number of structured and trainable parameters. As long as there is enough data to train these parameters, the neural network can learn the laws of things and make predictions and judgments on new samples. Convolutional neural network (CNN) is a deep feedforward neural network with local connections and weight sharing. Compared with fully connected feedforward neural networks, convolutional neural networks have fewer parameters, can extract local invariant features, and are often used to learn images.

[0004] At present, a typical convolutional neural network is generally a feedforward neural network composed of a cross-stacked convolutional layer and a pooling layer, followed by a fully connected layer. Generally, the convolutional layer and the pooling layer are regarded as the feature extraction part, and the fully connected layer is regarded as the prediction part. Convolutional neural networks have three structural characteristics: local connection, weight sharing, and pooling. These characteristics make convolutional neural networks invariant to a certain extent in translation, scaling, and rotation. Convolutional neural networks are mainly used in various tasks of image and video analysis, including image classification, face recognition, image segmentation, etc., and their accuracy is generally far beyond other neural network models. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a solder joint appearance image recognition method based on aggregation-calibration CNN, which uses an attention multi-scale convolutional neural network optimized by a feature fusion strategy to realize solder joint appearance recognition.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] The solder joint appearance image recognition method based on the aggregation-calibration CNN provided by the present invention comprises the following steps:

[0008] Get solder joint images;

[0009] Construct an aggregation-calibration module ACM, and input the solder joint image into the aggregation-calibration module for processing. The multi-scale module MAB in the aggregation-calibration module extracts solder joint features of different scales. The attention module CSAB in the aggregation-calibration module calibrates the extracted solder joint features.

[0010] Perform feature fusion processing on the solder joint features to obtain the solder joint appearance image;

[0011] Classify the processed solder joint appearance images.

[0012] Furthermore, the multi-scale module MAB uses multiple convolution operations to extract multiple different features at the same position of the feature map, and after extracting the features of the solder joint image through each convolution operation, each group of features is combined to obtain the solder joint features.

[0013] Furthermore, the multi-scale module MAB is provided with four parallel processing branches, which are as follows:

[0014] The first branch uses dimensionality reduction convolution to process the input feature map;

[0015] The second branch uses dilated convolution for image processing;

[0016] The third branch uses square convolution and two rectangular convolutions to extract features and perform feature fusion processing;

[0017] The fourth branch decomposes the convolution kernel into a pair of asymmetric convolution kernels, processes the image separately and outputs the feature map;

[0018] The four parallel branches perform feature aggregation in the channel direction respectively to form a feature map of different features extracted by multiple convolution kernels at the same position.

[0019] Furthermore, the attention module CSAB is provided with a channel attention mechanism for channel information calibration. The channel attention uses global average pooling GAP and global maximum pooling GMP to compress the spatial dimension of the input feature map; uses a multi-layer perceptron MLP to realize cross-channel interaction, and uses a Sigmoid function to generate channel attention weights; and calibrates the features.

[0020] Furthermore, the attention module CSAB also includes calibrating the spatial information of the clustered features; the spatial information calibration uses global average pooling GAP and global maximum pooling GMP to compress the spatial dimension of the input feature map; a convolutional layer is used to achieve cross-space interaction, and a Sigmoid function is used to generate spatial attention weights.

[0021] Furthermore, the aggregation-calibration modules ACM are connected by a convolutional layer with a preset step value, and feature fusion is achieved through jump connections between the aggregation-calibration modules ACM.

[0022] Furthermore, it also includes a prediction part, which includes a first part, namely the global average pooling GAP, and a second part, namely the bottleneck structure and the convolution layer; the feature map is output after averaging the global information through the global average pooling GAP; at the same time, the feature map is output through the convolution layer after passing through the bottleneck structure; the output results of the first part and the second part are fused and then sent to the classifier.

[0023] The solder joint appearance image recognition system based on the aggregation-calibration CNN provided by the present invention comprises:

[0024] An image input unit, used for acquiring and inputting a welding spot image;

[0025] Aggregation-calibration module ACM, used to receive solder joint images and perform solder joint feature extraction and calibration processing at different scales on the solder joint images; and realize feature fusion through skip connection;

[0026] In the prediction part, the feature maps are fused and output in parallel through global average pooling GAP, bottleneck structure and convolution layer.

[0027] Further, the aggregation-calibration module ACM includes a multi-scale module MAB and an attention module CSAB; the multi-scale module MAB extracts solder joint features of different scales; the attention module CSAB calibrates the extracted solder joint features;

[0028] The multi-scale module MAB uses multiple convolution operations to extract multiple different features at the same position of the feature map. After extracting the features of the solder joint image through each convolution operation, each group of features is combined to obtain the solder joint features.

[0029] The attention module CSAB is provided with a channel attention mechanism to calibrate channel information. The channel attention uses global average pooling GAP and global maximum pooling GMP to compress the spatial dimension of the input feature map; a multi-layer perceptron MLP is used to realize cross-channel interaction, and a sigmoid function is used to generate channel attention weights; and features are calibrated;

[0030] The attention module CSAB also includes calibrating the spatial information of the clustered features; the spatial information calibration uses global average pooling GAP and global maximum pooling GMP to compress the spatial dimension of the input feature map; a convolutional layer is used to achieve cross-space interaction, and a Sigmoid function is used to generate spatial attention weights.

[0031] Furthermore, the multi-scale module MAB is provided with four parallel processing branches, which are as follows:

[0032] The first branch uses dimensionality reduction convolution to process the input feature map;

[0033] The second branch uses dilated convolution for image processing;

[0034] The third branch uses square convolution and two rectangular convolutions to extract features and perform feature fusion processing;

[0035] The fourth branch decomposes the convolution kernel into a pair of asymmetric convolution kernels, processes the image separately and outputs the feature map;

[0036] The four parallel branches perform feature aggregation in the channel direction respectively to form a feature map of different features extracted by multiple convolution kernels at the same position.

[0037] The beneficial effects of the present invention are:

[0038] The present invention provides a method and system for solder joint appearance image recognition based on aggregation-calibration CNN. The method uses an attention multi-scale convolutional neural network optimized by a feature fusion strategy for solder joint appearance recognition. The method uses a multi-scale convolutional aggregation module to effectively distinguish the differences between similar solder joints and greatly enrich the solder joint features. At the same time, a dual-purpose attention module is used, which can calibrate the spatial information and channel information of the solder joint feature map, so that the model pays more attention to effective solder joint features and ignores useless information. The main body of the convolutional neural network model for solder joint appearance recognition is obtained by stacking this multi-scale structure with an attention mechanism, and finally the depooling strategy and feature fusion strategy are used to further improve the computational efficiency and performance.

[0039] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0041] Figure 1 Schematic diagram of multi-scale feature aggregation.

[0042] Figure 2 Schematic diagram of the enhancement of the convolution kernel.

[0043] Figure 3 Schematic diagram of the decomposition of the convolution kernel.

[0044] Figure 4 This is a schematic diagram of channel information calibration.

[0045] Figure 5 This is a schematic diagram of spatial information calibration.

[0046] Figure 6 Schematic diagram of feature fusion strategy in convolutional network.

[0047] Figure 7 Schematic diagram of the feature fusion strategy for the prediction part.

[0048] Figure 8 Schematic diagram of solder joint appearance recognition based on AcmNet.

[0049] Fig. 9 This is a schematic diagram of the body in white.

[0050] Fig.10 This is a schematic diagram of the solder joint appearance categories.

[0051] Fig.11 Schematic diagram of the feature graphs of the three models. DETAILED DESCRIPTION

[0052] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0053] Example 1

[0054] The abbreviations involved in this embodiment have the following meanings: global average pooling (GAP); global max pool (GMP); multiscale-assembly block (MAB); channel-spatial attention block (CSAB); and assembly-calibration module (ACM).

[0055] The feature fusion strategy optimized attention multi-scale convolutional neural network provided in this embodiment is used for the solder joint appearance recognition method. The method adopts a modular design method to construct AcmNet, which is used for solder joint appearance recognition and classification. Since the processing processes of resistance spot welding RSW are similar, the differences in solder joint appearance are also very small. In order to distinguish these small differences, a multi-scale module MAB is first designed to achieve the aggregation of a large number of features. Sufficiently rich features enable the model to learn the differences between similar solder joints. However, the multi-scale module MAB will also aggregate useless information such as background. In order to suppress these useless features, an attention module CSAB is designed to enable the model to pay more attention to solder joint features. Combining the multi-scale module MAB with the attention module CSAB can form a complete solder joint feature aggregation and calibration module, that is, the aggregation-calibration module ACM. The aggregation-calibration module ACM plays a major role in solder joint recognition and is the main structure of AcmNet. However, an efficient module also needs to be used effectively to achieve optimal performance, so the de-pooling strategy and feature fusion strategy are used to further improve the computational efficiency and performance of the model.

[0056] The main idea of ​​the aggregation-calibration module ACM provided in this embodiment is to enable the model to learn a more effective learning method by itself. Convolution operations have developed various variants, and these different convolution methods have their own advantages and disadvantages. For different classification tasks, it is often necessary to manually select the best convolution method to obtain better results. This is a heavy task, and this work is handed over to CNN itself. By combining the multi-scale convolution structure and the attention mechanism, the aggregation-calibration module ACM is proposed as the main structure of AcmNet. It is divided into two parts: feature aggregation and feature calibration. First, a variety of convolution operations are used in the multi-scale module MAB to extract rich solder joint image features, and then the attention mechanism is used in the attention module CSAB to calibrate the features, so that the network can capture the most effective solder joint features in various convolution results.

[0057] like Figure 1 As shown, Figure 1The figure is a schematic diagram of multi-scale feature aggregation. The similarity of different types of solder joint appearance images is very high. However, a certain convolution kernel only focuses on a specific feature, which will lead to insufficient extraction of features and the neglect of some key features, making it difficult to distinguish the differences in similarities between different solder joint appearance images. The multi-scale module MAB introduces multiple convolution operations to extract multiple different features at the same position of the feature map. The multi-scale module MAB includes four branches, each of which extracts features on the input feature map, and finally merges the four sets of feature maps to enrich the solder joint features. Since expanding the width of the network will lead to an increase in the amount of calculation, the four branches all use 1×1 convolution for dimensionality reduction, and the proportion of each branch in the module will be adjusted according to the characteristics of the depth of the network to form the best combination, thereby maximizing the feature extraction capabilities of various convolution operations.

[0058] The first branch has only one 1×1 convolution to reduce the dimension of the input feature map, retaining the input features as much as possible. The proportion of this branch is relatively small in the entire network.

[0059] The second branch uses 3×3 dilated convolution. The advantage of dilated convolution is that it increases the receptive field without introducing additional parameters. Generally, a larger receptive field is required in the shallow layer of the network to retain more complete local information. Therefore, the second branch accounts for a larger proportion in the shallow layer of AcmNet.

[0060] like Figure 2 As shown, Figure 2 The diagram shows the enhancement of the convolution kernel. The third branch is a feature fusion branch, which fuses the features extracted by the ordinary square convolution and two rectangular convolutions. Since the convolution operation is additive, this operation can be understood as the enhancement of the square convolution by the rectangular convolution. The fusion of the three convolutions is equivalent to an enhanced ordinary square convolution, and its central part is enhanced to the maximum extent. This branch has the strongest feature extraction capability and occupies a large proportion in the entire network.

[0061] like Figure 3 As shown, Figure 3 In order to decompose the convolution kernel, the convolution kernel is decomposed in the fourth branch, and a 3×3 ordinary convolution is decomposed into a 1×3 and a 3×1 asymmetric convolution kernel. The size of the feature map output by the two is the same, but the features are different. Asymmetric convolution can reduce one-third of the parameters and perform one more convolution operation, but the receptive field of both convolutions is only one-third of that of ordinary convolution, so asymmetric convolution with a small receptive field is more suitable for extracting highly abstract features. The closer to the output layer of the network, the more abstract the features of the solder joint image. Therefore, in the deep structure of AcmNet, the proportion of the fourth branch will increase, while the proportion of the second branch will decrease.

[0062] The four branches perform feature aggregation in the channel direction. Different features are extracted by multiple convolution kernels at the same position of the feature map, which greatly enriches the solder joint features. Since large convolution kernels have been gradually replaced by small convolution kernels, the basic convolution used by each branch is 3×3 in size. The multi-scale module MAB flexibly adjusts the proportion of each branch at different positions of the network according to the characteristics of different convolution operations, integrating the advantages of various convolution operations and reducing their respective disadvantages.

[0063] The attention module CSAB provided in this embodiment is used for feature calibration. Although the multi-scale module MAB enriches the features of solder joints, these features need to be further calibrated, that is, to emphasize information features and suppress less useful features, which requires the introduction of an attention mechanism. The calibration structure in the aggregation-calibration module ACM is essentially an attention module, namely the attention module CSAB, which can calibrate both the channel information and spatial information of the aggregated features.

[0064] like Figure 4 As shown, Figure 4 To calibrate channel information, the Squeeze-and-Excitation (SE) module is a channel attention mechanism that has been improved for channel information calibration. The CNN-based attention mechanism usually includes three stages: feature aggregation, weight generation, and feature recalibration. The Squeeze-and-Excitation SE module uses global average pooling (GAP) to achieve feature aggregation. Although this is a commonly used feature aggregation method, averaging global information will blur significant features. Different from the Squeeze-and-Excitation SE module, the feature aggregation of channel attention uses both global average pooling GAP and global maximum pooling (GMP) to compress the spatial dimension of the input feature map. GMP encodes the most significant part, representing another important clue of channel information. Fusion of the two can infer more refined channel attention weights.

[0065] In addition to enhancing feature aggregation, the bottleneck fully connected layer in the weight generation stage is removed, and only a multi-layer perceptron (MLP) is used to achieve cross-channel interaction to capture the dependencies between channels. The bottleneck layer is inserted to reduce the complexity of the squeeze-excitation SE module to improve its versatility. It is removed based on three considerations: (1) It destroys the direct correspondence between channels and their weights, which will have a negative impact on attention learning; (2) AcmNet does not have a convolutional layer with a large number of channels, and the number of neurons in the MLP used is already very small. Inserting a bottleneck fully connected layer to reduce complexity will not only have little effect, but also sacrifice the accuracy of the model; (3) There is no need to make the channel attention module highly versatile, because this module is dedicated to feature calibration of the multi-scale module MAB and is not used to optimize other models.

[0066] like Figure 5 As shown, Figure 5 For spatial information calibration, the spatial attention used for spatial information calibration is a variant of channel attention. The difference is that its GAP and GMP are based on spatial direction, and a convolution layer is used to achieve cross-space interaction. Finally, the Sigmoid function is still used to generate spatial attention weights. Therefore, although two attention mechanism modules are proposed, they are different forms of a module. It can calibrate both channel information and spatial information, so it is named channel-spatial attention module. The two forms are called channel form and spatial form, respectively. They can calibrate features in parallel or continuously. The specific method to be used will be determined based on experimental results.

[0067] The depooling strategy of this embodiment is as follows: Due to the modular design, the backbone of the model can be obtained by continuously stacking the aggregation-calibration modules ACM, but the connection method will affect the performance of the model to a certain extent. After the backbone is determined, the performance of the model will depend on the network depth, receptive field and number of channels. Among them, the depth of the network is easy to control, and it can be achieved by adjusting the number of aggregation-calibration modules ACM. Therefore, the receptive field and number of channels need to be considered when designing the connection method of the aggregation-calibration module ACM.

[0068] Pooling is the most commonly used downsampling method, and its purpose is to increase the receptive field. However, since it uses a fixed downsampling rule, it will cause information loss and cannot increase the number of channels. If pooling is used to increase the receptive field, a convolution layer must be introduced to increase the number of channels. By replacing the pooling operation with a convolution operation with a stride of 2, the model can learn parameters while downsampling, which not only increases the number of channels but also avoids a large amount of information loss. As the network deepens, these are very necessary. Therefore, each aggregation-calibration module ACM is connected by a convolution layer with a stride of 2, and no additional convolution layer needs to be introduced. This is also a strategy to improve computational efficiency.

[0069] The feature fusion strategy of this embodiment is as follows: The introduction of feature fusion is mainly to make up for the shortcomings of the multi-scale module MAB. The multi-scale module MAB realizes the aggregation of different features on the same set of feature maps, but cannot fuse the features at the same position on different feature maps. Since the feature differences of the solder joint images are small, strengthening the same feature is conducive to improving the feature extraction efficiency. Another reason for using feature fusion is that it does not increase the computational burden of the model. Therefore, both the feature extraction part (convolutional network) and the prediction part of the model adopt feature fusion strategies to achieve further performance improvement.

[0070] In convolutional networks, the idea of ​​feature fusion is to utilize shallow features. In the shallow layers of the network, the resolution of the solder joint image is higher and the information is more comprehensive. When the network is deepened, the image resolution will become lower and lower, the semantic information will be stronger, and the perception of details will be poor. Therefore, in order to further utilize shallow information, each aggregation-calibration module ACM introduces jump connections to achieve feature fusion. Figure 6 As shown, Figure 6 As a feature fusion strategy in convolutional networks, skip connections continuously add shallow features to deep features, which can not only further enrich the features of solder joint images, but also combine the advantages of shallow information and semantic information. Our skip connections are only used in the aggregation-calibration module ACM and are not used continuously because AcmNet is not very deep and gradient disappearance will not occur. The middle layer of the aggregation-calibration module ACM does not participate in the skip connection because the middle layer is in an incomplete stage. At this time, adding a skip connection will affect the calibration of the features by the attention module CSAB. Moreover, the features output by the multi-scale module MAB are already rich enough and do not need to be enriched again.

[0071] In the prediction part of the model, the idea of ​​feature fusion is parallel fusion. The prediction part is used for global feature integration, and there are two main methods: GAP and fully connected layer. Due to the disadvantage of a huge number of parameters in the fully connected layer, it has gradually been replaced by GAP. In fact, the fully connected layer can be implemented by convolution operations, and then a bottleneck layer can be introduced to greatly reduce the parameters. Another advantage of using a convolution layer is that its output is a feature map, which will not destroy the spatial information of the image. This is also the basis for it to be fused with GAP. The prediction part using convolution layers and GAP in parallel is shown in the figure. Figure 7 As shown, Figure 7 For the feature fusion strategy of the prediction part, in the first branch, GAP is used to average the global information without introducing any parameters, but the features obtained by simple averaging are relatively limited. In the second branch, a convolutional layer with parameters is used to refine the feature map, and the features obtained are more accurate. After the two branches are fused and sent to the classifier, the model can obtain more information and make more accurate classifications.

[0072] This paper provides solder joint appearance recognition based on AcmNet, including ACM-based aggregation-calibration module, depooling strategy and feature fusion. The AcmNet is used to solve the seven classification tasks of solder joint appearance, such as Figure 8 As shown, Figure 8 For solder joint appearance recognition based on AcmNet, an additional basic convolution module is introduced at the bottom of the model to increase the number of channels and reduce the size of the feature map to obtain some initial features of the solder joint image. It consists of three convolutional layers and one pooling layer. At this stage, the model pays equal attention to the background and solder joints.

[0073] by Figure 8 Taking the edge solder joint recognition process in as an example, in the main structure of AcmNet, the multi-scale module MAB first aggregates the initial features, and the model at this time pays more attention to the solder joint features. After that, due to the role of the attention module CSAB, the model will focus on the unique features of a certain category of solder joints. After adding jump connections to achieve feature fusion, the aggregation-calibration module ACM is stacked with a convolution with a stride of 2 to continuously extract solder joint features.

[0074] In the prediction part of AcmNet, the feature map contains a lot of semantic information. At this time, the parallel fusion based on GAP and Conv is used to classify and predict the appearance category of the solder joint at the pixel level.

[0075] This embodiment conducts experiments based on the constructed AcmNet, and the experimental process is as follows: In order to verify the effectiveness and superiority of the proposed method and model, the algorithm performance is tested and compared using welding spot image data from an automobile production site.

[0076] The process of establishing the data set: The appearance images of the body-in-white weld spots were collected from the welding workshop of a new energy vehicle company in Chongqing, China. The 14 production lines in the welding workshop each contain several or more than a dozen workstations. Without affecting the production rhythm, five industrial cameras were deployed at two specific workstations to take pictures of various local positions of the body-in-white. There are a large number of weld spots on each body-in-white, such as Fig. 9 As shown, Fig. 9 For the body in white, the photographed welds were cropped and unqualified welds were removed according to the analysis of the engineers. A total of 7,892 weld appearance images were obtained, which were divided into 7 categories: normal welds, copper-sticky welds, edge welds, overlapping welds, incomplete welds, spatter, distortion, etc. Fig.10 As shown, Fig.10 is the solder joint appearance category, among which a) normal solder joint, b) copper solder joint, c) edge solder joint, d) overlap solder joint, e) incomplete solder joint, f) spatter, g) distortion; when the amount of data is not large, it is more reasonable to divide the data in a ratio of 6:2:2, otherwise the number of images in the test set is too small, which may lead to a large accidental result. The solder joint appearance dataset is divided into training set, validation set and test set, and the number of samples of each dataset and sample are shown in Table 1.

[0077] Table 1 Data division

[0078]

[0079] The experimental process is as follows: the input size of AcmNet is 221×221. Before training, the solder joint images are adjusted to the input size to adapt to the model, and the pixel values ​​are normalized so that they fall between 0 and 1 to speed up the convergence. The number of training images in each batch is set to 16. The Adam algorithm is used to learn the model parameters. The classification cross entropy is used as the loss function. The model is trained using the learning rate decay strategy. The decay factor is set to 0.1, and the initial learning rate is set to 0.001. After 5 consecutive iterations, the learning rate is reduced if the loss of the validation set has not decreased. It will not be reduced after reaching 0.00001. The number of iterations is set to 30.

[0080] To avoid overfitting, a real-time data augmentation strategy was used during training to expand the training set data. Each batch of 16 images was randomly horizontally mirrored, vertically mirrored, and zoomed. The validation set and the test set were cross-validated after each iteration to detect problems in the training process early and reduce time consumption. The trained model was tested for accuracy on the test set.

[0081] The 16GB computer used in the experiment was configured with an Intel(R) Core(TM) i7-10750H CPU and an NVIDIA RTX2060 GPU. All experiments were completed under the deep learning framework TensorFlow.

[0082] The model performance verification process is as follows: First, the best attention module layout scheme is determined through some experiments, and then the performance of AcmNet is verified. The two forms of the attention module CSAB can calibrate features in a continuous or parallel manner. Taking the SE module as the benchmark, the performance of the model is compared under various schemes (since the SE module is channel attention, a group of experiments with only channel forms are added for comparison).

[0083] The results are shown in Table 2. The results show that the improved SE module can infer a more refined channel attention map, because only using the channel form can improve the accuracy by 0.9%. According to the number of floating-point operations (FLOPs) and the number of parameters of the two, it can be found that removing the bottleneck fully connected layer of the SE module has almost no effect on the complexity of the model (1GFLOPs = 10^9FLOPs). When the two forms of the attention module CSAB are used at the same time, the increase in complexity is still very small, but the accuracy is always higher than the solution with only channel attention, indicating that in AcmNet, the attention module CSAB is a very lightweight and effective dual-purpose attention module. It is easy to find that the arrangement order of the channel form and the spatial form does affect the overall performance of the model, because they focus on different features. The essential difference between the parallel scheme and the continuous scheme is that the parallel scheme generates a 3D attention map, because it has both channel attention and spatial attention. According to the experimental results, the 3D attention map is more effective than the continuous arrangement of the two forms.

[0084] Table 2 Performance of the model under different attention schemes

[0085]

[0086] After the structure of AcmNet is determined, the effectiveness of each local improvement strategy is verified. Table 3 shows the performance of the model after modifying the corresponding structure. The results show that the aggregation-calibration module ACM as the main body of the model determines the overall performance of the model, followed by the connection method of the aggregation-calibration module ACM, while the feature fusion strategy has little effect on the performance. In the multi-scale module MAB, although the enhanced convolution kernel of the third branch will increase the complexity of the model, the improvement in accuracy is obvious. The dilated convolution and decomposed convolution of the two-four branches play different roles due to their respective characteristics. The dilated convolution is more effective in the shallow layer, while the decomposed convolution is more effective in the deep layer, and the 1×1 convolution plays a role in greatly reducing the complexity. The attention module CSAB not only has a significant effect on improving accuracy, but more importantly, it hardly affects the complexity of the model, which further illustrates that the dual-purpose attention module has the advantages of high efficiency and light weight. As can be seen from the table, compared with the connection method of the pooling layer and the convolution layer, the depooling strategy can not only improve the accuracy, but also reduce the complexity to improve the computational efficiency. Feature fusion plays a role in further optimizing the performance. It can slightly improve the accuracy of the model without increasing the complexity.

[0087] Table 3 Performance of the model after modifying the local structure. “-” means no structure is modified.

[0088]

[0089] In order to further analyze how the local structure works, two images of weld spots (spatter and overlap) are randomly selected to visualize the feature maps of the two modified models with poor accuracy (89.3% and 92.2%), and the feature maps of the unmodified model (accuracy of 95.2%) are used as the benchmark for comparative experiments. Four representative groups of feature maps are selected from shallow to deep layers for each model, such as Fig.11 As shown, Fig.11 The following are the feature maps of the three models: a) Feature map after the ratio of enhanced convolution is reduced; b) Feature map after Pool+Conv is used; c) Feature map of the unmodified model. The results show that the feature maps of the three models are quite different. When the ratio of enhanced convolution in the multi-scale module MAB is reduced, the quality of the feature map has declined significantly. From the shallow layer of the model, the solder joint features cannot be extracted efficiently. There are even some black blank feature maps, which shows that enhanced convolution plays a major role in feature extraction. When the connection mode of the aggregation-calibration module ACM is changed to convolution and pooling, the feature map is not as bad as Figure a, but it can be seen that some key information is indeed lost, and the model cannot effectively distinguish solder joints from the background.

[0090] Finally, in order to avoid the randomness of data division, the generalization ability of AcmNet is discussed. The solder joint appearance dataset is randomly divided into five parts (ABCDE) and five-fold cross-validation is used to compare the accuracy under different datasets. Since the dataset is divided according to 6:2:2, five-fold cross-validation can ensure that each piece of data can participate equally in the training, validation and testing of AcmNet. The composition of the five datasets and the accuracy of AcmNet on each dataset (test set) are shown in Table 4. The results show that the accuracy of AcmNet fluctuates within 5%, with an average accuracy of 92.6%, and has a certain generalization ability.

[0091] Table 4 Five data sets and their corresponding accuracy

[0092]

[0093] The deep learning CNN provided in this embodiment is applied in the RSW solder joint appearance classification, and AcmNet is constructed to solve the solder joint appearance classification task. First, the multi-scale module MAB and the attention module CSAB are proposed to enrich and calibrate the solder joint features respectively. The multi-scale module MAB is a multi-scale convolution structure that can flexibly adjust the branch ratio. It can give full play to the advantages of different convolution operations such as dilated convolution, decomposed convolution and enhanced convolution to greatly enrich the solder joint features; the attention module CSAB is a dual-purpose attention module. Its two forms can generate attention weights in the channel direction and spatial direction of the feature map respectively, and the 3D attention map generated by parallel arrangement can achieve the best effect. Then, the convolution layer with a step size of 2 is used instead of the pooling layer, which enables AcmNet to learn parameters while downsampling, thereby improving the computational efficiency and performance of the model. Finally, the performance of AcmNet is further optimized by adopting the strategies of shallow feature fusion and parallel fusion in the feature extraction part and the prediction part respectively. The effectiveness of the proposed method and the superiority of AcmNet were verified through the solder joint appearance classification experiment. The experimental results show that the enhanced convolution of the third branch of the multi-scale module MAB plays the greatest role in the module. The attention module CSAB is a lightweight and efficient attention module. The accuracy of AcmNet reaches 95.2%, which can realize the automatic classification of solder joint appearance categories on the enterprise production line, improve the degree of production automation and save labor costs.

[0094] Example 2

[0095] The solder joint appearance image recognition system based on the aggregation-calibration CNN provided in this embodiment includes:

[0096] An image input unit, used for acquiring and inputting a welding spot image;

[0097] Aggregation-calibration module ACM, used to receive solder joint images and perform solder joint feature extraction and calibration processing at different scales on the solder joint images; and realize feature fusion through skip connection;

[0098] In the prediction part, the feature maps are fused and output in parallel through global average pooling GAP, bottleneck structure and convolution layer.

[0099] The aggregation-calibration module ACM includes a multi-scale module MAB and an attention module CSAB; the multi-scale module MAB extracts solder joint features of different scales; the attention module CSAB calibrates the extracted solder joint features;

[0100] The multi-scale module MAB uses multiple convolution operations to extract multiple different features at the same position of the feature map. After extracting the features of the solder joint image through each convolution operation, each group of features is combined to obtain the solder joint features.

[0101] The attention module CSAB is provided with a channel attention mechanism to calibrate channel information. The channel attention uses global average pooling GAP and global maximum pooling GMP to compress the spatial dimension of the input feature map; uses a multi-layer perceptron MLP to achieve cross-space interaction, and uses a Sigmoid function to generate channel attention weights; calibrates the features;

[0102] The attention module CSAB also includes calibrating the spatial information of the clustered features; the spatial information calibration uses global average pooling GAP and global maximum pooling GMP to compress the spatial dimension of the input feature map; a convolutional layer is used to achieve cross-space interaction, and a Sigmoid function is used to generate spatial attention weights.

[0103] The multi-scale module MAB is provided with four parallel processing branches, as follows:

[0104] The first branch uses 1×1 convolution to reduce the dimension of the input feature map;

[0105] The second branch uses 3×3 dilated convolution for image processing;

[0106] The third branch uses square convolution and two rectangular convolutions to extract features and perform feature fusion processing;

[0107] The fourth branch decomposes the convolution kernel into a pair of asymmetric convolution kernels, processes the image separately and outputs the feature map;

[0108] The four parallel branches perform feature aggregation in the channel direction respectively to form a feature map of different features extracted by multiple convolution kernels at the same position.

[0109] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A solder joint appearance image recognition method based on an aggregation-calibration CNN, characterized in that: The following steps are involved: Get solder joint images; Construct an aggregation-calibration module ACM, and input the solder joint image into the aggregation-calibration module for processing, and extract solder joint features of different scales through the multi-scale module MAB in the aggregation-calibration module; The extracted solder joint features are calibrated through the attention module CSAB in the aggregation-calibration module; Perform feature fusion processing on the solder joint features to obtain the solder joint appearance image; Classify the processed solder joint appearance images; The multi-scale module MAB uses multiple convolution operations to extract multiple different features at the same position of the feature map. After extracting the features of the solder joint image through each convolution operation, each group of features is combined to obtain the solder joint features. The multi-scale module MAB is provided with four parallel processing branches, as follows: The first branch uses dimensionality reduction convolution to process the input feature map; The second branch uses dilated convolution for image processing; The third branch uses square convolution and two rectangular convolutions to extract features and perform feature fusion processing; The fourth branch decomposes the convolution kernel into a pair of asymmetric convolution kernels, processes the image separately and outputs the feature map; The four parallel branches perform feature aggregation in the channel direction respectively to form a feature map of different features extracted by multiple convolution kernels at the same position.

2. The solder joint appearance image recognition method based on the aggregation-calibration CNN as claimed in claim 1, characterized in that: The attention module CSAB is provided with a channel attention mechanism for channel information calibration. The channel attention uses global average pooling GAP and global maximum pooling GMP to compress the spatial dimension of the input feature map; a multi-layer perceptron MLP is used to realize cross-channel interaction, and a sigmoid function is used to generate channel attention weights; Calibrate the features.

3. The appearance image recognition method based on the aggregation-calibration CNN according to claim 1, characterized in that: The attention module CSAB also includes calibrating the spatial information of the clustered features; The spatial information calibration uses global average pooling GAP and global maximum pooling GMP to compress the spatial dimension of the input feature map; uses a convolutional layer to achieve cross-space interaction, and uses a Sigmoid function to generate spatial attention weights.

4. The solder joint appearance image recognition method based on the aggregation-calibration CNN as claimed in claim 1, characterized in that: The aggregation-calibration modules ACM are connected by a convolutional layer with a preset step value, and feature fusion is achieved through jump connections between the aggregation-calibration modules ACM.

5. The solder joint appearance image recognition method based on the aggregation-calibration CNN as claimed in claim 1, characterized in that: It also includes a prediction part, which includes a first part, namely global average pooling GAP, and a second part, namely a bottleneck structure and a convolution layer; the feature map is output after averaging global information through global average pooling GAP; At the same time, the feature map is output through the convolution layer after passing through the bottleneck structure; the output results of the first part and the second part are fused and then sent to the classifier.

6. A solder joint appearance image recognition system based on an aggregation-calibration CNN, characterized by: include An image input unit, used for acquiring and inputting a welding spot image; Aggregation-calibration module ACM, used to receive solder joint images and perform solder joint feature extraction and calibration processing at different scales on the solder joint images; and realize feature fusion through skip connection; In the prediction part, the feature maps are fused and output in parallel through global average pooling GAP, bottleneck structure and convolution layer; The aggregation-calibration module ACM comprises a multi-scale module MAB; the multi-scale module MAB extracts solder joint features of different scales; The multi-scale module MAB uses multiple convolution operations to extract multiple different features at the same position of the feature map. After extracting the features of the solder joint image through each convolution operation, each group of features is combined to obtain the solder joint features. The multi-scale module MAB is provided with four parallel processing branches, as follows: The first branch uses dimensionality reduction convolution to process the input feature map; The second branch uses dilated convolution for image processing; The third branch uses square convolution and two rectangular convolutions to extract features and perform feature fusion processing; The fourth branch decomposes the convolution kernel into a pair of asymmetric convolution kernels, processes the image separately and outputs the feature map; The four parallel branches perform feature aggregation in the channel direction respectively to form a feature map of different features extracted by multiple convolution kernels at the same position.

7. The solder joint appearance image recognition system based on the aggregation-calibration CNN as claimed in claim 6, characterized in that: The aggregation-calibration module ACM comprises an attention module CSAB; The attention module CSAB calibrates the extracted solder joint features; The attention module CSAB is provided with a channel attention mechanism for channel information calibration. The channel attention uses global average pooling GAP and global maximum pooling GMP to compress the spatial dimension of the input feature map; a multi-layer perceptron MLP is used to realize cross-channel interaction, and Sigmoid is used to generate channel attention weights; Calibrate the features; The attention module CSAB also includes calibrating the spatial information of the clustered features; The spatial information calibration uses global average pooling GAP and global maximum pooling GMP to compress the spatial dimension of the input feature map; uses a convolutional layer to achieve cross-space interaction, and uses a Sigmoid function to generate spatial attention weights.

Images