Solder paste printing plate quality detection method, device, equipment and medium

Abstract

The invention relates to a solder paste printing plate quality detection method, device, equipment and medium, and the method comprises the steps: inputting a to-be-detected solder paste printing plate image into a pre-trained quality detection model to obtain a reconstructed solder paste printing plate image, carrying out the dimension reduction of the sample solder paste printing plate image through a space encoder, and obtaining a low-dimensional space coding feature; a space-time encoder decoder carries out capture operation on the space encoding features based on time flow and a space-time mode to obtain generation and actual output values of each time step, and selects target input of the time step from the generation and actual output values of the previous time step in a probability mode; in the space-time coding stage, space-time attention features are added in target input to obtain space-time features to be decoded, and in the space-time decoding stage, the space-time attention features are subtracted from the decoded space-time features to obtain space-time decoding features; the space decoder decodes and reconstructs the low-dimensional to-be-decoded features into a high-dimensional space reconstruction sample image; and obtaining a quality detection result according to the reconstructed solder paste printing plate image and the to-be-detected solder paste printing plate image.

Description

Technical Field

[0001] The present invention relates to the technical field of circuit boards, and in particular to a method, device, equipment and medium for detecting quality of solder paste printed boards. Background Art

[0002] Solder paste printing is a key process in electronic equipment manufacturing. The quality of solder paste printing directly affects the quality and product performance of printed circuit boards (PCBs). For example, the accuracy and uniformity of solder paste printing affect the performance of printed circuit boards. Generally, if there is insufficient, excessive or offset solder paste during solder paste printing, it will affect the reliability of welding and cause abnormalities in printed circuit boards during assembly and functional testing.

[0003] At present, the quality inspection of solder paste printing mainly relies on manual inspection and automated visual inspection (AVI) system. However, the accuracy of manual inspection is limited by the accumulated experience of operators, while the performance of AVI system depends on image processing algorithm and optical conditions, which is prone to false detection and missed detection in complex environments, reducing the detection accuracy and increasing production costs. Summary of the invention

[0004] The purpose of the present invention is to provide a method, device, equipment and medium for detecting the quality of solder paste printed boards, aiming to be able to detect the quality of solder paste printing in real time, improve the detection efficiency and detection accuracy, and quickly identify and feedback abnormalities, thereby improving the product quality and production efficiency of electronic equipment manufacturing and meeting the market demand for high-quality electronic products.

[0005] In order to achieve the above-mentioned purpose, a first aspect of an embodiment of the present disclosure provides a method for detecting quality of a solder paste printed board, the method comprising:

[0006] Acquire the image of the solder paste printed board to be inspected;

[0007] The image of the solder paste printed board to be inspected is input into a pre-trained quality inspection model to obtain the features to be identified output by the quality inspection model, wherein, during the training process, the spatial encoder of the quality inspection model converts the high-dimensional feature map of the sample solder paste printed board image into a low-dimensional feature map through a convolution layer, prevents covariance shift by normalizing the value of each layer through a normalization layer, and introduces nonlinear enhancement modeling capabilities through an activation layer to obtain low-dimensional spatial coding features. The spatiotemporal encoder-decoder of the quality inspection model performs spatial coding based on time flow and spatiotemporal patterns on the low-dimensional spatial coding features through an input gate, a forget gate, and an output gate. Capture operation to obtain the generated output value and actual output value of each time step, and select the target input for calculation in this time step from the generated output value and actual output value of the previous time step in a probabilistic manner, and determine the spatiotemporal attention feature, and add the spatiotemporal attention feature to the target input in the spatiotemporal encoding stage to obtain the spatiotemporal feature to be decoded, and subtract the spatiotemporal attention feature from the decoded spatiotemporal feature obtained by decoding in the spatiotemporal decoding stage to obtain the spatiotemporal decoding feature; the spatial decoder of the quality detection model decodes the low-dimensional features to be decoded into a reconstructed sample image in a high-dimensional space;

[0008] The reconstructed solder paste printed board image is compared with the image of the solder paste printed board to be inspected to obtain a reconstruction error, and a quality inspection result of the solder paste printed board is obtained based on the reconstruction error.

[0009] In a possible implementation, the low-dimensional spatial encoding features are subjected to time flow and spatiotemporal pattern capture operations through the input gate, the forget gate, and the output gate to obtain the generated output value and the actual output value of each time step, including:

[0010] The input gate performs spatiotemporal pattern capture through a unit gate that performs spatiotemporal pattern capture at the same time step of the previous layer to obtain a first feature, and performs spatiotemporal pattern capture through a unit gate that performs spatiotemporal pattern capture at the previous time step of the current layer to obtain a second feature, and performs time flow information capture through a unit gate that performs time flow capture at the previous time step of the current layer to obtain a third feature;

[0011] Determine a first spatiotemporal coding feature corresponding to the input gate according to the first feature, the second feature, the third feature and corresponding weights;

[0012] The forget gate captures the spatiotemporal pattern through the unit gate that captures the spatiotemporal pattern at the same time step of the previous layer to obtain the fourth feature, and captures the spatiotemporal pattern through the unit gate that captures the spatiotemporal pattern at the previous time step of the current layer to obtain the fifth feature, and captures the time flow information through the unit gate that captures the time flow at the previous time step of the current layer to obtain the sixth feature;

[0013] Determine a second spatiotemporal coding feature corresponding to the input gate according to the third feature, the fourth feature, the fifth feature and the corresponding weights;

[0014] The output gate captures the spatiotemporal pattern through the unit gate that captures the spatiotemporal pattern in the previous time step of the current layer, thereby obtaining the seventh feature and the eighth feature, and captures the time flow information through the unit gate that captures the time flow in the current time step, thereby obtaining the ninth feature;

[0015] Determine the third spatiotemporal coding feature corresponding to the input gate according to the seventh feature, the eighth feature, the ninth feature and the corresponding weights, obtain the generated output value of each time step according to the third spatiotemporal coding feature, and match the actual output value corresponding to the sample solder paste printing board image with the generated output value based on the time step;

[0016] Among them, the unit gate for capturing spatiotemporal patterns is determined based on the unit gate and output gate for capturing time flow, and the unit gate for capturing time flow is determined based on the forget gate, input gate, and the weight matrix of the convolution operation for reducing the number of channels to half.

[0017] In one possible implementation, the low-dimensional spatial encoding features are operated based on time flow and spatiotemporal pattern capture through the following formula to obtain the generated output value for each time step:

[0018]

[0019]

[0020]

[0021]

[0022]

[0023] Where l and t represent the neuron layer and time step, respectively. They represent the input gate, forget gate and output gate respectively. is the unit gate for updating the unit state of the lth layer based on the time flow in the tth time step, is the unit gate for updating the unit state of the lth layer based on the time flow in the t-1th time step, is the unit gate of the l-1th layer in the tth time step that updates the unit state based on the time flow, W i , U i , W c , W f , U f , W f , WO , U O , W O are weights, is the hidden state of the l-1th layer in the t-1th time step, is the hidden state of the l-1th layer in the tth time step, is the hidden state of the lth layer in the tth time step, The candidate information generated by the lth layer in the tth time step is generated by the weighted sum of the target input at the tth time step and the hidden state at the t-1th time step, and then generated by the tanh activation function. is a 1×1 convolution kernel, N c is the number of channels of the feature, σ is the sigmoid activation function, which is used to map the result to between 0 and 1, ο is the Hadamard product operation, and tanh is the hyperbolic tangent activation function, which is used to map the result to between -1 and 1. For the concatenation operation, the unit state of the lth layer at the t-1th time step and the unit state of the l-1th layer at the tth time step are concatenated to simultaneously consider the spatiotemporal relationship between the previous and next layers.

[0024] In a possible implementation, the determining of the spatiotemporal attention feature, adding the spatiotemporal attention feature to the target input in the spatiotemporal encoding stage to obtain the spatiotemporal feature to be decoded, and subtracting the spatiotemporal attention feature from the decoded spatiotemporal feature obtained by decoding in the spatiotemporal decoding stage to obtain the spatiotemporal decoding feature, includes:

[0025] According to the weight matrix of the convolution operation and the convolution kernel size of the convolution operation, the hidden state with any channel is converted into a first attention feature map with a preset channel;

[0026] Copying the first attention feature map in a channel manner to obtain the spatiotemporal attention feature, wherein the channel manner copying makes the spatiotemporal attention feature match the number of channels of the target input;

[0027] In the spatiotemporal encoding stage, the target input and the spatiotemporal attention feature are summed to add the spatiotemporal attention feature to the target input to obtain a spatiotemporal feature to be decoded;

[0028] In the spatiotemporal decoding stage, the spatiotemporal features to be decoded are spatiotemporally decoded to obtain decoded spatiotemporal features, and the decoded spatiotemporal features are subtracted from the spatiotemporal attention features to obtain the spatiotemporal decoding features.

[0029] In a possible implementation, the selecting, in a probabilistic manner, a target input for calculation at the current time step from a generated output value and an actual output value at a previous time step includes:

[0030] Determine a sampling mask for the real input and the generated input at this time step according to the probability factor, wherein the probability factor is reduced and updated according to a preset cooling factor before each determination of the sampling mask;

[0031] According to the sampling mask, the target input for calculation in the current time step is selected from the generated output value and the actual output value of the previous time step.

[0032] In a possible implementation, the high-dimensional feature map of the sample solder paste printed board image is converted into a low-dimensional feature map through a convolution layer, the covariance shift is prevented by normalizing the value of each layer through a normalization layer, and the nonlinear enhancement modeling capability is introduced through an activation layer to obtain a low-dimensional spatial encoding feature, including:

[0033] The convolution kernels with different convolution kernel sizes are slid on the sample solder paste printing board image to perform weighted summation of the local area, and the local features of the sample solder paste printing board image are respectively extracted to obtain a low-dimensional feature map, wherein the convolution kernels with different convolution kernel sizes respectively capture the features of the sample solder paste printing board image in terms of edge, texture, and color through different convolution kernels;

[0034] The normalization layer calculates the mean and variance of each feature on each input batch for the low-dimensional feature map of each batch input, and normalizes each input batch so that the difference between the mean and 0 meets the first preset condition, and the difference between the variance and 1 meets the second preset condition through a learnable scaling factor and offset, thereby obtaining a normalized output to stabilize the input distribution of each layer and prevent covariance shift;

[0035] The negative values ​​in the normalized output are set to 0 through the activation layer through a nonlinear activation function, while the positive values ​​remain unchanged, thereby introducing nonlinear enhanced modeling capabilities and obtaining the low-dimensional spatial encoding features.

[0036] In a possible implementation, the decoding and reconstructing of the low-dimensional features to be decoded into a reconstructed sample image in a high-dimensional space includes:

[0037] Inputting the to-be-decoded features into the first hidden layer through a plurality of neurons in the input layer;

[0038] The neurons in the multiple hidden layers perform weighted sum operations on the reconstructed features output by the neurons in the previous layer to obtain the reconstructed features output by the neurons in this layer, wherein each neuron in each hidden layer is connected to multiple neurons in the previous hidden layer and to multiple neurons in the next hidden layer;

[0039] The reconstructed features output by multiple neurons in the last hidden layer are weighted and summed through the output layer to reconstruct a reconstructed sample image in a high-dimensional space.

[0040] A second aspect of the embodiments of the present disclosure provides a solder paste printing board quality detection device, comprising:

[0041] An acquisition module is configured to acquire an image of a solder paste printed board to be inspected;

[0042] The input module is configured to input the image of the solder paste printed board to be inspected into a pre-trained quality inspection model to obtain the features to be identified output by the quality inspection model, wherein the spatial encoder of the quality inspection model converts the high-dimensional feature map of the sample solder paste printed board image into a low-dimensional feature map through a convolution layer, prevents covariance shift based on the value of each layer through a normalization layer, introduces nonlinear enhancement modeling capabilities through an activation layer to obtain low-dimensional spatial encoding features, and the spatiotemporal encoder-decoder of the quality inspection model performs temporal flow and time-based spatial encoding on the obtained low-dimensional spatial encoding features through an input gate, a forget gate, and an output gate. The empty mode capture operation is performed to obtain the generated output value and the actual output value of each time step, and the target input for calculation in this time step is selected from the generated output value and the actual output value of the previous time step in a probabilistic manner, and the spatiotemporal attention feature is determined, and the spatiotemporal attention feature is added to the target input in the spatiotemporal encoding stage to obtain the spatiotemporal feature to be decoded, and the spatiotemporal attention feature is subtracted from the decoded spatiotemporal feature obtained by decoding in the spatiotemporal decoding stage to obtain the spatiotemporal decoding feature; the spatial decoder of the quality detection model decodes the low-dimensional features to be decoded into a reconstructed sample image in a high-dimensional space;

[0043] The detection and identification module is configured to compare the reconstructed solder paste printed board image with the image of the solder paste printed board to be detected to obtain a reconstruction error, and obtain a quality detection result of the solder paste printed board based on the reconstruction error.

[0044] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0045] According to a fourth aspect of the embodiments of the present disclosure, an electronic device is provided, including:

[0046] a memory having a computer program stored thereon;

[0047] A processor is used to execute the computer program in the memory to implement the steps of any one of the methods in the first aspect.

[0048] The present invention provides a method, device, equipment and medium for detecting the quality of solder paste printed boards. Compared with the prior art, it has the following beneficial effects:

[0049] Through the combination of spatial encoder convolutional layers, normalization layers, and activation layers, the high-dimensional feature map of the sample solder paste printed board image can be effectively converted into a low-dimensional feature map, while preventing covariance shift and introducing nonlinear enhancement modeling capabilities, thereby improving the efficiency and accuracy of feature extraction.

[0050] Furthermore, by controlling the input gate, forget gate, and output gate of the spatiotemporal encoder-decoder, the low-dimensional spatial encoding features can be decoded and reconstructed based on the time flow and spatiotemporal pattern capture operations, thereby capturing the spatiotemporal variation characteristics in the solder paste printing board image and providing richer information for quality inspection. The introduction of spatiotemporal attention features in the spatiotemporal encoding stage can enhance the model's attention to key information and improve the model's sensitivity to quality issues of solder paste printing boards. At the same time, subtracting the spatiotemporal attention features in the spatiotemporal decoding stage can further purify the decoding results and improve the accuracy of quality inspection.

[0051] Furthermore, the spatial decoder can decode and reconstruct the low-dimensional features to be decoded into a reconstructed sample image in a high-dimensional space, and compare it with the image of the solder paste printing board to be tested to obtain the reconstruction error. By evaluating the reconstruction error, the quality problems of the solder paste printing board, such as whether the printing thickness, flatness, printing area, etc. meet the requirements, can be accurately judged.

[0052] At the same time, it can realize the automatic detection of solder paste printing board quality without manual intervention, greatly improving the detection efficiency and accuracy. At the same time, it can be applied to quality inspection on large-scale production lines, providing strong support for quality control and cost control in the electronics manufacturing industry.

[0053] In summary, by adopting the structure of spatial encoder, spatiotemporal encoder-decoder and spatial decoder, the automatic detection of the quality of solder paste printed boards is realized, which has significant technical effects such as efficient feature extraction, spatiotemporal pattern capture, spatiotemporal attention mechanism, high-precision reconstruction and automatic detection.

[0054] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0056] Figure 1 It is a flow chart of a method for detecting quality of a solder paste printed board according to an embodiment of the specification.

[0057] Figure 2 It is a block diagram of a solder paste printing board quality detection device shown in an embodiment of the specification.

[0058] Figure 3 It is a block diagram of another solder paste printing board quality detection device shown in an embodiment of the specification. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0061] The present invention provides a method for detecting the quality of a solder paste printed board. Figure 1 The flowchart of a method for detecting the quality of a solder paste printed board according to an embodiment is shown. The method comprises:

[0062] In step S11, an image of a solder paste printed board to be inspected is obtained;

[0063] In step S12, the image of the solder paste printed board to be inspected is input into a pre-trained quality inspection model to obtain the features to be identified output by the quality inspection model, wherein, during the training process, the spatial encoder of the quality inspection model converts the high-dimensional feature map of the sample solder paste printed board image into a low-dimensional feature map through a convolution layer, prevents covariance shift by normalizing the values ​​of each layer through a normalization layer, and obtains low-dimensional spatial coding features by introducing nonlinear enhancement modeling capabilities through an activation layer. The spatiotemporal encoder-decoder of the quality inspection model performs spatial coding on the low-dimensional spatial coding features based on time flow and The spatiotemporal pattern capture operation obtains the generated output value and the actual output value of each time step, and selects the target input for calculation in this time step from the generated output value and the actual output value of the previous time step in a probabilistic manner, and determines the spatiotemporal attention feature, and adds the spatiotemporal attention feature to the target input in the spatiotemporal encoding stage to obtain the spatiotemporal feature to be decoded, and subtracts the spatiotemporal attention feature from the decoded spatiotemporal feature obtained by decoding in the spatiotemporal decoding stage to obtain the spatiotemporal decoding feature; the spatial decoder of the quality detection model decodes the low-dimensional features to be decoded into a reconstructed sample image in a high-dimensional space;

[0064] In the disclosed embodiment, the spatiotemporal encoder decoder receives the low-dimensional features of the spatial encoder as input. The features are captured based on the time flow and spatiotemporal pattern through the input gate, forget gate and output gate (i.e., LSTM or GRU and other recurrent neural network units). These gating mechanisms help the model learn the temporal and spatial relationships between features.

[0065] In the disclosed embodiment, at each time step, the spatiotemporal encoder-decoder generates an output value and compares it with the actual output value. In this way, the model can gradually optimize its prediction ability.

[0066] In the disclosed embodiment, the model probabilistically selects the target input for calculation in this time step from the generated output value and the actual output value of the previous time step. Based on the target input, the model determines the spatiotemporal attention features, which reflect the key information of the current time step. In the spatiotemporal encoding stage, the model adds the spatiotemporal attention features to the target input to obtain the spatiotemporal features to be decoded. In the spatiotemporal decoding stage, the model subtracts the spatiotemporal attention features from the decoded spatiotemporal features obtained by decoding to obtain the spatiotemporal decoding features. This process helps the model capture and represent spatiotemporal information more accurately.

[0067] In the disclosed embodiment, the spatial decoder receives the spatiotemporal decoding features of the spatiotemporal encoder-decoder as input, and decodes and reconstructs the low-dimensional features into a reconstructed sample image in a high-dimensional space through a series of operations such as deconvolution layers and upsampling layers.

[0068] Among them, the spatial encoder (S-Encoder) is converted from the original spatial data Extract spatial features from Where Nc, Nh, and Nw represent the number of channels, height, and width, respectively. S-Encoder contains multiple modules, each of which consists of a convolutional layer, a batch normalization layer, and a ReLU activation layer. The convolutional layer converts the high-dimensional feature map into a low-dimensional feature map, the batch normalization layer prevents covariance shift by normalizing the values ​​of each layer, and the ReLU activation layer introduces nonlinearity to enhance the modeling ability of CRRN. The spatial decoder (S-Decoder) performs the opposite operation of the S-Encoder. The S-Decoder generates reconstructed spatial data and low-dimensional space features S-Decoder also consists of multiple modules. Unlike S-Encoder, S-Decoder uses a deconvolution operation called transposed convolution instead of convolution. Deconvolution reconstructs low-dimensional feature maps into high-dimensional feature maps. The last module of S-Decoder does not include a ReLU activation layer, ensuring that the generated output has a valid range.

[0069] In step S13, the reconstructed solder paste printed board image is compared with the image of the solder paste printed board to be inspected to obtain a reconstruction error, and a solder paste printed board quality inspection result is obtained based on the reconstruction error.

[0070] In the disclosed embodiments, the comparison process usually involves a comparison at the pixel level, that is, calculating the difference between the two images at corresponding pixel positions. In order to quantify this difference, the model will use one or more error calculation methods, such as mean square error (MSE), mean absolute error (MAE) or structural similarity index (SSIM). These error indicators can reflect the degree of similarity between the reconstructed image and the original image, thereby providing a basis for quality assessment. The calculated error value will undergo further processing, such as normalization, weighting, etc., to ensure that the error value can accurately reflect the quality problems of the solder paste printed board.

[0071] In the disclosed embodiment, during the quality inspection process, one or more thresholds are usually set to determine whether the reconstruction error exceeds an acceptable range. These thresholds can be set according to actual application requirements and verified by experiments to ensure their accuracy. The calculated reconstruction error is compared with the set threshold to determine whether the quality of the solder paste printed board to be inspected is qualified. If the reconstruction error is less than or equal to the threshold, the quality of the solder paste printed board is considered to be qualified; otherwise, the quality is considered to be unqualified.

[0072] In the disclosed embodiment, the quality inspection results are output in an appropriate form, such as text, image or number, etc. These results can be used to guide the subsequent production process, such as whether reprinting is needed, adjusting printing parameters, etc.

[0073] The reconstruction error is further analyzed to identify possible causes of quality problems. These causes may include errors in the printing process, material quality issues, or limitations of the model itself. Based on the results of the error analysis, the quality inspection model can be further optimized. This includes adjusting the model structure, increasing training data, optimizing training parameters, etc. to improve the accuracy and generalization ability of the model.

[0074] The reconstructed sample image is compared with the original sample image to calculate the reconstruction error. The reconstruction error reflects the model's ability to reconstruct the input image and is also an important indicator for evaluating the model's quality detection performance.

[0075] The above technical scheme can effectively convert the high-dimensional feature map of the sample solder paste printed board image into a low-dimensional feature map through the combination of the spatial encoder convolution layer, the normalization layer and the activation layer, while preventing covariance shift and introducing nonlinear enhanced modeling capabilities, thereby improving the efficiency and accuracy of feature extraction.

[0076] Furthermore, by controlling the input gate, forget gate, and output gate of the spatiotemporal encoder-decoder, the low-dimensional spatial encoding features can be decoded and reconstructed based on the time flow and spatiotemporal pattern capture operations, thereby capturing the spatiotemporal variation characteristics in the solder paste printing board image and providing richer information for quality inspection. The introduction of spatiotemporal attention features in the spatiotemporal encoding stage can enhance the model's attention to key information and improve the model's sensitivity to quality issues of solder paste printing boards. At the same time, subtracting the spatiotemporal attention features in the spatiotemporal decoding stage can further purify the decoding results and improve the accuracy of quality inspection.

[0077] Furthermore, the spatial decoder can decode and reconstruct the low-dimensional features to be decoded into a reconstructed sample image in a high-dimensional space, and compare it with the image of the solder paste printing board to be tested to obtain the reconstruction error. By evaluating the reconstruction error, the quality problems of the solder paste printing board, such as whether the printing thickness, flatness, printing area, etc. meet the requirements, can be accurately judged.

[0078] At the same time, it can realize the automatic detection of solder paste printing board quality without manual intervention, greatly improving the detection efficiency and accuracy. At the same time, it can be applied to quality inspection on large-scale production lines, providing strong support for quality control and cost control in the electronics manufacturing industry.

[0079] In summary, by adopting the structure of spatial encoder, spatiotemporal encoder-decoder and spatial decoder, the automatic detection of the quality of solder paste printed boards is realized, which has significant technical effects such as efficient feature extraction, spatiotemporal pattern capture, spatiotemporal attention mechanism, high-precision reconstruction and automatic detection.

[0080] In a possible implementation, the low-dimensional spatial encoding features are subjected to time flow and spatiotemporal pattern capture operations through the input gate, the forget gate, and the output gate to obtain the generated output value and the actual output value of each time step, including:

[0081] In step S21, the input gate performs spatiotemporal pattern capture through a unit gate that performs spatiotemporal pattern capture at the same time step of the previous layer to obtain a first feature, and performs spatiotemporal pattern capture through a unit gate that performs spatiotemporal pattern capture at the previous time step of the current layer to obtain a second feature, and performs time flow information capture through a unit gate that performs time flow capture at the previous time step of the current layer to obtain a third feature;

[0082] In the disclosed embodiment, the first layer captures the same time step: the input gate first captures the spatiotemporal pattern through the unit gate (such as the hidden state of LSTM or GRU) of the previous layer at the same time step to obtain the first feature. This feature reflects the spatiotemporal pattern information of the previous layer at the same time step.

[0083] Capture of the previous time step of this layer: Then, the input gate is captured by the unit gate that captures the spatiotemporal pattern of the previous time step of this layer to obtain the second feature. This feature reflects the spatiotemporal pattern information of the previous time step of this layer in the time series.

[0084] Time flow information capture: The input gate also captures time flow information through the unit gate (such as the cell state of LSTM or GRU) of the previous time step of this layer, and obtains the third feature. This feature contains the accumulated information from the beginning of the time series to the current time step.

[0085] In step S22, a first spatiotemporal coding feature corresponding to the input gate is determined according to the first feature, the second feature, the third feature and the corresponding weights;

[0086] In the disclosed embodiment, after obtaining the first feature, the second feature, and the third feature, the input gate determines the corresponding first spatiotemporal coding feature by weighted summation or other fusion methods according to these features and their corresponding weights (these weights can be obtained through training and learning). This feature integrates the spatiotemporal pattern information of the same time step of the previous layer and the previous time step of the current layer, as well as the time flow information.

[0087] In step S23, the forget gate performs spatiotemporal pattern capture through the unit gate that performs spatiotemporal pattern capture at the same time step of the previous layer to obtain the fourth feature, and performs spatiotemporal pattern capture through the unit gate that performs spatiotemporal pattern capture at the previous time step of the current layer to obtain the fifth feature, and performs time flow information capture through the unit gate that performs time flow capture at the previous time step of the current layer to obtain the sixth feature;

[0088] In the disclosed embodiment, similar to the input gate, the forget gate is also captured by the unit gate that captures the spatiotemporal pattern at the same time step in the previous layer to obtain the fourth feature.

[0089] At the same time, the forget gate also captures the spatiotemporal pattern through the unit gate of the previous time step of this layer to obtain the fifth feature.

[0090] Capturing time flow information: The forget gate also captures time flow information through the unit gate that captured time flow in the previous time step of this layer, but the sixth feature obtained here is different from that of the input gate (although they may be based on the same cell state, the weights and fusion methods may be different).

[0091] In step S24, a second spatiotemporal coding feature corresponding to the input gate is determined according to the third feature, the fourth feature, the fifth feature and the corresponding weights;

[0092] In the disclosed embodiment, after obtaining the fourth feature, the fifth feature, and the sixth feature, the forget gate determines the corresponding second spatiotemporal coding feature according to these features and their corresponding weights (these weights can also be obtained through training and learning) through weighted summation or other fusion methods. This feature also integrates the spatiotemporal pattern information of the same time step of the previous layer and the previous time step of the current layer, as well as the time flow information, but the weight and fusion method are different from those of the input gate, which reflects the uniqueness of the forget gate in determining which information should be retained or forgotten.

[0093] In step S25, the output gate performs spatiotemporal pattern capture through the unit gate that performs spatiotemporal pattern capture at the previous time step of the current layer to obtain the seventh feature and the eighth feature, and performs time flow information capture through the unit gate that performs time flow capture at the time step of the current layer to obtain the ninth feature;

[0094] In the disclosed embodiment, the output gate is captured by the unit gate (such as the hidden state of LSTM or GRU) of the previous time step of the current layer to obtain the seventh feature and the eighth feature. These two features represent the different spatiotemporal pattern information of the previous time step of the current layer in the time series. It should be noted that the two features here may come from different unit gates or the same unit gate but captured using different weights.

[0095] Time flow information capture: The output gate also captures time flow information through the unit gate (such as the cell state of LSTM or GRU) of this layer time step to obtain the ninth feature. This feature contains the accumulated information from the beginning of the time series to the current time step, which is crucial for generating the output value of the current time step.

[0096] In step S26, according to the seventh feature, the eighth feature, the ninth feature and the corresponding weights, the third spatiotemporal coding feature corresponding to the input gate is determined, and the generated output value of each time step is obtained according to the third spatiotemporal coding feature, and the actual output value corresponding to the sample solder paste printing board image is matched with the generated output value based on the time step;

[0097] In the disclosed embodiment, after obtaining the seventh feature, the eighth feature, and the ninth feature, the output gate performs weighted summation or other fusion methods based on these features and their corresponding weights (these weights are obtained through training and learning) to determine the corresponding third spatiotemporal coding feature. This feature integrates the spatiotemporal pattern information of the previous time step of this layer and the time flow information of the current time step.

[0098] Generate output value: Based on the third spatiotemporal encoding feature, the output gate generates the generated output value of the current time step through a nonlinear activation function (such as sigmoid or softmax). This output value represents the model's prediction of the quality of the solder paste printing board image at the current time step.

[0099] Matching of actual output values ​​with generated output values: The model matches the actual output values ​​corresponding to the sample solder paste printed board image with the generated output values ​​based on the time step. This step usually involves calculating the error between the two (such as cross entropy loss) for subsequent back propagation and model optimization.

[0100] Among them, the unit gate for capturing spatiotemporal patterns is determined based on the unit gate and output gate for capturing time flow, and the unit gate for capturing time flow is determined based on the forget gate, input gate, and the weight matrix of the convolution operation for reducing the number of channels to half.

[0101] Spatiotemporal pattern capturing gates: The updates of these gates (such as the hidden state of LSTM or GRU) may depend on the current state of the gates that capture temporal flow (such as the cell state) and the output gate. This means that when capturing spatiotemporal patterns, the model takes into account the accumulated temporal flow information of the current time step and the needs of the output gate.

[0102] Time flow capture unit gates: The updates of these unit gates (such as the cell state of LSTM or GRU) depend on the forget gate, the input gate, and the weight matrix of the convolution operation that reduces the number of channels to half. The forget gate determines what information should be retained in the cell state, the input gate determines what new information should be added to the cell state, and the convolution operation is used to adjust the number of channels of the feature to suit the needs of the model.

[0103] In the disclosed embodiment, the proposed CSTM is developed based on the convolutional long short-term memory network (ConvLSTM) and the spatiotemporal long short-term memory network (ST-LSTM). Unlike the traditional LSTM, which is difficult to extract spatial features, the input gate and the forget gate respectively capture and fuse the spatiotemporal pattern information and time flow information from different time steps and layers, forming their corresponding spatiotemporal coding features. The convolution operation enables the ConvLSTM to extract temporal and spatial information at the same time, while using fewer parameters than the LSTM. The operation description of the ConvLSTM is the construction basis of the CSTM.

[0104] In one possible implementation, the low-dimensional spatial encoding features are operated based on time flow and spatiotemporal pattern capture through the following formula to obtain the generated output value for each time step:

[0105]

[0106]

[0107]

[0108]

[0109]

[0110] Where l and t represent the neuron layer and time step, respectively. They represent the input gate, forget gate and output gate respectively. is the unit gate for updating the unit state of the lth layer based on the time flow in the tth time step, is the unit gate for updating the unit state of the lth layer based on the time flow in the t-1th time step, is the unit gate of the l-1th layer in the tth time step that updates the unit state based on the time flow, W i , U i , W c , W f , U f , W f , W O , U O , W O are weights, is the hidden state of the l-1th layer in the t-1th time step, is the hidden state of the l-1th layer in the tth time step, is the hidden state of the lth layer in the tth time step, The candidate information generated by the lth layer in the tth time step is generated by the weighted sum of the target input at the tth time step and the hidden state at the t-1th time step, and then generated by the tanh activation function. is a 1×1 convolution kernel, N c is the number of channels of the feature, σ is the sigmoid activation function, which is used to map the result to between 0 and 1, ο is the Hadamard product operation, and tanh is the hyperbolic tangent activation function, which is used to map the result to between -1 and 1. For the concatenation operation, the unit state of the lth layer at the t-1th time step and the unit state of the l-1th layer at the tth time step are concatenated to simultaneously consider the spatiotemporal relationship between the previous and next layers.

[0111] It can be explained that the input gate Used to determine how much of the new information needs to be stored in the cell state, the forget gate It is used to determine how much information in the previous unit state needs to be "forgotten", the output gate It is used to determine how much unit state information needs to be passed to the next layer or output, and ο is the Hadamard product operation used to multiply the corresponding elements one by one.

[0112] Furthermore, through the forget gate And the weighted splicing state is used to update the forgotten part of the unit state, so that the network can dynamically control how much historical information is retained. Represents the "new information" that needs to be added to the cell state in the current time step.

[0113] Further, Represents the unit state of the tth time step and the lth layer It consists of two parts: one part is the weighted fusion result of the unit state of the previous time step and the unit state of the previous layer of the current time step, and the other part is the addition of new information. These two parts are connected through the forget gate. and input gate Dynamic control is performed to achieve information retention and update. The historical information can be partially retained (through the forget gate) and the unit state can be updated according to the current input (through the input gate), thereby achieving recursive information update.

[0114] In a possible implementation, the determining of the spatiotemporal attention feature, adding the spatiotemporal attention feature to the target input in the spatiotemporal encoding stage to obtain the spatiotemporal feature to be decoded, and subtracting the spatiotemporal attention feature from the decoded spatiotemporal feature obtained by decoding in the spatiotemporal decoding stage to obtain the spatiotemporal decoding feature, includes:

[0115] In step A1, according to the weight matrix of the convolution operation and the convolution kernel size of the convolution operation, the hidden state with any channel is converted into a first attention feature map with a preset channel;

[0116] Convolution operation is performed on hidden states with any channels using a preset convolution kernel size and weight matrix. This step aims to extract key spatiotemporal features in the hidden state and convert them into a first attention feature map with a preset number of channels. The size of the convolution kernel determines the receptive field during feature extraction, while the weight matrix is ​​learned through training to optimize the effect of feature extraction.

[0117] Channel adjustment: The number of channels of the first attention feature map is preset according to the model design, which is usually different from the number of channels of the target input. Therefore, the output of this step is a feature map with a specific number of channels for subsequent attention mechanism construction.

[0118] In step A2, the first attention feature map is copied in a channel manner to obtain the spatiotemporal attention feature, wherein the channel manner copying makes the spatiotemporal attention feature match the number of channels of the target input;

[0119] In order to match the spatiotemporal attention features with the target input, the first attention feature map needs to be copied channel by channel until its number of channels is the same as the number of channels of the target input. This step ensures that the spatiotemporal attention features can be element-by-element with the target input in subsequent operations.

[0120] In step A3, in the spatiotemporal encoding stage, the target input and the spatiotemporal attention feature are summed to add the spatiotemporal attention feature to the target input to obtain a spatiotemporal feature to be decoded;

[0121] In the spatiotemporal encoding stage, the target input is summed with the spatiotemporal attention features. This step aims to add the key information in the spatiotemporal attention features to the target input, thereby enhancing the target input's ability to perceive key spatiotemporal information. The feature map obtained after the summation operation is called the spatiotemporal feature to be decoded, which contains the fusion information of the target input and the spatiotemporal attention features.

[0122] In step A4, in the spatiotemporal decoding stage, the spatiotemporal features to be decoded are spatiotemporally decoded to obtain decoded spatiotemporal features, and the decoded spatiotemporal features are subtracted from the spatiotemporal attention features to obtain the spatiotemporal decoding features.

[0123] Spatiotemporal decoding: In the spatiotemporal decoding phase, the spatiotemporal features to be decoded are subjected to spatiotemporal decoding operations. This step aims to convert the spatiotemporal features to be decoded back to the original spatiotemporal domain or a more understandable representation. The spatiotemporal decoding operation may involve a series of steps such as deconvolution, upsampling, or recurrent neural networks, depending on the architecture of the model.

[0124] Difference operation: Difference operation is performed on the decoded spatiotemporal features and the spatiotemporal attention features. This step aims to extract the difference information in the decoded spatiotemporal features relative to the spatiotemporal attention features, which may contain spatiotemporal features that are not fully captured in the target input. The feature map obtained after the difference operation is called the spatiotemporal decoded feature, which will be used in subsequent classification, regression or other tasks.

[0125] Among them, as the sequence length increases, the number of hidden states transferred from the hidden state of the encoder to the corresponding hidden state of the decoder also increases. This leads to the long-term dependency problem. In order to solve this problem, the present invention proposes a spatiotemporal attention (ST-Attention) mechanism, which directly transfers the information of the corresponding hidden state between the encoder and the decoder without being restricted by the sequence length. The ST-Attention mechanism extracts the ST-Attention graph to represent the features of the encoder's hidden state, and then reuses the ST-Attention graph when the decoder generates the corresponding hidden state. Since the ST-Attention graph acts as a bridge between the encoder and the decoder, the proposed ST-Attention mechanism can effectively solve the long-term dependency problem.

[0126] ST-Attention mechanism extracts the influence of the next hidden state E t+i E t To this end, we define the spatial features applied to the hidden state E t ST-Attention graph Updated as follows:

[0127] A t =tanh(W A E t )

[0128] in, : represents the weight matrix of the convolution operation, and Nk represents the kernel size of the convolution. The hidden state Et with Nc channels is converted to the ST-Attention map At with 1 channel. At is replicated channel by channel to match the number of channels of Et. The replicated ST-Attention map Applied to the hidden state Et as follows:

[0129]

[0130] In the decoder, the ST-Attention map At obtained from the encoder is applied to the hidden state in the opposite way. Subtracting Dt from get:

[0131]

[0132] The present invention reuses At obtained from the encoder, so At acts as a shortcut path between the encoder and the decoder. The operation of reversing the attention map in the encoder and the decoder is to use the last hidden state ET of the encoder as the initial hidden state Dt of the decoder. The last hidden state Et is a feature that contains global information of the input sequence. As the length of the sequence increases, it becomes more difficult for a hidden state of a fixed size to encode the information of the input sequence due to the increase in context. If the ST-Attention mechanism is applied, the local information of each input can be effectively transmitted through the ST-Attention graph, which enables the decoder to directly access the corresponding ST-Attention graph when generating each output. Since the local information is captured in the ST-Attention graph, the global information can be densely captured in the last hidden state Et of the encoder.

[0133] Through the above steps, the spatiotemporal attention mechanism successfully introduces the ability to capture and enhance key spatiotemporal information in the spatiotemporal encoder-decoder architecture. This mechanism extracts key features through convolution operations, copies the number of channels that match the target input in a channel-by-channel manner, adds the spatiotemporal attention features to the target input through summation operations, and extracts spatiotemporal decoding features through spatiotemporal decoding and difference operations. These steps together constitute the core implementation technology principles of the spatiotemporal attention mechanism, providing the quality inspection model with powerful spatiotemporal feature capture and parsing capabilities.

[0134] In a possible implementation, the selecting, in a probabilistic manner, a target input for calculation at the current time step from a generated output value and an actual output value at a previous time step includes:

[0135] In step B1, a sampling mask for the real input and the generated input at this time step is determined according to a probability factor, wherein the probability factor is reduced and updated according to a preset cooling factor before each determination of the sampling mask;

[0136] In step B2, according to the sampling mask, the target input for calculation in the current time step is selected from the generated output value and the actual output value of the previous time step.

[0137] The ST-Encoder-Decoder part of CRRN adopts the proposed ST-Encoder and ST-Decoder. For ease of reading, Indicates the hidden state of ST-Encoder and ST-Decoder.

[0138] In ST-Encoder, the hidden state Is updated as follows:

[0139]

[0140] Hidden state at the last time step Copied to hidden state Hidden states in ST-Decoder Updated as follows:

[0141]

[0142] Among them, the top hidden state Represented as X t To train the ST-Encoder-Decoder model, a planned sampling technique is used to probabilistically select the generated output Xt and the actual value Xt of the previous time step. The planned sampling formula is as follows:

[0143]

[0144] in, is the input of the model, Xt and Xt are the real input and generated input respectively, TF is the sampling mask for selecting Xt and Xt, following the Bernoulli distribution P(TF) = TF(1-TF) 1-TF , where p is the probability of selecting Xt. In the early stages of learning, p is set high and then gradually reduced during training. Planned sampling techniques can address the exposure bias problem by guiding the learning process to be similar to the testing process.

[0145] In addition, a denoising autoencoder technique is applied to prevent the CRRN from behaving like an identity function. During the training phase, random noise is added to the input and some input elements are set to zero. The denoising autoencoder enhances the anti-interference ability of the original data.

[0146] In a possible implementation, the high-dimensional feature map of the sample solder paste printed board image is converted into a low-dimensional feature map through a convolution layer, the covariance shift is prevented by normalizing the value of each layer through a normalization layer, and the nonlinear enhancement modeling capability is introduced through an activation layer to obtain a low-dimensional spatial encoding feature, including:

[0147] In step C1, a plurality of convolution kernels with different convolution kernel sizes are slid on the sample solder paste printing board image to perform weighted summation of local areas, and local features of the sample solder paste printing board image are respectively extracted to obtain a low-dimensional feature map, wherein the convolution kernels with different convolution kernel sizes respectively capture the features of the sample solder paste printing board image in terms of edge, texture, and color through different convolution kernels;

[0148] Convolution operation: Use multiple convolution kernels of different sizes to slide on the sample solder paste printed board image and perform weighted summation of the local area. This step aims to extract local features in the image, which may include information such as edges, textures, and colors. Convolution kernels of different sizes can capture features of different scales. Smaller convolution kernels can usually capture detailed features such as edges and textures, while larger convolution kernels can capture more global features such as color and shape.

[0149] Low-dimensional feature maps: After the convolution operation, a series of low-dimensional feature maps are obtained. These feature maps retain the key information in the image but reduce the dimension of the data, thereby reducing the amount of computation and storage requirements.

[0150] In step C2, the normalization layer calculates the mean and variance of each feature on each input batch for the low-dimensional feature map of each batch input, and normalizes each input batch so that the difference between the mean and 0 satisfies the first preset condition and the difference between the variance and 1 satisfies the second preset condition through a learnable scaling factor and an offset, thereby obtaining a normalized output to stabilize the input distribution of each layer and prevent covariance shift;

[0151] Mean and variance calculation: The normalization layer calculates the mean and variance of each feature on each input batch for the low-dimensional feature map of each batch input. This step aims to evaluate the distribution of the current batch data.

[0152] Normalization: Through learnable scaling factors and offsets, the difference between the mean and 0 on each input batch satisfies the first preset condition (such as close to 0), and the difference between the variance and 1 satisfies the second preset condition (such as close to 1). This step is designed to normalize the distribution of input data to make it more stable. The normalization operation helps prevent covariance shift, that is, as the number of network layers increases, the distribution of input data may change, resulting in a decrease in model performance.

[0153] Normalized output: After the normalization operation, the normalized output is obtained. This output will be used as the input of the subsequent layers to ensure that the input distribution of each layer is relatively stable.

[0154] In step C3, the negative values ​​in the normalized output are set to 0 through the activation layer through a nonlinear activation function, and the positive values ​​remain unchanged, thereby introducing nonlinear enhanced modeling capabilities to obtain the low-dimensional spatial encoding features.

[0155] Non-linear activation function: The activation layer uses a non-linear activation function (such as ReLU) to set negative values ​​in the normalized output to 0, while keeping positive values ​​unchanged. This step is designed to introduce non-linearity to enhance the modeling ability of the model. Non-linear activation functions enable the model to learn more complex feature representations and improve the generalization performance of the model.

[0156] Spatial encoding features: After being processed by nonlinear activation functions, low-dimensional spatial encoding features are obtained. These features not only retain the key information in the image, but also introduce nonlinearity, so that the model can better capture the complex features in the solder paste printed board image.

[0157] Through the above steps, the feature extraction and spatial encoding of the solder paste printed board image are realized. The multi-scale convolution kernel is used to extract local features in the image, the normalization layer is used to stabilize the input distribution and prevent covariance shift, and the activation layer introduces nonlinearity through nonlinear activation functions to enhance the modeling ability of the model.

[0158] In a possible implementation, the decoding and reconstructing of the low-dimensional features to be decoded into a reconstructed sample image in a high-dimensional space includes:

[0159] In step D1, the feature to be decoded is input into the first hidden layer through a plurality of neurons in the input layer;

[0160] Input layer: The input layer consists of multiple neurons, each of which is responsible for receiving one or more features to be decoded. These features are usually low-dimensional representations extracted from the encoder part, containing key information of the solder paste printed board image.

[0161] Feature transfer: The neurons in the input layer pass the received features to be decoded to the first hidden layer. This step is the starting point for the conversion of features from low-dimensional space to high-dimensional space.

[0162] In step D2, the neurons in the multiple hidden layers perform weighted summation operations on the reconstructed features output by the neurons in the previous layer to obtain the reconstructed features output by the neurons in the current layer, wherein each neuron in each hidden layer is connected to multiple neurons in the previous hidden layer and to multiple neurons in the next hidden layer;

[0163] Weighted summation: In a multi-layer hidden layer, each neuron performs a weighted summation operation on the reconstructed features output by the neurons in the previous layer. This step aims to generate the input of the current neuron by linearly combining the output of the neurons in the previous layer.

[0164] Connection mode: Each neuron in each hidden layer is connected to multiple neurons in the previous hidden layer, and also to multiple neurons in the next hidden layer. This fully connected mode ensures the flow and transmission of information in the neural network.

[0165] Feature reconstruction: Through weighted summation and activation functions (such as ReLU, sigmoid, etc.), each hidden layer will output new reconstructed features. These features gradually approach the high-dimensional representation of the original image during the layer-by-layer transmission process.

[0166] In step D3, the reconstructed features output by multiple neurons in the last hidden layer are weighted and summed through the output layer to reconstruct a reconstructed sample image in a high-dimensional space.

[0167] Weighted summation: In the output layer, the reconstructed features output by multiple neurons in the last hidden layer are weighted summed. This step is the end point of the feature reconstruction process, which aims to combine high-dimensional features into a complete image.

[0168] Image generation: The result of weighted summation is output through the neurons of the output layer to form the pixel values ​​of the reconstructed sample image. These pixel values ​​are combined together to form the reconstructed sample image in the high-dimensional space.

[0169] Image quality: The quality of the reconstructed sample image depends on the structure and training effect of the neural network decoder. By optimizing the network parameters and training process, the accuracy and clarity of the reconstructed image can be improved.

[0170] Through the above steps, the neural network decoder successfully reconstructs the low-dimensional features to be decoded into a reconstructed sample image in a high-dimensional space. The input layer is responsible for receiving features, and the multi-layer hidden layer gradually approaches the high-dimensional representation of the original image through weighted summation and activation functions, while the output layer is responsible for combining high-dimensional features into a complete image. In this process, the fully connected mode ensures the effective transmission of information and the accuracy of reconstructed features.

[0171] The present disclosure also provides a solder paste printing board quality detection device, see Figure 2 As shown, including:

[0172] An acquisition module 210 is configured to acquire an image of a solder paste printed board to be inspected;

[0173] The input module 220 is configured to input the image of the solder paste printed board to be inspected into a pre-trained quality inspection model to obtain the features to be identified output by the quality inspection model, wherein the spatial encoder of the quality inspection model converts the high-dimensional feature map of the sample solder paste printed board image into a low-dimensional feature map through a convolution layer, prevents covariance shift by normalizing the values ​​of each layer through a normalization layer, and obtains low-dimensional spatial coding features by introducing nonlinear enhancement modeling capabilities through an activation layer. The spatiotemporal encoder-decoder of the quality inspection model performs temporal flow-based modeling on the obtained low-dimensional spatial coding features through an input gate, a forget gate, and an output gate. and the spatiotemporal pattern capture operation to obtain the generated output value and the actual output value of each time step, and select the target input for calculation in this time step from the generated output value and the actual output value of the previous time step in a probabilistic manner, and determine the spatiotemporal attention feature, and add the spatiotemporal attention feature to the target input in the spatiotemporal encoding stage to obtain the spatiotemporal feature to be decoded, and subtract the spatiotemporal attention feature from the decoded spatiotemporal feature obtained by decoding in the spatiotemporal decoding stage to obtain the spatiotemporal decoding feature; the spatial decoder of the quality detection model decodes the low-dimensional features to be decoded into a reconstructed sample image in a high-dimensional space;

[0174] The detection and identification module 230 is configured to compare the reconstructed solder paste printed board image with the image of the solder paste printed board to be detected to obtain a reconstruction error, and obtain a solder paste printed board quality detection result based on the reconstruction error.

[0175] In a possible implementation, the input module 220 is configured as follows:

[0176] The input gate performs spatiotemporal pattern capture through a unit gate that performs spatiotemporal pattern capture at the same time step of the previous layer to obtain a first feature, and performs spatiotemporal pattern capture through a unit gate that performs spatiotemporal pattern capture at the previous time step of the current layer to obtain a second feature, and performs time flow information capture through a unit gate that performs time flow capture at the previous time step of the current layer to obtain a third feature;

[0177] Determine a first spatiotemporal coding feature corresponding to the input gate according to the first feature, the second feature, the third feature and corresponding weights;

[0178] The forget gate captures the spatiotemporal pattern through the unit gate that captures the spatiotemporal pattern at the same time step of the previous layer to obtain the fourth feature, and captures the spatiotemporal pattern through the unit gate that captures the spatiotemporal pattern at the previous time step of the current layer to obtain the fifth feature, and captures the time flow information through the unit gate that captures the time flow at the previous time step of the current layer to obtain the sixth feature;

[0179] Determine a second spatiotemporal coding feature corresponding to the input gate according to the third feature, the fourth feature, the fifth feature and the corresponding weights;

[0180] The output gate captures the spatiotemporal pattern through the unit gate that captures the spatiotemporal pattern in the previous time step of the current layer, thereby obtaining the seventh feature and the eighth feature, and captures the time flow information through the unit gate that captures the time flow in the current time step, thereby obtaining the ninth feature;

[0181] Determine the third spatiotemporal coding feature corresponding to the input gate according to the seventh feature, the eighth feature, the ninth feature and the corresponding weights, obtain the generated output value of each time step according to the third spatiotemporal coding feature, and match the actual output value corresponding to the sample solder paste printing board image with the generated output value based on the time step;

[0182] Among them, the unit gate for capturing spatiotemporal patterns is determined based on the unit gate and output gate for capturing time flow, and the unit gate for capturing time flow is determined based on the forget gate, input gate, and the weight matrix of the convolution operation for reducing the number of channels to half.

[0183] In a possible implementation, the input module 220 is configured to perform time flow and spatiotemporal pattern capture operations on the low-dimensional spatial encoding features through the following formula to obtain a generated output value for each time step:

[0184]

[0185]

[0186]

[0187]

[0188]

[0189] Where l and t represent the neuron layer and time step, respectively. They represent the input gate, forget gate and output gate respectively. is the unit gate for updating the unit state of the lth layer based on the time flow in the tth time step, is the unit gate for updating the unit state of the lth layer based on the time flow in the t-1th time step, is the unit gate of the l-1th layer in the tth time step that updates the unit state based on the time flow, W i , U i , W c , W f , U f , Wf , W O , U O , W O are weights, is the hidden state of the l-1th layer in the t-1th time step, is the hidden state of the l-1th layer in the tth time step, is the hidden state of the lth layer in the tth time step, The candidate information generated by the lth layer in the tth time step is generated by the weighted sum of the target input at the tth time step and the hidden state at the t-1th time step, and then generated by the tanh activation function. is a 1×1 convolution kernel, N c is the number of channels of the feature, σ is the sigmoid activation function, which is used to map the result to between 0 and 1, ο is the Hadamard product operation, and tanh is the hyperbolic tangent activation function, which is used to map the result to between -1 and 1. For the concatenation operation, the unit state of the lth layer at the t-1th time step and the unit state of the l-1th layer at the tth time step are concatenated to simultaneously consider the spatiotemporal relationship between the previous and next layers.

[0190] In a possible implementation, the input module 220 is configured as follows:

[0191] According to the weight matrix of the convolution operation and the convolution kernel size of the convolution operation, the hidden state with any channel is converted into a first attention feature map with a preset channel;

[0192] Copying the first attention feature map in a channel manner to obtain the spatiotemporal attention feature, wherein the channel manner copying makes the spatiotemporal attention feature match the number of channels of the target input;

[0193] In the spatiotemporal encoding stage, the target input and the spatiotemporal attention feature are summed to add the spatiotemporal attention feature to the target input to obtain a spatiotemporal feature to be decoded;

[0194] In the spatiotemporal decoding stage, the spatiotemporal features to be decoded are spatiotemporally decoded to obtain decoded spatiotemporal features, and the decoded spatiotemporal features are subtracted from the spatiotemporal attention features to obtain the spatiotemporal decoding features.

[0195] In a possible implementation, the input module 220 is configured as follows:

[0196] Determine a sampling mask for the real input and the generated input at this time step according to the probability factor, wherein the probability factor is reduced and updated according to a preset cooling factor before each determination of the sampling mask;

[0197] According to the sampling mask, the target input for calculation in the current time step is selected from the generated output value and the actual output value of the previous time step.

[0198] In a possible implementation, the input module 220 is configured as follows:

[0199] The convolution kernels with different convolution kernel sizes are slid on the sample solder paste printing board image to perform weighted summation of the local area, and the local features of the sample solder paste printing board image are respectively extracted to obtain a low-dimensional feature map, wherein the convolution kernels with different convolution kernel sizes respectively capture the features of the sample solder paste printing board image in terms of edge, texture, and color through different convolution kernels;

[0200] The normalization layer calculates the mean and variance of each feature on each input batch for the low-dimensional feature map of each batch input, and normalizes each input batch so that the difference between the mean and 0 meets the first preset condition, and the difference between the variance and 1 meets the second preset condition through a learnable scaling factor and offset, thereby obtaining a normalized output to stabilize the input distribution of each layer and prevent covariance shift;

[0201] The negative values ​​in the normalized output are set to 0 through the activation layer through a nonlinear activation function, while the positive values ​​remain unchanged, thereby introducing nonlinear enhanced modeling capabilities and obtaining the low-dimensional spatial encoding features.

[0202] In a possible implementation, the input module 220 is configured as follows:

[0203] Inputting the to-be-decoded features into the first hidden layer through a plurality of neurons in the input layer;

[0204] The neurons in the multiple hidden layers perform weighted sum operations on the reconstructed features output by the neurons in the previous layer to obtain the reconstructed features output by the neurons in this layer, wherein each neuron in each hidden layer is connected to multiple neurons in the previous hidden layer and to multiple neurons in the next hidden layer;

[0205] The reconstructed features output by multiple neurons in the last hidden layer are weighted and summed through the output layer to reconstruct a reconstructed sample image in a high-dimensional space.

[0206] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the method described in any one of the aforementioned embodiments are implemented.

[0207] The present disclosure also provides an electronic device, including:

[0208] a memory having a computer program stored thereon;

[0209] A processor is used to execute the computer program in the memory to implement the steps of the method in any one of the aforementioned embodiments.

[0210] Figure 3 The solder paste printing board quality detection device 100 shown includes: a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, such as through a bus 1002. Optionally, the solder paste printing board quality detection device 100 may also include a communication component, which can be used for data interaction between the device 100 and other devices, such as data transmission and / or data reception. It should be noted that in actual scheduling, the communication component is not limited to one, and the structure of the solder paste printing board quality detection device 100 does not constitute a limitation on the embodiments of the present application.

[0211] Processor 1001 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 1001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0212] The bus 1002 may include a path to transmit information between the above components. The bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 1002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0213] The memory 1003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage medium, other magnetic storage devices, or any other medium that can be used to carry or store program code and can be read by a computer, without limitation herein.

[0214] The memory 1003 is used to store program codes for executing the embodiments of the present disclosure, and the execution is controlled by the processor 1001. The processor 1001 is used to execute the program codes stored in the memory 1003 to implement the steps shown in the above-mentioned solder paste printing board quality detection method embodiment.

[0215] The embodiment of the present disclosure also provides a computer-readable storage medium having program code stored thereon. When the program code is executed by a processor, the steps and corresponding contents of the aforementioned solder paste printing board quality detection method embodiment can be implemented.

[0216] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments; within the technical concept of the present disclosure, various changes, modifications, substitutions and variations may be made to these embodiments, and these changes, modifications, substitutions and variations all fall within the protection scope of the present disclosure.

[0217] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction, and they should also be regarded as the contents disclosed in this disclosure. In order to avoid unnecessary repetition, this disclosure will not further describe various possible combinations. The technical scope of this application is not limited to the contents in the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A method for detecting the quality of a solder paste printed board, characterized in that: The method comprises: Acquire the image of the solder paste printed board to be inspected; The image of the solder paste printed board to be inspected is input into a pre-trained quality inspection model to obtain a reconstructed solder paste printed board image output by the quality inspection model, wherein, during the training process, the spatial encoder of the quality inspection model converts the high-dimensional feature map of the sample solder paste printed board image into a low-dimensional feature map through a convolution layer, prevents covariance shift by normalizing the value of each layer through a normalization layer, and obtains low-dimensional spatial coding features by introducing nonlinear enhancement modeling capabilities through an activation layer. The spatiotemporal encoder-decoder of the quality inspection model performs temporal flow and spatiotemporal modeling on the low-dimensional spatial coding features through an input gate, a forget gate, and an output gate. A pattern capture operation is performed to obtain a generated output value and an actual output value at each time step, and a target input for calculation at this time step is selected from the generated output value and the actual output value of the previous time step in a probabilistic manner, and the spatiotemporal attention feature is determined, and the spatiotemporal attention feature is added to the target input in the spatiotemporal encoding stage to obtain a spatiotemporal feature to be decoded, and the spatiotemporal attention feature is subtracted from the decoded spatiotemporal feature obtained by decoding in the spatiotemporal decoding stage to obtain a spatiotemporal decoding feature; the spatial decoder of the quality detection model reconstructs the low-dimensional features to be decoded into a reconstructed sample image in a high-dimensional space; The reconstructed solder paste printed board image is compared with the image of the solder paste printed board to be inspected to obtain a reconstruction error, and a quality inspection result of the solder paste printed board is obtained based on the reconstruction error.

2. The solder paste printing board quality detection method according to claim 1, characterized in that: The input gate, the forget gate, and the output gate are used to perform time flow and spatiotemporal pattern capture operations on the low-dimensional spatial encoding features to obtain the generated output value and the actual output value of each time step, including: The input gate performs spatiotemporal pattern capture through a unit gate that performs spatiotemporal pattern capture at the same time step of the previous layer to obtain a first feature, and performs spatiotemporal pattern capture through a unit gate that performs spatiotemporal pattern capture at the previous time step of the current layer to obtain a second feature, and performs time flow information capture through a unit gate that performs time flow capture at the previous time step of the current layer to obtain a third feature; Determine a first spatiotemporal coding feature corresponding to the input gate according to the first feature, the second feature, the third feature and corresponding weights; The forget gate captures the spatiotemporal pattern through the unit gate that captures the spatiotemporal pattern at the same time step of the previous layer to obtain the fourth feature, and captures the spatiotemporal pattern through the unit gate that captures the spatiotemporal pattern at the previous time step of the current layer to obtain the fifth feature, and captures the time flow information through the unit gate that captures the time flow at the previous time step of the current layer to obtain the sixth feature; Determine a second spatiotemporal coding feature corresponding to the input gate according to the third feature, the fourth feature, the fifth feature and the corresponding weights; The output gate captures the spatiotemporal pattern through the unit gate that captures the spatiotemporal pattern in the previous time step of the current layer, thereby obtaining the seventh feature and the eighth feature, and captures the time flow information through the unit gate that captures the time flow in the current time step, thereby obtaining the ninth feature; Determine the third spatiotemporal coding feature corresponding to the input gate according to the seventh feature, the eighth feature, the ninth feature and the corresponding weights, obtain the generated output value of each time step according to the third spatiotemporal coding feature, and match the actual output value corresponding to the sample solder paste printing board image with the generated output value based on the time step; Among them, the unit gate for capturing spatiotemporal patterns is determined based on the unit gate and output gate for capturing time flow, and the unit gate for capturing time flow is determined based on the forget gate, input gate, and the weight matrix of the convolution operation for reducing the number of channels to half.

3. The solder paste printing board quality detection method according to claim 2, characterized in that: The low-dimensional spatial encoding features are operated based on time flow and spatiotemporal pattern capture through the following formula to obtain the generated output value of each time step: Where l and t represent the neuron layer and time step, respectively. They represent the input gate, forget gate and output gate respectively. is the unit gate for updating the unit state of the lth layer based on the time flow in the tth time step, is the unit gate for updating the unit state of the lth layer based on the time flow in the t-1th time step, is the unit gate of the l-1th layer in the tth time step that updates the unit state based on the time flow, W i , U i , W c , W f , U f , W f , W O , U O , W O are weights, is the hidden state of the l-1th layer in the t-1th time step, is the hidden state of the l-1th layer in the tth time step, is the hidden state of the lth layer in the tth time step, The candidate information generated by the lth layer in the tth time step is generated by the weighted sum of the target input at the tth time step and the hidden state at the t-1th time step, and then generated by the tanh activation function. is a 1×1 convolution kernel, N c is the number of channels of the feature, σ is the sigmoid activation function, which is used to map the result to between 0 and 1, ο is the Hadamard product operation, and tanh is the hyperbolic tangent activation function, which is used to map the result to between -1 and 1. For the concatenation operation, the unit state of the lth layer at the t-1th time step and the unit state of the l-1th layer at the tth time step are concatenated to simultaneously consider the spatiotemporal relationship between the previous and next layers.

4. The method for detecting quality of solder paste printed boards according to claim 1, characterized in that: The determining of the spatiotemporal attention features, adding the spatiotemporal attention features to the target input in the spatiotemporal encoding stage to obtain the spatiotemporal features to be decoded, and subtracting the spatiotemporal attention features from the decoded spatiotemporal features obtained by decoding in the spatiotemporal decoding stage to obtain the spatiotemporal decoding features, includes: According to the weight matrix of the convolution operation and the convolution kernel size of the convolution operation, the hidden state with any channel is converted into a first attention feature map with a preset channel; Copying the first attention feature map in a channel manner to obtain the spatiotemporal attention feature, wherein the channel manner copying makes the spatiotemporal attention feature match the number of channels of the target input; In the spatiotemporal encoding stage, the target input and the spatiotemporal attention feature are summed to add the spatiotemporal attention feature to the target input to obtain a spatiotemporal feature to be decoded; In the spatiotemporal decoding stage, the spatiotemporal features to be decoded are spatiotemporally decoded to obtain decoded spatiotemporal features, and the decoded spatiotemporal features are subtracted from the spatiotemporal attention features to obtain the spatiotemporal decoding features.

5. The method for detecting quality of solder paste printed boards according to claim 1, characterized in that: The selecting the target input for calculation in the current time step from the generated output value and the actual output value of the previous time step in a probabilistic manner includes: Determine a sampling mask for the real input and the generated input at this time step according to the probability factor, wherein the probability factor is reduced and updated according to a preset cooling factor before each determination of the sampling mask; According to the sampling mask, the target input for calculation in the current time step is selected from the generated output value and the actual output value of the previous time step.

6. The method for detecting quality of a solder paste printed board according to any one of claims 1 to 5, characterized in that: The high-dimensional feature map of the sample solder paste printed board image is converted into a low-dimensional feature map through the convolution layer, the covariance shift is prevented by normalizing the value of each layer through the normalization layer, and the nonlinear enhancement modeling capability is introduced through the activation layer to obtain low-dimensional spatial encoding features, including: The convolution kernels with different convolution kernel sizes are slid on the sample solder paste printing board image to perform weighted summation of the local area, and the local features of the sample solder paste printing board image are respectively extracted to obtain a low-dimensional feature map, wherein the convolution kernels with different convolution kernel sizes respectively capture the features of the sample solder paste printing board image in terms of edge, texture, and color through different convolution kernels; The normalization layer calculates the mean and variance of each feature on each input batch for the low-dimensional feature map of each batch input, and normalizes each input batch so that the difference between the mean and 0 meets the first preset condition, and the difference between the variance and 1 meets the second preset condition through a learnable scaling factor and offset, thereby obtaining a normalized output to stabilize the input distribution of each layer and prevent covariance shift; The negative values ​​in the normalized output are set to 0 through the activation layer through a nonlinear activation function, while the positive values ​​remain unchanged, thereby introducing nonlinear enhanced modeling capabilities and obtaining the low-dimensional spatial encoding features.

7. The method for detecting quality of solder paste printed boards according to any one of claims 1 to 5, characterized in that: The reconstructing the low-dimensional features to be decoded into a reconstructed sample image in a high-dimensional space by decoding and reconstructing the features to be decoded includes: Inputting the to-be-decoded features into the first hidden layer through a plurality of neurons in the input layer; The neurons in the multiple hidden layers perform weighted sum operations on the reconstructed features output by the neurons in the previous layer to obtain the reconstructed features output by the neurons in this layer, wherein each neuron in each hidden layer is connected to multiple neurons in the previous hidden layer and to multiple neurons in the next hidden layer; The reconstructed features output by multiple neurons in the last hidden layer are weighted and summed through the output layer to reconstruct a reconstructed sample image in a high-dimensional space.

8. A solder paste printing board quality detection device, characterized in that: include: An acquisition module is configured to acquire an image of a solder paste printed board to be inspected; The input module is configured to input the image of the solder paste printed board to be inspected into a pre-trained quality inspection model to obtain a reconstructed solder paste printed board image output by the quality inspection model, wherein, during the training process, the spatial encoder of the quality inspection model converts the high-dimensional feature map of the sample solder paste printed board image into a low-dimensional feature map through a convolution layer, prevents covariance shift by normalizing the value of each layer through a normalization layer, and obtains low-dimensional spatial coding features by introducing nonlinear enhancement modeling capabilities through an activation layer. The spatiotemporal encoder-decoder of the quality inspection model performs time-based spatial coding on the low-dimensional spatial coding features through an input gate, a forget gate, and an output gate. The flow and spatiotemporal pattern capture operation is performed to obtain the generated output value and the actual output value of each time step, and the target input for calculation in this time step is selected from the generated output value and the actual output value of the previous time step in a probabilistic manner, and the spatiotemporal attention feature is determined, and the spatiotemporal attention feature is added to the target input in the spatiotemporal encoding stage to obtain the spatiotemporal feature to be decoded, and the spatiotemporal attention feature is subtracted from the decoded spatiotemporal feature obtained by decoding in the spatiotemporal decoding stage to obtain the spatiotemporal decoding feature; the spatial decoder of the quality detection model reconstructs the reconstructed sample image of the high-dimensional space by decoding the low-dimensional features to be decoded; The detection and identification module is configured to compare the reconstructed solder paste printed board image with the image of the solder paste printed board to be detected to obtain a reconstruction error, and obtain a quality detection result of the solder paste printed board based on the reconstruction error.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 7.

Images