A smt processing method of fusing text and defect images

Abstract

The application provides an SMT processing method for fusing text and defect images, comprising: obtaining AOI detection text and SMT defect images, extracting semantic features of the AOI detection text; extracting visual features of the SMT defect images, fusing the semantic features and the visual features to obtain fused features; constructing a correlation matrix between defect types and process parameters in the form of a knowledge graph, updating the correlation matrix, constructing a loss function according to the correlation matrix, updating a current process parameter vector to obtain an intermediate process parameter vector; calculating a recommended process parameter vector based on the current process parameter vector, and adjusting the SMT processing process. Based on the fusion of semantic features and visual features, the historical data of SMT processing can be fully utilized to screen out the intermediate process parameter vector, the recommended process parameter vector with high accuracy can be calculated by using the intermediate process parameter vector, and the SMT processing process can be adjusted by using the recommended process parameter vector, thereby reducing the probability of defects in SMT processing.

Description

Technical Field

[0001] This invention relates to the field of SMT processing technology, and in particular to an SMT processing method that integrates text and defect images. Background Technology

[0002] In modern electronics manufacturing, automated optical inspection (AOI) is an important quality inspection method widely used in SMT (Surface Mount Technology) production processes for defect detection. However, AOI inspection reports typically contain a large amount of text descriptions and image data, which are crucial for defect analysis and process optimization. Traditional analysis methods rely on manual interpretation, which is inefficient and prone to errors.

[0003] Existing defect analysis methods typically only process based on a single modality, lacking deep fusion of cross-modal information, and most systems cannot effectively combine textual descriptions in inspection reports with image data for comprehensive analysis. Furthermore, although a large amount of historical data is accumulated during SMT processing, existing systems cannot fully utilize this historical data to optimize process parameters.

[0004] Therefore, there is a need for a method that, by integrating semantic and visual features, can fully utilize historical data from SMT processing to calculate a highly accurate recommended process parameter vector. This recommended process parameter vector can then be used to adjust the SMT processing technology and reduce the probability of defects during SMT processing. Summary of the Invention

[0005] To overcome the problems existing in related technologies, the purpose of this invention is to provide an SMT processing method that integrates text and defect images. Based on the integration of semantic features and visual features, this method can make full use of historical data of SMT processing to calculate a highly accurate recommended process parameter vector. The recommended process parameter vector is then used to adjust the SMT processing process, thereby reducing the probability of defects occurring during SMT processing.

[0006] A surface mount technology (SMT) processing method that integrates text and defect images includes:

[0007] Acquire AOI-detected text and SMT defect images;

[0008] Extract the semantic features of the AOI-detected text;

[0009] Extract the visual features of the SMT defect image;

[0010] The semantic features and the visual features are fused in a multimodal manner to obtain fused features; the defect type is determined based on the fused features.

[0011] A knowledge graph is used to construct an association matrix between defect types and process parameters;

[0012] Update the correlation matrix;

[0013] A loss function is constructed based on the correlation matrix, and the current process parameter vector is updated using the loss function to obtain the intermediate process parameter vector.

[0014] Based on the intermediate process parameter vector, a recommended process parameter vector is calculated, and the SMT processing technology is adjusted using the recommended process parameter vector.

[0015] In a preferred embodiment of the present invention, the extraction of semantic features from the AOI-detected text includes:

[0016] A large language model is used to encode the AOI detection text to obtain initial encoded features;

[0017] The initial encoded features are iteratively updated to obtain the final encoded features;

[0018] The final encoded features are subjected to a nonlinear transformation to obtain the transformed encoded features;

[0019] By introducing classification markers into the transformed encoded features, the semantic features of the AOI detected text are obtained.

[0020] In a preferred embodiment of the present invention, the step of iteratively updating the initial coding features to obtain the final coding features includes:

[0021] The initial encoded features are subjected to layer normalization to obtain the first normalized feature;

[0022] Multi-head attention calculation is performed on the first normalized feature to obtain the first attention matrix;

[0023] The first attention matrix is ​​added to the initial encoded features to obtain the second encoded features;

[0024] The second encoded feature is subjected to layer normalization, multi-head attention calculation, and matrix summation to obtain the final encoded feature.

[0025] In a preferred embodiment of the present invention, the extraction of visual features from the SMT defect image includes:

[0026] The defect location and defect morphology of the SMT defect image are extracted using the ResNet model. The defect location and defect morphology constitute the visual features of the SMT defect image.

[0027] In a preferred embodiment of the present invention, the step of multimodal fusion of the semantic features and the visual features to obtain fused features includes:

[0028] The semantic features and the visual features are fused according to the following formula:

[0029] F=α.MLP(T)+(1-α).V+β.(MLP(T)⊙V);

[0030] Where F is the fusion feature, α is the first fusion weight, β is the second fusion weight, MLP is a multilayer perceptron used to align the feature dimensions of the semantic feature and the visual feature; T is the semantic feature, V is the visual feature, and ⊙ is element-wise multiplication.

[0031] In a preferred embodiment of the present invention, a correlation matrix between defect types and process parameters is constructed in the form of a knowledge graph, including:

[0032] Construct the defect type set and process parameter set according to the following formulas:

[0033] De = {d1, d2, ..., d} M};

[0034] Pe = {p1, p2, ..., p} N};

[0035] Where De is the set of defect types, Pe is the set of process parameters; d1 is the first defect type, d2 is the second defect type, and d M This represents the Mth defect type; p1 is the first process parameter, p2 is the second process parameter, and p... N This is the Nth process parameter;

[0036] The mapping relationship between the elements of the defect type set and the elements of the process parameter set is constructed based on the knowledge graph;

[0037] Based on the mapping relationship, an association matrix is ​​constructed between defect types and process parameters; the element W in the i-th row and j-th column of the association matrix... i,j Used to represent the i-th defect d i and the j-th process parameter p j The strength of the correlation between them.

[0038] In a preferred embodiment of the present invention, updating the correlation matrix includes:

[0039] The update matrix is ​​calculated based on the correlation between the current defect type and process parameters;

[0040] The (t+1)th correlation matrix is ​​calculated using the update matrix and the t-th correlation matrix according to the following formula:

[0041] W t+1 =Wt +η.ΔW;

[0042] ΔW i,j =var(d i ).var(p j )+∈cov(d i ,p j );

[0043] Among them, W t+1 Let W be the (t+1)th incidence matrix. t Let $\frac{ ... i,j To update the element in the i-th row and j-th column of the matrix, d i For the i-th defect type, p j For the j-th process parameter, var(d i ) represents the variance of the i-th defect type, var(p j Let ) represent the variance of the j-th process parameter, ∈ be the adjustment parameter, and cov(d) be the variance of the j-th process parameter. i ,p j Let be the covariance between the i-th defect type and the j-th process parameter.

[0044] In a preferred embodiment of the present invention, the step of constructing a loss function based on the correlation matrix and updating the current process parameter vector using the loss function to obtain an intermediate process parameter vector includes:

[0045]

[0046] Where Cost represents the loss function, P(d i |Pe) represents the probability of the i-th defect type occurring under the set of process parameters; Let λ represent the severity weight of the i-th defect type, λ be the regularization coefficient, and P0 represent the current process parameter vector. The square of the L2 norm of the difference between the set of process parameters and the current process parameter vector;

[0047] The constraint condition for the j-th process parameter in the set of process parameters is:

[0048] p min ≤p j ≤p max ;

[0049] Where, p min p represents the minimum value of the process parameter. max This indicates the maximum value of the process parameter;

[0050] Under the constraint of the j-th process parameter, search for the minimum value of the loss function, and take the set of process parameters corresponding to the minimum value of the loss function as the intermediate process parameter vector.

[0051] In a preferred embodiment of the present invention, the step of calculating the recommended process parameter vector based on the intermediate process parameter vector includes:

[0052] The recommended process parameter vector is calculated using the following formula:

[0053]

[0054] Where, p rec To recommend the process parameter vector, p middle For intermediate process parameter vectors, Δp j γ is the adjustment amount for the j-th process parameter. j Let be the event weight for the i-th defect type occurring under the j-th process parameter, and N be the total number of process parameters.

[0055] In a preferred embodiment of the present invention, the step of encoding the AOI detection text using a large language model to obtain initial encoded features includes:

[0056] The AOI-detected text is encoded according to the following formula:

[0057] H1=TokenEmbed(X)+PosEmbed(X);

[0058] Where H1 represents the initial encoding features, TokenEmbed(X) represents the word embedding matrix of the word sequence of the AOI detection text, PosEmbed(X) represents the position encoding matrix of the word sequence of the AOI detection text, and X is the word sequence of the AOI detection text.

[0059] The beneficial effects of this invention are as follows:

[0060] The SMT processing method fusion method provided by this invention includes acquiring AOI detection text and SMT defect images. The AOI detection text includes manually judged text descriptions, and the SMT defect images include image data captured by a camera. Semantic features are extracted from the AOI detection text, and visual features are extracted from the SMT defect images. The semantic and visual features are fused using a multimodal approach to obtain fused features; the defect type is determined based on the fused features. A multilayer perceptron is used to capture the data and structure of the semantic features, which can also enhance the nonlinearity of the semantic features. A residual network is used to extract visual features. The residual network uses a residual learning architecture to extract image features. The residual network has a large number of layers, solving the gradient vanishing and gradient exploding problems. A knowledge graph is used to construct an association matrix between defect types and process parameters. The association matrix reflects the correlation between each defect type and all process parameters, demonstrating the influence of different process parameters on each defect type. The association matrix is ​​used to establish the relationship between M defect types and N process parameters. An update matrix is ​​calculated based on the correlation between defect types and process parameters. A loss function is constructed based on the correlation matrix, and the current process parameter vector is updated using the loss function to obtain an intermediate process parameter vector. The intermediate process parameter vector represents the process parameters corresponding to the minimum value of the loss function. A recommended process parameter vector is calculated based on the current process parameter vector, and the SMT processing technology is adjusted using the recommended process parameter vector. The recommended process parameter vector is calculated from the correlation matrix and historical data. Adjusting process parameters such as placement pressure and oven temperature profiles can effectively prevent defects during SMT processing. This invention, by integrating semantic and visual features, can fully utilize historical SMT processing data to filter out intermediate process parameter vectors, and then use these intermediate process parameter vectors to calculate highly accurate recommended process parameter vectors. Using these recommended process parameter vectors to adjust the SMT processing technology reduces the probability of defects during SMT processing. Attached Figure Description

[0061] Figure 1 This is a flowchart of the SMT processing method for fusing text and defect images according to the present invention;

[0062] Figure 2 This is a flowchart of the present invention for extracting semantic features of AOI-detected text;

[0063] Figure 3 This is a flowchart of the iterative update of the initial coding features according to the present invention. Detailed Implementation

[0064] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0065] Example 1

[0066] like Figure 1 As shown, this embodiment provides an SMT processing method that integrates text and defect images, including:

[0067] S1: Obtain AOI-detected text and SMT defect images.

[0068] S2: Extract the semantic features of the AOI-detected text.

[0069] S3: Extract the visual features of the SMT defect image.

[0070] S4: Perform multimodal fusion of the semantic features and the visual features to obtain fused features; determine the defect type based on the fused features.

[0071] S5: Use a knowledge graph to construct the association matrix between defect types and process parameters.

[0072] S6: Update the correlation matrix.

[0073] S7: Construct a loss function based on the correlation matrix, and use the loss function to update the current process parameter vector to obtain the intermediate process parameter vector.

[0074] S8: Calculate a recommended process parameter vector based on the intermediate process parameter vector, and adjust the SMT processing technology using the recommended process parameter vector.

[0075] An AOI inspection report includes AOI inspection text and SMT defect images. The AOI inspection text consists of one or more defect description statements, each of which can be broken down into multiple descriptive words. SMT defect images are obtained by capturing images of the SMT component surface using a camera. The AOI inspection text contains semantic information determined by human judgment, while the SMT defect images contain image data captured by the camera.

[0076] like Figure 2 As shown, the extraction of semantic features from the AOI-detected text includes:

[0077] S21: Use a large language model to encode the AOI detection text to obtain initial encoded features.

[0078] S22: Iteratively update the initial coding features to obtain the final coding features.

[0079] S23: Perform a nonlinear transformation on the final encoded features to obtain the transformed encoded features.

[0080] S24: Introduce classification markers into the transformed encoded features to obtain the semantic features of the AOI detected text.

[0081] The large language model used in this embodiment is GPT-4. GPT-4 supports processing various input formats such as text and images. It also features extensive context support, powerful reasoning and creation capabilities, and multi-turn dialogue capabilities. GPT-4 is used to perform word embedding and positional encoding on the AOI detection text. Token embedding is a method that converts words and symbols in the AOI detection text into vector representations. Since the words and symbols in the AOI detection text are discrete and cannot be directly used for computation, they need to be mapped to a continuous vector space. Positional encoding provides GPT-4 with information about the positions of words or symbols in the input sequence. When processing sequential data such as AOI detection text, positional information is crucial for GPT-4 to understand context and semantics. Positional embedding is a method that encodes positional information into vectors.

[0082] like Figure 3 As shown, the iterative update of the initial encoded features to obtain the final encoded features includes:

[0083] S221: Perform layer normalization on the initial encoded features to obtain the first normalized feature.

[0084] S222: Perform multi-head attention calculation on the first normalized feature to obtain the first attention matrix.

[0085] S223: Add the initial encoding feature to the first attention matrix to obtain the second encoding feature.

[0086] S224: Perform layer normalization, multi-head attention calculation, and matrix summation on the second encoded feature to obtain the final encoded feature.

[0087] Layer normalization is a method of normalizing data within a single layer of each training sample. In steps S221-S224, layer normalization reduces the bias of internal covariates of encoded features by adjusting the mean of each encoded feature to 0 and the variance of each encoded feature to 1. Multi-head attention computation allows the model to capture information in different representation subspaces. Multi-head attention improves the model's expressive power by executing multiple attention operations in parallel. Each attention operation focuses on different aspects of the input sequence, allowing the model to capture richer semantic information.

[0088] The semantic features of the AOI-detected text are extracted using the following formula:

[0089] H1=TokenEmbed(X)+PosEmbed(X);

[0090] H k =MultiHeadAttn(LN(H k-1 ))+H k-1 ;

[0091] H' k =FFN(LN(H k ))+H k ;

[0092] T = CLS(H') k );

[0093] Where H1 represents the initial encoding features, TokenEmbed(X) represents the word embedding matrix of the word sequence in the AOI detection text, PosEmbed(X) represents the position encoding matrix of the word sequence in the AOI detection text, and X is the word sequence of the AOI detection text; H k Let H represent the k-th encoded feature, MultiHeadAttn represent multi-head attention operation, LN represent layer normalization, and H represent the k-th encoded feature. k-1 H' represents the (k-1)th encoded feature; k Let represent the encoded feature after the k-th transformation, FFN represent the feedforward neural network, CLS represent the classification label, and T represent the semantic features of the text detected by AOI.

[0094] If the iteration is performed twice, then T is calculated using H2; if the iteration is performed k-1 times, then H2 is used. kCalculate T, where k ≥ 2. For example, when k = 3, iterate twice. First, perform steps S221-S223 on the initial encoded features, i.e., perform the first iteration, to obtain the second encoded feature. Then, perform layer normalization and multi-head attention calculation on the second encoded feature H2 to obtain k-1 attention matrices, i.e., the second attention matrix. Add the second attention matrix to the second encoded feature to obtain the third encoded feature, which is the final encoded feature. Perform steps S23-S24 on the final encoded feature to obtain the semantic features of the AOI detected text.

[0095] For H k Perform layer normalization, and then use the result of layer normalization, LN(H) k The input is a feedforward neural network, which consists of multiple fully connected layers and typically contains one or more hidden layers. The function of the feedforward neural network is to perform a non-linear transformation on the layer normalization result LN(Hk), enabling the model to learn more complex data features.

[0096] Classification tags capture global semantic information of the entire input sequence and are suitable for classification tasks. The classification tag of this invention is the CLS tag. The CLS tag is a special tag that is usually located at the beginning of the input sequence. The role of the CLS tag is to provide a global contextual representation for the classification task.

[0097] This embodiment provides a method for SMT processing that fuses text and defect images. The method includes acquiring AOI detection text and SMT defect images. The AOI detection text includes manually judged text descriptions, and the SMT defect images include image data captured by a camera. Semantic features are extracted from the AOI detection text, and visual features are extracted from the SMT defect images. The semantic and visual features are fused using a multimodal approach to obtain fused features; the defect type is determined based on these fused features. A multilayer perceptron is used to capture the data and structure of the semantic features, which also enhances the nonlinearity of the semantic features. A residual network is used to extract visual features. The residual network employs a residual learning architecture to extract image features. The residual network has a large number of layers, solving the problems of vanishing and exploding gradients. A knowledge graph is used to construct an association matrix between defect types and process parameters. The association matrix reflects the correlation between each defect type and all process parameters, demonstrating the influence of different process parameters on each defect type. The association matrix is ​​used to establish the relationship between M defect types and N process parameters. An update matrix is ​​calculated based on the correlation between defect types and process parameters. A loss function is constructed based on the correlation matrix, and the current process parameter vector is updated using the loss function to obtain an intermediate process parameter vector. The intermediate process parameter vector represents the process parameters corresponding to the minimum value of the loss function. A recommended process parameter vector is calculated based on the current process parameter vector, and the SMT processing technology is adjusted using the recommended process parameter vector. The recommended process parameter vector is calculated from the correlation matrix and historical data. Adjusting process parameters such as placement pressure and oven temperature profiles can effectively prevent defects during SMT processing. This invention, by integrating semantic and visual features, can fully utilize historical SMT processing data to filter out intermediate process parameter vectors, and then use these intermediate process parameter vectors to calculate highly accurate recommended process parameter vectors. Using these recommended process parameter vectors to adjust the SMT processing technology reduces the probability of defects during SMT processing.

[0098] Example 2

[0099] like Figure 1 As shown, this embodiment provides an SMT processing method that integrates text and defect images. This embodiment describes the differences between it and Embodiment 1. The method includes:

[0100] S1: Obtain AOI-detected text and SMT defect images.

[0101] S2: Extract the semantic features of the AOI-detected text.

[0102] S3: Extract the visual features of the SMT defect image.

[0103] S4: Perform multimodal fusion of the semantic features and the visual features to obtain fused features; determine the defect type based on the fused features.

[0104] S5: Use a knowledge graph to construct the association matrix between defect types and process parameters.

[0105] S6: Update the correlation matrix.

[0106] S7: Construct a loss function based on the correlation matrix, and use the loss function to update the current process parameter vector to obtain the intermediate process parameter vector.

[0107] S8: Calculate a recommended process parameter vector based on the intermediate process parameter vector, and adjust the SMT processing technology using the recommended process parameter vector.

[0108] The extraction of visual features from the SMT defect image includes:

[0109] The ResNet model is used to extract the defect location and morphology of the SMT defect image. The defect location and morphology constitute the visual features of the SMT defect image. The ResNet model addresses the vanishing and exploding gradient problems by introducing a residual learning mechanism. The ResNet model utilizes skip connections to construct the network. For example, if the input is `input` and the output is `H(input)`, the ResNet model learns the residual `residual = H(input) - input`. The input and residual are added together through skip connections to obtain the final output of one module of the ResNet model. The ResNet model consists of multiple such modules.

[0110] The common defect types and corresponding defect locations in the production process of SMT (Surface Mounted Technology) are as follows: (1) Bridging: There is solder adhesion between two or more solder joints that should not be connected, resulting in an electrical short circuit. It usually occurs between pads, especially in areas with dense pins. (2) Too much solder: Too much solder at the solder joint causes solder pile-up, affecting the electrical connection. It mainly occurs on the pads. (3) Reversal: Polarized components do not correspond to the polarized positions on the PCB. It occurs at the positions of polarized components. (4) Solder cracks or breaks: Cracks appear in the soldered parts. It occurs at the solder joints, especially in areas of stress concentration.

[0111] The step of fusing the semantic features and the visual features in a multimodal manner to obtain fused features includes:

[0112] The semantic features and the visual features are fused according to the following formula:

[0113] F=α.MLP(T)+(1-α).Res(V)+β.(MLP(T)⊙Res(V));

[0114] Where F is the fusion feature, α is the first fusion weight, β is the second fusion weight, MLP is a multilayer perceptron used to align the feature dimensions of the semantic feature and the visual feature; T is the semantic feature, V is the visual feature, Res is the ResNet model, and ⊙ is element-wise multiplication.

[0115] In this embodiment, the sum of the first fusion weight and the second fusion weight is less than or equal to 1, i.e., α + β ≤ 1. This invention employs a multilayer perceptron to process semantic features. The multilayer perceptron consists of multiple perceptron layers, each including an input layer, a hidden layer, and an output layer. The input layer receives data, the hidden layer extracts high-level features from the input data, and the output layer performs data classification or regression analysis based on the task type. This invention adds semantic features, visual features, and the coupling amount between semantic and visual features to fuse semantic and visual features, thus obtaining fused features.

[0116] Example 3

[0117] like Figure 1 As shown, this embodiment provides an SMT processing method that integrates text and defect images. This embodiment describes the differences between it and Embodiment 1. The method includes:

[0118] S1: Obtain AOI-detected text and SMT defect images.

[0119] S2: Extract the semantic features of the AOI-detected text.

[0120] S3: Extract the visual features of the SMT defect image.

[0121] S4: Perform multimodal fusion of the semantic features and the visual features to obtain fused features; determine the defect type based on the fused features.

[0122] S5: Use a knowledge graph to construct the association matrix between defect types and process parameters.

[0123] S6: Update the correlation matrix.

[0124] S7: Construct a loss function based on the correlation matrix, and use the loss function to update the current process parameter vector to obtain the intermediate process parameter vector.

[0125] S8: Calculate a recommended process parameter vector based on the intermediate process parameter vector, and adjust the SMT processing technology using the recommended process parameter vector.

[0126] The method of constructing a correlation matrix between defect types and process parameters using a knowledge graph includes:

[0127] Construct the defect type set and process parameter set according to the following formulas:

[0128] De = {d1, d2, ..., d} M};

[0129] Pe = {p1, p2, ..., p} N};

[0130] Where De is the set of defect types, Pe is the set of process parameters; d1 is the first defect type, d2 is the second defect type, and d M This represents the Mth defect type; p1 is the first process parameter, p2 is the second process parameter, and p... N This is the Nth process parameter;

[0131] The mapping relationship between the elements of the defect type set and the elements of the process parameter set is constructed based on the knowledge graph;

[0132] Based on the mapping relationship, an association matrix is ​​constructed between defect types and process parameters; the element W in the i-th row and j-th column of the association matrix... i,j Used to represent the i-th defect d i and the j-th process parameter p j The strength of the correlation between them.

[0133] Defect types include solder bridges, cold solder joints, insufficient solder, and reverse soldering. Process parameters include reflow soldering temperature and solder paste thickness. The correlation matrix consists of the mapping relationship between the defect type set and the process parameter set. The element W in the correlation matrix... i,j The larger the value of the correlation matrix, the higher the correlation between the j-th process parameter and the i-th defect type. The values ​​of elements in the correlation matrix are greater than or equal to 0. The size of the correlation matrix is ​​M×N, where M is the total number of defect types and N is the total number of process parameter categories. This invention uses the correlation matrix to analyze the correlation between defect types and process parameters, thereby performing root cause diagnosis of defects occurring during SMT processing.

[0134] Updating the correlation matrix includes:

[0135] The update matrix is ​​calculated based on the correlation between the current defect type and process parameters;

[0136] The (t+1)th correlation matrix is ​​calculated using the update matrix and the t-th correlation matrix according to the following formula:

[0137] W t+1 =W t +η.ΔW;

[0138] ΔW i,j =var(d i ).var(p j )+∈cov(d i ,p j );

[0139] Among them, W t+1 Let W be the (t+1)th incidence matrix. t Let $\frac{ ... i,j To update the element in the i-th row and j-th column of the matrix, d i For the i-th defect type, p j For the j-th process parameter, var(d i ) represents the variance of the i-th defect type, var(p j Let ) represent the variance of the j-th process parameter, ∈ be the adjustment parameter, and cov(d) be the variance of the j-th process parameter. i ,p j Let be the covariance between the i-th defect type and the j-th process parameter.

[0140] Multiply the variance of the i-th defect type by the variance of the j-th process parameter, multiply the smoothing term by the covariance between the i-th defect type and the j-th process parameter, and then add the above two terms to obtain the element ΔW in the i-th row and j-th column of the update matrix. i,j The learning rate η is greater than 0 and less than 1. In this embodiment, the learning rate η is set to 0.1. The learning rate is multiplied by the update matrix to gradually update the correlation matrix.

[0141] The step of constructing a loss function based on the correlation matrix and updating the current process parameter vector using the loss function to obtain an intermediate process parameter vector includes:

[0142]

[0143] Where Cost represents the loss function, P(d i |Pe) represents the probability of the i-th defect type occurring under the set of process parameters; Let λ represent the severity weight of the i-th defect type, λ be the regularization coefficient, and P0 represent the current process parameter vector. The square of the L2 norm of the difference between the set of process parameters and the current process parameter vector;

[0144] The constraint condition for the j-th process parameter in the set of process parameters is:

[0145] p min ≤p j ≤p max ;

[0146] Where, p min p represents the minimum value of the process parameter. max This indicates the maximum value of the process parameter;

[0147] Under the constraint of the j-th process parameter, search for the minimum value of the loss function, and take the set of process parameters corresponding to the minimum value of the loss function as the intermediate process parameter vector.

[0148] Construct a loss function where, for each process parameter in the process parameter set, the probability of the i-th defect type occurring under the given process parameter set conditions is multiplied by the severity weight of the i-th defect type, where i starts from 1 and gradually increases to M. After M iterations, the M values ​​are multiplied by the severity weight of the i-th defect type. Summate the results and add λ. Once the loss function is obtained, the process parameters in the process parameter set are adjusted, the minimum value of the loss function Cost is searched, and the process parameter set corresponding to the minimum value of the loss function Cost is used as an intermediate process parameter vector. The process parameter set is a row vector.

[0149] The calculation of the recommended process parameter vector based on the intermediate process parameter vector includes:

[0150] The recommended process parameter vector is calculated using the following formula:

[0151]

[0152] Where, p rec To recommend the process parameter vector, p middle For intermediate process parameter vectors, Δp j γ is the adjustment amount for the j-th process parameter. j γ represents the event weight for the i-th defect type occurring under the j-th process parameter, where N is the total number of process parameters. j For P(p) j |d i The probability weights, Δp j The calculation formula is as follows:

[0153] Δp j =u j +σ j .Normal(0,1);

[0154] Among them, u j For Δp j The adjusted mean, σ j For Δp j The adjusted variance is given by Normal, which is a normal distribution with a mean of 0 and a standard deviation of 1. Historical data are analyzed to obtain u. j and σj .

[0155] Experimental verification shows that the SMT processing method integrating text and defect images in this embodiment has a recommended SMT process optimization time of 5.6 hours, while the traditional method has a process optimization time of 8.6 hours (h represents hours). Therefore, this embodiment improves the optimization efficiency by 35%, reduces the false alarm rate to below 1%, and also supports cross-production line process adaptation, thus enhancing the intelligence level of the AOI inspection system.

[0156] Example 4

[0157] This embodiment provides an intelligent traffic task execution device based on image dehazing, including:

[0158] The data acquisition module is used to acquire AOI detection text and SMT defect images;

[0159] The semantic feature extraction module is used to extract the semantic features of the AOI detected text;

[0160] A visual feature extraction module is used to extract the visual features of the SMT defect image;

[0161] A multimodal fusion module is used to fuse the semantic features and the visual features in a multimodal manner to obtain fused features; and to determine the defect type based on the fused features.

[0162] The association matrix construction module is used to construct an association matrix between defect types and process parameters in the form of a knowledge graph.

[0163] An association matrix update module is used to update the association matrix;

[0164] The current process parameter vector update module is used to construct a loss function based on the correlation matrix, and update the current process parameter vector using the loss function to obtain the intermediate process parameter vector.

[0165] The SMT processing technology adjustment module is used to calculate a recommended process parameter vector based on the intermediate process parameter vector, and to adjust the SMT processing technology using the recommended process parameter vector.

[0166] The SMT processing apparatus for fusing text and defect images in this embodiment is used to execute the SMT processing method for fusing text and defect images in any one of the embodiments 1-3.

[0167] This embodiment also provides a computer device, which may be a server. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor in this computer design provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communication with external terminals via a network connection.

[0168] This embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the SMT processing method for fusing text and defect images as described in any one of Embodiments 1-3. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0169] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0170] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for SMT processing that integrates text and defect images, characterized in that, include: Acquire AOI-detected text and SMT defect images; Extract the semantic features of the AOI-detected text; Extract the visual features of the SMT defect image; The semantic features and the visual features are fused using a multimodal method to obtain fused features; Determine the defect type based on the fusion features; A knowledge graph is used to construct an association matrix between defect types and process parameters; Update the correlation matrix; A loss function is constructed based on the correlation matrix, and the current process parameter vector is updated using the loss function to obtain the intermediate process parameter vector. Based on the intermediate process parameter vector, a recommended process parameter vector is calculated, and the SMT processing technology is adjusted using the recommended process parameter vector.

2. The SMT processing method for fusing text and defect images according to claim 1, characterized in that, The extraction of semantic features from the AOI-detected text includes: A large language model is used to encode the AOI detection text to obtain initial encoded features; The initial encoded features are iteratively updated to obtain the final encoded features; The final encoded features are subjected to a nonlinear transformation to obtain the transformed encoded features; By introducing classification markers into the transformed encoded features, the semantic features of the AOI detected text are obtained.

3. The SMT processing method for fusing text and defect images according to claim 2, characterized in that, The iterative update of the initial encoded features to obtain the final encoded features includes: The initial encoded features are subjected to layer normalization to obtain the first normalized feature; Multi-head attention calculation is performed on the first normalized feature to obtain the first attention matrix; The first attention matrix is ​​added to the initial encoded features to obtain the second encoded features; The second encoded feature is subjected to layer normalization, multi-head attention calculation, and matrix summation to obtain the final encoded feature.

4. The SMT processing method for fusing text and defect images according to claim 1, characterized in that, The extraction of visual features from the SMT defect image includes: The defect location and defect morphology of the SMT defect image are extracted using the ResNet model. The defect location and defect morphology constitute the visual features of the SMT defect image.

5. The SMT processing method for fusing text and defect images according to claim 4, characterized in that, The step of fusing the semantic features and the visual features in a multimodal manner to obtain fused features includes: The semantic features and the visual features are fused according to the following formula: F=α.MLP(T)+(1-α).V+β.(MLP(T)⊙V); Where F is the fusion feature, α is the first fusion weight, β is the second fusion weight, MLP is a multilayer perceptron used to align the feature dimensions of the semantic feature and the visual feature; T is the semantic feature, V is the visual feature, and ⊙ is element-wise multiplication.

6. The SMT processing method for fusing text and defect images according to claim 1, characterized in that, The method of constructing a correlation matrix between defect types and process parameters using a knowledge graph includes: Construct the defect type set and process parameter set according to the following formulas: De = {d1, d2, ..., d} M }; For={p1,p2,...,p N }; Where De is the set of defect types, Pe is the set of process parameters; d1 is the first defect type, d2 is the second defect type, and d M This represents the Mth defect type; p1 is the first process parameter, p2 is the second process parameter, and p... N This is the Nth process parameter; The mapping relationship between the elements of the defect type set and the elements of the process parameter set is constructed based on the knowledge graph; Based on the mapping relationship, an association matrix is ​​constructed between defect types and process parameters; the element W in the i-th row and j-th column of the association matrix... i,j Used to represent the i-th defect d i and the j-th process parameter p j The strength of the correlation between them.

7. The SMT processing method for fusing text and defect images according to claim 1, characterized in that, Updating the correlation matrix includes: The update matrix is ​​calculated based on the correlation between the current defect type and process parameters; The (t+1)th correlation matrix is ​​calculated using the update matrix and the t-th correlation matrix according to the following formula: IN t+1 =In t +η.ΔW; ΔW i,j =var(d i ).var(p j )+∈cov(d i ,p j ); Among them, W t+1 Let W be the (t+1)th incidence matrix. t Let $\frac{ ... i,j To update the element in the i-th row and j-th column of the matrix, d i For the i-th defect type, p j For the j-th process parameter, var(d i ) represents the variance of the i-th defect type, var(p j Let ) represent the variance of the j-th process parameter, ∈ be the adjustment parameter, and cov(d) i ,p j Let be the covariance between the i-th defect type and the j-th process parameter.

8. The SMT processing method for fusing text and defect images according to claim 6, characterized in that, The step of constructing a loss function based on the correlation matrix and updating the current process parameter vector using the loss function to obtain an intermediate process parameter vector includes: Where Cost represents the loss function, P(d i |Pe) represents the probability of the i-th defect type occurring under the set of process parameters; ωd i Let λ represent the severity weight of the i-th defect type, λ be the regularization coefficient, and P0 represent the current process parameter vector. The square of the L2 norm of the difference between the set of process parameters and the current process parameter vector; The constraint condition for the j-th process parameter in the set of process parameters is: p min ≤p j ≤p max ; Where, p min p represents the minimum value of the process parameter. max This indicates the maximum value of the process parameter; Under the constraint of the j-th process parameter, search for the minimum value of the loss function, and take the set of process parameters corresponding to the minimum value of the loss function as the intermediate process parameter vector.

9. The SMT processing method for fusing text and defect images according to claim 1, characterized in that, The calculation of the recommended process parameter vector based on the intermediate process parameter vector includes: The recommended process parameter vector is calculated using the following formula: Where, p rec To recommend the process parameter vector, p middle For intermediate process parameter vectors, Δp j γ is the adjustment amount for the j-th process parameter. j Let be the event weight for the i-th defect type occurring under the j-th process parameter, and N be the total number of process parameters.

10. The SMT processing method for fusing text and defect images according to claim 2, characterized in that, The method of encoding the AOI detection text using a large language model to obtain initial encoded features includes: The AOI-detected text is encoded according to the following formula: H1=TokenEmbed(X)+PosEmbed(X); Where H1 represents the initial encoding features, TokenEmbed(X) represents the word embedding matrix of the word sequence of the AOI detection text, PosEmbed(X) represents the position encoding matrix of the word sequence of the AOI detection text, and X is the word sequence of the AOI detection text.

Images