A method and system for identifying the length of intelligent bonding wire

Abstract

The present invention relates to the field of electronic manufacturing inspection technology, and discloses a method and system for identifying the length of intelligent bonding wire. Among them, a method for identifying the length of intelligent bonding wire includes the following steps: collecting image sequence information during the dynamic deformation process of the bonding wire; constructing a feature extraction model based on a convolutional neural network; combining the bending degree and speed in the obtained motion parameters of the bonding wire, and using a long short-term memory network to model the dynamic changes of the bonding wire; inputting the set of feature vectors into the trained dynamic change model to obtain the hidden state representing the dynamic change law of the bonding wire, and using a regression algorithm to obtain the length value of the bonding wire in each frame of image. The present invention can accurately obtain the dynamic length of the bonding wire through multi-angle collection, feature extraction, dynamic modeling and regression algorithm in electronic manufacturing inspection, and improve production quality and efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic manufacturing inspection, and more specifically, it relates to a method and system for identifying the length of intelligent bonding wires. Background Art

[0002] In the current field of electronic manufacturing, especially in the production and application scenarios of bonding wires in flexible electronic manufacturing, accurately identifying the length of bonding wires is crucial for ensuring product quality and production efficiency. However, there are many deficiencies in the existing methods for identifying the length of bonding wires. For example, traditional methods based on manual measurement are not only inefficient but also easily affected by human factors, resulting in large measurement errors. Even when using some automated measurement means, in complex production environments, such as ultra-small production microenvironments, due to the existence of various interference factors such as temperature and humidity, existing length identification systems based on simple image processing or single-parameter modeling are difficult to accurately capture the dynamic length changes of bonding wires under different conditions. This makes it impossible to obtain the length information of bonding wires in a timely and accurate manner during the production process, which may in turn lead to quality defects in products and increase production costs. Summary of the Invention

[0003] The present invention provides a method for identifying the length of intelligent bonding wires, including:

[0004] Image acquisition: acquiring image sequence information of the dynamic deformation process of the bonding wire from different angles by means of multiple high-speed cameras;

[0005] Feature extraction: constructing a feature extraction model based on a convolutional neural network, inputting the image sequence information into the feature extraction model, extracting the morphological features of the bonding wire, and forming a set of feature vectors;

[0006] Dynamic modeling: combining the bending degree and speed in the obtained motion parameters of the bonding wire, using a long short-term memory network to model the dynamic changes of the bonding wire; taking the set of feature vectors as the input, and at the same time integrating the bending degree and speed parameters into the long short-term memory network, enabling the long short-term memory network to learn the dynamic change law of the bonding wire, and obtaining a trained dynamic change model;

[0007] Length calculation: inputting the set of feature vectors into the trained dynamic change model to obtain the hidden state representing the dynamic change law of the bonding wire, and using a regression algorithm to obtain the length value of the bonding wire in each frame of the image.

[0008] In a preferred embodiment, the image sequence information is denoted as an image sequence set , , representing the th frame image, Represents the total number of image sequences; for each frame of the acquired image sequence In each frame of the image in, the Gaussian filtering formula is used to perform a convolution operation on the image to remove noise interference, and a preprocessed image sequence is obtained .

[0009] In a preferred embodiment, the input of the feature extraction model is the preprocessed image , which passes through multiple convolutional layers and pooling layers ; the depth convolution operation is expressed as:

[0010] ;

[0011] wherein, is the input feature map, represents the weight value of the convolution kernel at the position , is the bias; and are the coordinate offsets during the sliding of the convolution kernel; is the feature value output at the coordinate after the convolution operation; the morphological features of the bonding wire are extracted through the feature extraction model to form a set of feature vectors , represents the feature vector of the i-th frame image, represents the total number of feature vectors, and the total number of feature vectors is the same as the total number of image sequences; these feature vectors will be used as important inputs for subsequent dynamic modeling.

[0012] In a preferred embodiment, the input gate , forget gate , output gate and the candidate value of the new memory cell are calculated by the following formulas respectively:

[0013] ;

[0014] ;

[0015] ;

[0016] ;

[0017] wherein, is the sigmoid activation function; is the hyperbolic tangent activation function; , respectively represent the first and second weight matrices, , respectively represent the third and fourth weight matrices; and respectively represent the fifth and sixth weight matrices; and respectively represent the seventh and eighth weight matrices; is the eigenvector corresponding to the th frame image; represents the hidden state at time k = 1 ; and and and respectively represent the first, second, third, and fourth bias vectors;

[0018] The update formula for the memory cell state is as follows:

[0019] ;

[0020] where represents the memory cell state at time ; represents the memory cell state at time represents the forget gate value at time ; represents the input gate value at time ; represents the new memory cell candidate value at time ;

[0021] The hidden state is updated to:

[0022] ;

[0023] where represents the hidden state at time ; represents the memory cell state at time ; represents the output gate value at time ; represents the hyperbolic tangent activation function;

[0024] The expression of the output layer is:

[0025] ;

[0026] where represents the motion parameters of the bonding wire at time

[0027] Train a long short-term memory network, and then obtain a dynamic change model of the hidden state of the dynamic change law of the bonding wire by removing the output layer. The dynamic change model can capture the dynamic change characteristics of the bonding wire at different times.

[0028] In a preferred embodiment, in the length calculation, the length of the bonding wire in each frame of image is calculated by a regression algorithm; the length calculation formula is:

[0029] ;

[0030] Where is the regression function, and the length of the th frame of image is , represents the hidden state at time .

[0031] In a preferred embodiment, sensors are set on the movement path of the bonding wire to collect these parameters in real time. An angle sensor is used to obtain the bending degree , and a speed sensor is used to obtain the speed .

[0032] In a preferred embodiment, a method for identifying the length of an intelligent bonding wire further includes:

[0033] Fast image preprocessing. For each frame of image in the obtained image sequence , a simplified median filtering method is used to replace Gaussian filtering;

[0034] Construction of a lightweight feature extraction model. Build a lightweight convolutional neural network feature extraction model MobileNet;

[0035] Dynamic modeling integrating action information. Combining the obtained operation action types of maintenance personnel and the feature vector of the bonding wire , use a gated recurrent unit to model the dynamic change of the bonding wire to obtain a dynamic change model of the bonding wire integrating action information ;

[0036] Length calculation based on device parameters. According to the trained dynamic change model of the bonding wire integrating action information , combined with the hardware parameters of the handheld detection device, calculate the length of the bonding wire in each frame of image through an optimized regression algorithm.

[0037] In a preferred embodiment, the gated recurrent unit passes through a reset gate and an update gate To control the information flow; the inputs are the feature vector of the image, the type of operation actions of the maintenance personnel and the hidden state at the previous moment , the reset gate and the update gate are respectively:

[0038] ;

[0039] ;

[0040] where, is the sigmoid activation function, , , , respectively represent the ninth, tenth, eleventh, and twelfth weight matrices, , respectively represent the fifth and sixth bias vectors, represents the feature vector of the i-th frame image;

[0041] The candidate hidden state is:

[0042] ;

[0043] where, is the hyperbolic tangent activation function, , respectively represent the thirteenth and fourteenth weight matrices, represents the seventh bias vector;

[0044] The hidden state is updated to:

[0045] ;

[0046] where, represents the hidden state at time ; represents the update gate value at time ; represents the hidden state at time, that is, the hidden state at the previous moment; represents the candidate hidden state at time ;

[0047] The expression of the output layer is:

[0048] ;

[0049] where represents the motion parameters of the bonding wire at time;

[0050] Train the gated recurrent unit, and then obtain the dynamic change model of the bonding wire that fuses the action information after removing the output layer 。

[0051] In a preferred embodiment, the length formula is calculated using an optimized regression algorithm as follows:

[0052] ;

[0053] Wherein, is the optimized regression function, is the length of the bonding wire in the th frame image, represents the hidden state at time , is the type of maintenance personnel's operation action, is the resolution of the handheld detection device, is the frame rate of the handheld detection device.

[0054] In a preferred embodiment, an intelligent bonding wire length recognition system includes:

[0055] An image acquisition module that uses a handheld detection device tool to acquire an image sequence containing the bonding wire;

[0056] A feature extraction module that constructs a lightweight convolutional neural network feature extraction model and uses depthwise separable convolution and pooling layers to process the images;

[0057] An action information fusion and dynamic modeling module that combines the type of maintenance personnel's operation action and the feature vector of the bonding wire, and uses a gated recurrent unit to model the dynamic changes of the bonding wire;

[0058] A length calculation module that calculates the length of the bonding wire in each frame image according to the trained dynamic change model of the bonding wire that fuses the action information and the feature vector set, in combination with the hardware parameters of the handheld detection device, through an optimized regression algorithm.

[0059] The beneficial effects of the present invention are as follows:

[0060] Improve the recognition accuracy: By constructing a complex and deeper convolutional neural network model and adding an attention mechanism module, the morphological features of the bonding wire can be extracted more accurately. At the same time, combined with the improved long short-term memory network for dynamic modeling, fully considering the influence of microenvironment parameters (temperature, humidity, production line speed, etc.) on the length of the bonding wire, high-precision length recognition is achieved, and the recognition accuracy is improved compared with traditional methods.

[0061] Adapting to different scenarios: For different application scenarios, such as the repair scenario of portable wearable flexible electronic devices, the algorithm is further optimized. Methods such as rapid image preprocessing and construction of lightweight feature extraction models are adopted to reduce the computational complexity while ensuring the measurement accuracy, adapting to the limited resources of portable devices and broadening the application scope of the system. Brief Description of the Drawings

[0062] Figure 1 is a flowchart of a method for identifying the length of an intelligent bonding wire of the present invention;

[0063] Figure 2 is a diagram showing an example of the data collected by the present invention;

[0064] Figure 3 is an example diagram of the convolutional layer of the present invention;

[0065] Figure 4 is an example diagram of the pooling layer of the present invention. Detailed Description of the Invention

[0066] Now, the subject matter described herein will be discussed with reference to example embodiments. It should be understood that discussing these embodiments is only to enable those skilled in the art to better understand and thus implement the subject matter described herein. Without departing from the scope of protection of the content of this specification, changes can be made to the functions and arrangements of the elements discussed. Each example can omit, substitute, or add various processes or components as needed. Additionally, the features described in some examples can also be combined in other examples.

[0067] In at least one embodiment of the present invention, a method for identifying the length of an intelligent bonding wire is disclosed, as Figure 1 shown, including:

[0068] Step100, Image acquisition, acquiring image sequence information of the dynamic deformation process of the bonding wire from different angles by means of multiple groups of high-speed cameras;

[0069] In an embodiment of the present invention, during the flexible electronic manufacturing process, the bonding wire may undergo dynamic deformation due to factors such as the operation of production equipment, the characteristics of the material itself, and external environmental factors. For example, in the flexible electronic manufacturing scenario where images are acquired by high-speed cameras, the bonding wire will be in a state of dynamic deformation.

[0070] In an embodiment of the present invention, the image sequence information is denoted as the image sequence set , where , represents the th frame image, represents the total number of the image sequence; for the acquired image sequence For each frame of the image, the Gaussian filtering formula is used to perform a convolution operation on the image to remove noise interference, obtaining a preprocessed image sequence. . The Gaussian filtering formula is:

[0071] ;

[0072] Among them, is the image coordinate, is the center of the Gaussian kernel function, is the standard deviation of the Gaussian kernel function, used to control the smoothness. is the constant part related to pi, is the natural constant, serving as the base of the exponential function in this formula. Through this step, the quality of the subsequent processed images is higher, reducing the impact of noise on feature extraction.

[0073] Step200, Feature extraction, construct a feature extraction model based on a convolutional neural network, input the image sequence information into the feature extraction model, extract the morphological features of the bonding wire, and form a set of feature vectors;

[0074] In an embodiment of the present invention, the CNN model generally consists of a convolutional layer, a pooling layer, a fully connected layer, etc., and the input is the preprocessed image , passing through multiple convolutional layers and pooling layers ;

[0075] The deep convolution operation can be expressed as:

[0076] ;

[0077] Among them, is the input feature map, represents the weight value of the convolutional kernel at position , is the bias. and are the coordinate offsets during the sliding of the convolutional kernel, used to traverse the corresponding positions of the convolutional kernel and the input feature map for multiplication operations. is the feature value output at coordinate after the convolution operation. The pooling layer mainly performs downsampling on the convolved feature map, reducing the data volume while retaining the main features. Through this model, the morphological features of the bonding wire are extracted, forming a set of feature vectors , , , respectively represent the feature vectors of the 1st, 2nd, frame images, Represents the total number of feature vectors, which is the same as the total number of image sequences; these feature vectors will serve as important inputs for subsequent dynamic modeling.

[0078] Step 300, dynamic modeling. Combine the bending degree and speed in the obtained movement parameters of the bonding wire, and use the long short-term memory network to model the dynamic changes of the bonding wire; take the set of feature vectors as the input, and at the same time incorporate the bending degree and speed parameters into the long short-term memory network, allowing the long short-term memory network to learn the dynamic change rules of the bonding wire to obtain a trained dynamic change model.

[0079] In an embodiment of the present invention, these parameters can be collected in real time by setting sensors on the movement path of the bonding wire. For example, an angle sensor is used to obtain the bending degree , and a speed sensor is used to obtain the speed .

[0080] In an embodiment of the present invention, LSTM is a special recurrent neural network (RNN), which controls the flow of information through an input gate, a forget gate, and an output gate. The input is the feature vector and the hidden state at time , the memory cell state , the input gate , the forget gate , the output gate and the new memory cell candidate value are calculated respectively by the following formulas ;

[0081] ;

[0082] ;

[0083] ;

[0084] where is the sigmoid activation function; is the hyperbolic tangent activation function; , respectively represent the first and second weight matrices, , respectively represent the third and fourth weight matrices; , respectively represent the fifth and sixth weight matrices; , respectively represent the seventh and eighth weight matrices; is the feature vector corresponding to the th frame of the image; represents The moment hidden state, when k = 1 ; , , , respectively represent the first, second, third, and fourth bias vectors.

[0085] The update formula for the memory cell state is as follows:

[0086] ;

[0087] where, represents the memory cell state at time ; represents the memory cell state at time represents the forgetting gate value at time ; represents the input gate value at time ; represents the new memory cell candidate value at time ;

[0088] The hidden state is updated to:

[0089] ;

[0090] where, represents the hidden state at time ; represents the memory cell state at time ; represents the output gate value at time ; represents the hyperbolic tangent activation function;

[0091] The expression of the output layer is:

[0092] ;

[0093] where represents the motion parameters of the bonding wire at time

[0094] Train the long short-term memory network, and then remove the output layer to obtain the dynamic change model of the hidden state of the dynamic change law of the bonding wire ; The dynamic change model can capture the dynamic change characteristics of the bonding wire at different times.

[0095] Step400, length calculation, input the feature vector set into the trained dynamic change model to obtain the hidden state representing the dynamic change law of the bonding wire, and use the regression algorithm to obtain the length value of the bonding wire in each frame of the image:

[0096] ;

[0097] Among them, is the regression function, and the length of the th frame image is , represents the hidden state at time .

[0098] Through the above steps, a real-time length sequence of the bonding wire during dynamic deformation is finally generated , where , , represent the 1st, 2nd, and mth lengths respectively, the total number of the m-length sequences, and further, the actual length or total length L of the bonding wire can be obtained by modeling. , represents the model used to calculate the length L, which can be a multi-layer perceptron or a linear regression model.

[0099] In an embodiment of the present invention, an example of the foregoing method for identifying the length of an intelligent bonding wire is provided:

[0100] Image preprocessing:

[0101] As Figure 2 shown, in a certain flexible electronics manufacturing factory, 5 groups of high-speed cameras are used to collect image sequences of the dynamic deformation process of the bonding wire from different angles. For example, in a single acquisition task, a total of 100 frames of images are obtained, forming an image sequence set ;

[0102] For each frame of image , the Gaussian filtering formula is used for calculation and convolution operation, where ;

[0103] is the center of the image. After processing, the preprocessed image sequence is obtained.

[0104] Feature extraction:

[0105] A feature extraction model based on a convolutional neural network (CNN) is constructed. Taking the 10th preprocessed image as an example, it is input into the model. As Figure 3 shown, the convolution kernel size of the convolution layer in the model is 3x3, and the stride is 1. After passing through this convolution layer, the size of the feature map obtained is the original image size minus the convolution kernel size plus 1. As Figure 4 shown, the pooling layer Max pooling is adopted with a pooling kernel size of 2x2 and a stride of 2. After multiple convolutional layers and pooling layers, the morphological features of the frame image are extracted to form a feature vector .

[0106] Dynamic modeling:

[0107] Combined with the bending degree and speed of the obtained bonding wire motion parameters, the long short-term memory network (LSTM) is used to model the dynamic changes of the bonding wire. Taking the feature vector as an example,

[0108] Input gate:

[0109] ;

[0110] Among them, is the input gate value at the 10th moment; is the sigmoid activation function; , respectively represent the first and second weight matrices of the input gate; is the feature vector corresponding to the 10th frame image; is the hidden state at the 9th moment; represents the first bias vector.

[0111] Forget gate:

[0112] ;

[0113] Among them, is the forget gate value at the 10th moment; is the sigmoid activation function; , respectively represent the third and fourth weight matrices of the input gate, used to adjust the influence degree of the previous moment hidden state on the forget gate calculation result; represents the second bias vector.

[0114] Output gate:

[0115] ;

[0116] Among them, is the output gate value at the 10th moment; is the sigmoid activation function; , respectively represent the fifth and sixth weight matrices of the input gate, used to adjust the previous moment hidden state Degree of influence on the calculation result of the output gate; Represents the third bias vector.

[0117] New memory cell candidate value:

[0118] ;

[0119] Among them, Is the new memory cell candidate value at the 10th moment; Is the hyperbolic tangent activation function; Is the seventh weight matrix in the new memory cell candidate value, used to adjust the feature vector Degree of influence on the calculation result of the new memory cell candidate value; Is the eighth weight matrix in the new memory cell candidate value, used to adjust the previous moment hidden state Degree of influence on the calculation result of the new memory cell candidate value; Is the fourth bias vector in the new memory cell candidate value.

[0120] The memory cell state is updated to:

[0121] ;

[0122] Among them, Is the updated memory cell state at the 10th moment; Is the memory cell state at the 9th moment; Is the forget gate value at the 10th moment, controlling the degree to which the memory cell state at the 9th moment Is retained to the 10th moment; Is the input gate value at the 10th moment, controlling the degree to which the new memory cell candidate value Enters the memory cell state; Is the new memory cell candidate value at the 10th moment.

[0123] The hidden state is updated to:

[0124] ;

[0125] Among them, Is the updated hidden state at the 10th moment; Is the hyperbolic tangent activation function; Is the output gate value at the 10th moment, controlling the degree to which the memory cell state Is output to the hidden state; Is the updated memory cell state at the 10th moment.

[0126] Length calculation:

[0127] Calculate the length of the bonding wire in each frame of the image through a regression algorithm;

[0128] The length calculation formula is:

[0129] ;

[0130] Among them, is the regression function, and the length of the th frame of the image is After calculation, .

[0131] In an embodiment of the present invention, in view of the limited resources of portable devices in the maintenance scenario of wearable flexible electronic devices, relevant technologies are further optimized and expanded; an intelligent method for identifying the length of bonding wire further includes:

[0132] Step101, rapid image preprocessing. Due to the limited resources of the handheld device, for each frame of the image sequence obtained, a simplified median filtering method is used to replace Gaussian filtering. .

[0133] In an embodiment of the present invention, median filtering replaces the value of a pixel point in the image with the median of the pixel values in the neighborhood of this point. Let the neighborhood size be , for the pixel point in the image, its pixel value after median filtering is the middle value after sorting all the pixel values in the neighborhood. Here represents the side length of the median filtering neighborhood. This step can reduce noise interference while reducing the computational complexity and adapting to the computing power of portable devices, and obtain the preprocessed image sequence .

[0134] Step102, construction of a lightweight feature extraction model. Construct a lightweight convolutional neural network (CNN) feature extraction model MobileNet. MobileNet uses depthwise separable convolutions, decomposing the standard convolution into a depthwise convolution and a pointwise convolution, which greatly reduces the model parameters and computational amount. Let the input be the preprocessed image , passing through multiple depthwise separable convolutional layers and pooling layers . The depthwise convolution operation can be expressed as:

[0135] ;

[0136] Among them, is the depthwise convolution kernel, and are the convolution kernels at and The index in the direction, is the value of the input feature map at the position ; is the value of the output feature map at the position . The pointwise convolution operation can be expressed as:

[0137] ;

[0138] where, is the pointwise convolution kernel, is the bias, and are also the indices of the convolution kernel in the and directions, is the value of the input feature map at the position ; is the value of the output feature map at the position . The pooling layer also downsamples the convolved feature map. The morphological features of the bonding wire are extracted through this lightweight model to form a set of feature vectors , , , represent the feature vectors of the 1st, 2nd, frame images respectively, represents the total number of feature vectors, and the total number of feature vectors is the same as the total number of the image sequence.

[0139] Step103, Dynamic modeling of the fused action information, combining the obtained operation action types of the maintenance personnel and the set of feature vectors of the bonding wire , and using the gated recurrent unit (GRU) to model the dynamic changes of the bonding wire. GRU is a simplified RNN, and it controls the information flow through the reset gate and the update gate . Let the input be the feature vector of the image and the hidden state at the previous moment , the reset gate and the update gate are respectively:

[0140] ;

[0141] ;

[0142] where, is the sigmoid activation function, , , , represent the ninth, tenth, eleventh, and twelfth weight matrices respectively, , respectively represent the fifth and sixth offset vectors represents the feature vector of the i-th frame image

[0143] The candidate hidden state is:

[0144] ;

[0145] where is the hyperbolic tangent activation function 、 respectively represent the thirteenth and fourteenth weight matrices represents the seventh offset vector

[0146] The hidden state is updated to:

[0147] ;

[0148] where represents the hidden state at time ; represents the update gate value at time ; represents the hidden state at time, that is, the hidden state of the previous time represents the candidate hidden state at time ;

[0149] The expression of the output layer is:

[0150]

[0151] where represents the motion parameters of the bonding wire at time

[0152] Train the gated recurrent unit, and then remove the output layer to obtain the dynamic change model of the bonding wire that fuses the action information .

[0153] Step104, Length calculation based on device parameters. According to the trained dynamic change model , and the feature vector set , combined with the hardware parameters of the handheld detection device (such as resolution 、frame rate ), calculate the length of the bonding wire in each frame image through an optimized regression algorithm. The formula for calculating the length using the optimized regression algorithm is as follows:

[0154] ;

[0155] where is the optimized regression function is the length of the bonding wire in the nth frame image, represents the hidden state at time , is the type of operation action of the maintenance personnel, is the resolution of the handheld detection device, is the frame rate of the handheld detection device.

[0156] In an embodiment of the present invention, based on the field of flexible electronics education practice, an intelligent bonding wire length recognition method further includes:

[0157] Step201, data preprocessing, obtaining a data set , and preprocessing the feature vector data in the data set;

[0158] In an embodiment of the present invention, the data set is a data set obtained in the context of a flexible electronics education practice scenario, which contains feature vector data related to the dynamic change of the bonding wire length; normalizing the feature vector set , setting the feature vector , and the normalization formula is:

[0159] ;

[0160] where, represents the minimum value in the feature vector set , represents the maximum value in the feature vector set . After calculation by this formula, the normalized feature vector set is obtained.

[0161] Step202, model construction and training, constructing a deep learning model based on a long short-term memory network (LSTM). Using the normalized feature vector set and the corresponding length set as training data. The model input is the feature vector and the previous hidden state , and the memory cell state .

[0162] The input gate , the forget gate , the output gate and the new memory cell candidate value are calculated respectively by the following formulas:

[0163]

[0164]

[0165]

[0166]

[0167] Among them, is the Sigmoid activation function; , , , , , , , respectively represent the fifteenth, sixteenth, seventeenth, eighteenth, nineteenth, twentieth, twenty-first, and twenty-second weight matrices, which are used to adjust the influence degree of input information on each calculation result; , , , are the eighth, ninth, tenth, and eleventh bias vectors respectively, which are used to increase the non-linear expression ability of the model.

[0168] The memory cell state is updated as:

[0169] ;

[0170] Among them represents element-wise multiplication, that is, the forget gate multiplied element-wise with the memory cell state at the previous moment, plus the result of the input gate multiplied element-wise with the new memory cell candidate value to obtain the updated memory cell state .

[0171] The hidden state is updated as:

[0172] , that is, the output gate multiplied element-wise with the memory cell state processed by the hyperbolic tangent function to obtain the updated hidden state .

[0173] After multiple iterations of training, the model learns the mapping relationship between the feature vector and the length, and obtains the trained model .

[0174] Step203, virtual simulation interaction implementation. In the virtual simulation environment, obtain the parameter bending degree set by the student, speed , convert it into a format that matches the feature vectors in the training dataset, denoted as the converted feature vectors ;

[0175] Input into the trained model , and predict the length of the bonding wire according to the model output . Through graphic rendering technology, the dynamic change of the length of the bonding wire is displayed in real time in the virtual environment when the bending degree and speed set are changed. This step realizes the virtual simulation interaction function and provides an intuitive learning experience for students.

[0176] Through the above steps, a virtual simulation environment is constructed, and the dynamic simulation of the bonding wire length is realized based on the deep learning algorithm. Students can intuitively observe the length change of the bonding wire under different bending degrees and speeds. This interactive learning method can significantly improve students' understanding of the dynamic change process of the bonding wire in flexible electronics manufacturing.

[0177] The embodiments of the present invention have been described above, but these embodiments are not limited to the above specific implementation manners. The above specific implementation manners are merely illustrative rather than restrictive. Under the inspiration of this embodiment, those of ordinary skill in the art can also make more equivalent embodiments in various forms, all of which fall within the protection scope of this embodiment.

Claims

1. A method for identifying the length of an intelligent bonding wire, characterized in that: include: Image acquisition: using multiple sets of high-speed cameras to collect image sequence information from different angles during the dynamic deformation of the bonding wire; Feature extraction: construct a feature extraction model based on convolutional neural network, input the image sequence information into the feature extraction model, extract the morphological features of the bonding wire, and form a feature vector set; Dynamic modeling, combining the bending degree and speed of the bonding wire motion parameters obtained, and using the long short-term memory network to model the dynamic changes of the bonding wire; The feature vector set is used as input, and the bending degree and speed parameters are integrated into the long short-term memory network, so that the long short-term memory network can learn the dynamic change law of the bonding wire and obtain a trained dynamic change model; Length calculation: Input the feature vector set into the trained dynamic change model to obtain the hidden state representing the dynamic change law of the bonding wire, and use the regression algorithm to obtain the length value of the bonding wire in each frame of the image.

2. The method for identifying the length of an intelligent bonding wire according to claim 1, characterized in that: Record the image sequence information as an image sequence set , , Representative Frame image, Indicates the total number of image sequences; for the acquired image sequences For each frame of the image, the Gaussian filter formula is used to perform convolution operation on the image to remove noise interference and obtain the preprocessed image sequence .

3. The method for identifying the length of an intelligent bonding wire according to claim 2, characterized in that: The feature extraction model input is the preprocessed image , after multiple convolutional layers and pooling layer ; The depth convolution operation is expressed as: ； in, is the input feature map, Indicates that the convolution kernel is at position The weight value at is bias; and It is the coordinate offset during the sliding process of the convolution kernel; After the convolution operation, the coordinate The eigenvalue output at the output; the morphological features of the bonding wire are extracted through the feature extraction model to form a feature vector set , Represents the feature vector of the i-th frame image, Represents the total number of feature vectors, which is the same as the total number of image sequences; these feature vectors will serve as important inputs for subsequent dynamic modeling.

4. The method for identifying the length of an intelligent bonding wire according to claim 1, characterized in that: Input gate of LSTM network , Forget Gate , output gate and new memory cell candidate values Calculated by the following formulas: ； ； ； ； in, is the sigmoid activation function; is the hyperbolic tangent activation function; , denote the first and second weight matrices respectively, , denote the third and fourth weight matrices respectively; , denote the fifth and sixth weight matrices respectively; , denote the seventh and eighth weight matrices respectively; It corresponds to Feature vector of frame image; express Hidden state at all times, when k=1 ; , , , denote the first, second, third, and fourth bias vectors respectively; The memory cell state update formula is as follows: ； in, Indicates time The state of memory cells; express The state of memory cells at each moment; Indicates time The forget gate value of Indicates time The input gate value of Indicates time New memory cell candidate value; The hidden state is updated to: ； in, Indicates time The hidden state of Indicates time The state of memory cells; Indicates time The output gate value of represents the hyperbolic tangent activation function; The expression of the output layer is: ； in express Motion parameters of the bonding wire at each moment; Train the long short-term memory network, and then remove the output layer to obtain the dynamic change model of the hidden state of the dynamic change law of the bonding wire. ; The dynamic change model can capture the dynamic change characteristics of the bonding wire at different times.

5. The method for identifying the length of an intelligent bonding wire according to claim 1, characterized in that: In the length calculation, the length of the bonding wire in each frame of the image is calculated by a regression algorithm; the length calculation formula is: ； in, is the regression function, The length of the frame image is , Indicates time The hidden state of .

6. The method for identifying the length of an intelligent bonding wire according to claim 1, characterized in that: Sensors are set up on the movement path of the bonding wire to collect these parameters in real time, and angle sensors are used to obtain the degree of bending. , use the speed sensor to obtain the speed .

7. The method for identifying the length of an intelligent bonding wire according to claim 2, characterized in that: Also includes: Fast image preprocessing, for the acquired image sequence Each frame of the image , a simplified median filtering method is used instead of Gaussian filtering; Lightweight feature extraction model construction, build a lightweight convolutional neural network feature extraction model MobileNet; Dynamic modeling of fusion action information, combined with the acquired maintenance personnel operation action types and the characteristic vector of the bond wire , the gated recurrent unit is used to model the dynamic changes of the bonding wire to obtain the dynamic change model of the bonding wire that integrates the action information ; Length calculation based on device parameters, dynamic change model of bonding wire based on trained fusion motion information ,Combined with the hardware parameters of the handheld detection device, the length of the bonding wire in each frame of the image is calculated through an optimized regression algorithm.

8. The method for identifying the length of an intelligent bonding wire according to claim 7, characterized in that: The gated recurrent unit is reset by and update gate To control the flow of information; the input is the feature vector of the image and the type of operation action of the maintenance personnel and the hidden state at the previous moment , the reset gate and update gate are: ； ； in, is the sigmoid activation function, , , , Respectively represent the ninth, tenth, eleventh, and twelfth weight matrices, , Respectively represent the fifth and sixth bias vectors, Represents the feature vector of the i-th frame image; The candidate hidden states are: ； in, is the hyperbolic tangent activation function, , denote the thirteenth and fourteenth weight matrices respectively, represents the seventh bias vector; The hidden state is updated to: ； in, Indicates time The hidden state of Indicates time The update gate value of express The hidden state at the moment, that is, the hidden state at the previous moment; Indicates time Candidate hidden states of ; The expression of the output layer is: ； in express Motion parameters of the bonding wire at each moment; Train the gated recurrent unit and then remove the output layer to get the dynamic change model of the bond wire that incorporates the action information. .

9. The method for identifying the length of an intelligent bonding wire according to claim 7, characterized in that: The formula for calculating the length using the optimized regression algorithm is as follows: ； in, is the optimized regression function, It is The length of the bonding wire in the frame image, Indicates time The hidden state of It is the type of operation action performed by maintenance personnel. is the resolution of the handheld detection device, is the frame rate of the handheld detection device.

10. An intelligent bonding wire length identification system, characterized in that: A method for identifying the length of an intelligent bonding wire according to any one of claims 1 to 9, comprising: An image acquisition module, using a handheld detection device tool to acquire an image sequence containing the bonded gold wire; Feature extraction module, which builds a lightweight convolutional neural network feature extraction model and processes images using deep separable convolution and pooling layers; The action information fusion and dynamic modeling module combines the operation action type of the maintenance personnel and the feature vector of the bonding wire, and uses the gated recurrent unit to model the dynamic changes of the bonding wire; The length calculation module calculates the length of the bonding wire in each frame of the image through an optimized regression algorithm based on the trained dynamic change model of the bonding wire that integrates the motion information and the feature vector set, combined with the hardware parameters of the handheld detection device.

Images