A method for identifying polarity marks of electronic components and related equipment

Abstract

The present invention provides a method for identifying polarity identification of electronic components and related equipment, which comprises the following steps: obtaining image data of multiple electronic components, coordinate labels and text descriptions of each electronic component; inputting the image data of multiple electronic components, coordinate labels and text descriptions of each electronic component into a polarity identification identification model to train the polarity identification identification model, and obtaining a trained polarity identification identification model; inputting the acquired image data of target electronic components and text descriptions of the target electronic components into the trained polarity identification identification model for classification and prediction, and obtaining classification results and prediction results of the target electronic components; and obtaining polarity identification results based on the classification results and predictions of the target electronic components; thereby improving the accuracy of polarity identification.

Description

Technical Field

[0001] The present invention relates to the technical field of component detection, and in particular to a method for identifying polarity marks of electronic components and related equipment. Background Art

[0002] In the field of electronic manufacturing, the identification of polarity marks of electronic components is an important step to ensure the normal assembly of chips and the effective functioning of their functions. Traditional polarity mark identification methods are divided into simple image processing methods such as edge detection and template matching, which mainly rely on optical or electron microscopes for thermal inspection and require manual design of corresponding features and template design. These methods not only require a certain amount of time and effort, but are also easily affected by the subjective judgment of operators. In addition, the feature extraction capabilities are limited and cannot fully capture the high-dimensional feature information of electronic component images. They are easily disturbed by external factors such as light and noise under complex backgrounds, resulting in low accuracy in the identification of electronic components. Summary of the invention

[0003] The present invention provides a method for identifying polarity marks of electronic components and related equipment, the purpose of which is to improve the accuracy of polarity mark identification.

[0004] In order to achieve the above object, the present invention provides a method for identifying polarity marks of electronic components, comprising:

[0005] Step 1, obtaining a plurality of electronic component image data, a coordinate label for describing the position coordinates of each electronic component, and a text description for describing the content of each electronic component image data;

[0006] Step 2, inputting a plurality of electronic component image data, a coordinate label and a text description of each electronic component into a polarity mark recognition model to train the polarity mark recognition model, thereby obtaining a trained polarity mark recognition model;

[0007] Step 3, inputting the acquired target electronic component image data and the text description used to describe the content of the target electronic component image data into the trained polarity mark recognition model for classification and prediction, and obtaining the polarity mark recognition result of the target electronic component, the polarity mark recognition result including the classification result and the prediction result;

[0008] The polarity mark recognition model includes a Swin Transformer module for extracting visual features, a Transformer module for extracting text features, a multimodal fusion module for fusing visual features and text features, and a classification prediction module;

[0009] The input end of the Swin Transformer module is used to receive multiple electronic component image data, and during the training process, the input end of the Transformer module is used to receive the coordinate label and text description of each electronic component. After training, the input end of the Transformer module is used to receive the text description of each electronic component;

[0010] The output of the Swin Transformer module and the output of the Transformer module are connected to the input of the multimodal fusion module;

[0011] The output end of the multimodal fusion module is connected to the input end of the classification prediction module, and the output end of the classification prediction module is used to output the polarity identification recognition result of each electronic component.

[0012] Furthermore, before step 2, it also includes:

[0013] Performing data enhancement processing on multiple electronic component image data to obtain multiple enhanced electronic component image data;

[0014] Use regular matching to process the coordinate labels of each electronic component, divide the coordinate labels into sentences, and preprocess each sentence to obtain the preprocessed coordinate labels;

[0015] Use a word segmentation tool to segment the text description of each electronic component to obtain a segmented text description.

[0016] Specifically, the Swin Transformer module is used to extract visual features. The process of extracting visual features includes:

[0017] For each electronic component image data, the electronic component image data is mapped to a high-dimensional feature space for feature extraction to obtain a shallow visual feature map of each electronic component image data;

[0018] For each shallow visual feature map, the shallow visual feature map is segmented to obtain multiple non-overlapping image window blocks, and each image window block is position-encoded to obtain the encoded visual features;

[0019] For each image window block, the image window block and its internal shallow visual features are calculated to obtain the attention score of each image window block;

[0020] The attention scores of all image window blocks are combined and convolved to obtain the visual features of each electronic component.

[0021] Specifically, the expression for position encoding of each image window block is:

[0022]

[0023] Among them, PE represents the position encoding, pos represents the position of the element in the sequence, i represents the dimension index of the position encoding, and d model Represents the dimensions of the model.

[0024] Specifically, the visual features of each electronic component are:

[0025] T tran =Tran N (T0), N = 1, 2, ..., 8

[0026] Among them, represents the visual feature vector of electronic components, H, W and C0 represent the height, width and number of channels of the image respectively, and T0 represents the shallow feature map of the electronic components.

[0027] Specifically, the Transformer module is used to extract text features. The process of extracting text features is as follows:

[0028] Download the text encoder model and model parameters from the pre-trained neural network model;

[0029] Initialize and freeze the embedding layer parameters in the text encoder model using the model parameters;

[0030] The coordinate label and text description of each electronic component are input into the embedding layer of the text encoder model for transformation to obtain the text features of each electronic component.

[0031] Specifically, the multimodal fusion module includes:

[0032] Self-attention layer, cross-attention layer, first fusion attention layer, second fusion attention layer, multi-layer perceptron;

[0033] The first input end of the first fused attention layer and the first input end of the second fused attention layer are both connected to the output end of the SwinTransformer module;

[0034] The output of the first fusion attention layer is connected to the first input of the cross attention layer;

[0035] The output of the second fusion attention layer is connected to the input of the multilayer perceptron;

[0036] The output end of the multilayer perceptron is connected to the second input end of the first fusion attention layer and the second input end of the cross attention layer respectively;

[0037] The input end of the self-attention layer is connected to the output end of the Transformer module, and the output end of the self-attention layer is connected to the second input end of the second fusion attention layer;

[0038] The output of the criss-cross attention layer is connected to the input of the classification prediction module.

[0039] Furthermore, the multimodal fusion module is used to fuse visual features and text features. The fusion process includes:

[0040] Perform self-attention calculation on text features to capture the dependencies and global features between word units and obtain text features after self-attention calculation;

[0041] The text features and visual features after self-attention calculation are input into the second fusion attention layer for fusion to obtain the first fusion features;

[0042] Input the first fused feature into the multi-layer perceptron for feature conversion and nonlinear activation to obtain the activated first fused feature;

[0043] Input the visual feature and the activated first fusion feature into the first fusion attention layer for fusion, and obtain the second fusion feature;

[0044] The activated first fusion feature and the second fusion feature are input into the cross attention layer for fusion to obtain the fusion feature of each electronic component.

[0045] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for identifying the polarity mark of an electronic component is implemented.

[0046] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method for identifying polarity marks of electronic components when executing the computer program.

[0047] The above scheme of the present invention has the following beneficial effects:

[0048] The present invention obtains multiple electronic component image data, coordinate labels for describing the position coordinates of each electronic component, and text descriptions for describing the content of each electronic component image data; inputs the multiple electronic component image data, the coordinate labels of each electronic component, and the text descriptions into a polarity identification recognition model to train the polarity identification recognition model, thereby obtaining a trained polarity identification recognition model; inputs the acquired target electronic component image data and the text descriptions for describing the content of the target electronic component image data into the trained polarity identification recognition model for classification and prediction, thereby obtaining a polarity identification recognition result of the target electronic component, wherein the polarity identification recognition result includes a classification result and a prediction result; the polarity identification recognition model includes a Swin Transformer module for extracting visual features, a Transformer module for extracting text features, a multimodal fusion module for fusing visual features and text features, and a classification prediction module; compared with the prior art, the present invention uses the Swin in the polarity identification recognition model to extract text features, a multimodal fusion module for fusing visual features and text features, and a classification prediction module. The Transformer module and the Transformer module respectively extract the visual features and text features of the image for comparative learning to improve the discrimination of feature extraction; by fusing the visual features and text features through the multimodal fusion module, the model can refer to the global image information during recognition, flexibly learn the complex interdependence between different modalities, and have a deeper understanding and then integrate the semantic connections between different modalities to enhance the feature extraction ability of the model, thereby improving the accuracy of polarity mark recognition.

[0049] Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of a flow chart of an embodiment of the present invention;

[0051] Figure 2 Schematic diagram of the structure of the polarity mark recognition model in an embodiment of the present invention;

[0052] Figure 3 Schematic diagram of the result of the multimodal fusion module in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0055] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a locking connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0056] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0057] In view of the existing problems, the present invention provides a method for identifying polarity marks of electronic components and related equipment.

[0058] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a method for identifying polarity marks of electronic components, comprising:

[0059] Step 1, obtaining a plurality of electronic component image data, a coordinate label for describing the position coordinates of each electronic component, and a text description for describing the content of each electronic component image data;

[0060] Step 2, inputting a plurality of electronic component image data, a coordinate label and a text description of each electronic component into a polarity mark recognition model to train the polarity mark recognition model, thereby obtaining a trained polarity mark recognition model;

[0061] Step 3, inputting the acquired target electronic component image data and the text description used to illustrate the content of the target electronic component image data into the trained polarity identification model for classification and prediction, and obtaining the polarity identification result of the target electronic component, the polarity identification result including the classification result and the prediction result, the classification result is the polarity of the target electronic component, and the prediction result is the identification position of the target electronic component;

[0062] The polarity mark recognition model includes a Swin Transformer module for extracting visual features, a Transformer module for extracting text features, a multimodal fusion module for fusing visual features and text features, and a classification prediction module;

[0063] The input end of the Swin Transformer module is used to receive multiple electronic component image data, and during the training process, the input end of the Transformer module is used to receive the coordinate label and text description of each electronic component. After training, the input end of the Transformer module is used to receive the text description of each electronic component;

[0064] The output of the Swin Transformer module and the output of the Transformer module are connected to the input of the multimodal fusion module;

[0065] The output end of the multimodal fusion module is connected to the input end of the classification prediction module, and the output end of the classification prediction module is used to output the polarity identification recognition result of each electronic component.

[0066] Specifically, step 1 includes:

[0067] Use a high-resolution camera with a resolution of more than 12 million pixels to capture multiple angles and views of electronic components of different models under different backgrounds and lighting conditions, and obtain multiple electronic component image data to ensure that all possible industrial production scenarios are covered to ensure the integrity and clarity of the collected images;

[0068] According to various polarity label definitions in the label system, the polarity mark of the electronic component image is marked with a suitable polarity label, and the coordinate position of the polarity mark of the electronic component is marked to obtain the coordinate label of each electronic component;

[0069] The image data of each electronic component is described with appropriate text to obtain a text description of each electronic component, thereby obtaining a data set containing supervisory information such as coordinate labels and text descriptions.

[0070] It should be noted that the different types of electronic components include 12 categories of data, including electrolytic capacitors, ordinary diodes, light-emitting diodes, transistors, connectors, batteries, field-effect transistors, rectifier bridges, potentiometers, optocouplers, piezoelectric buzzers, transformers, etc., with a total of about 30,000 data items.

[0071] Most preferably, before step 2, the method further comprises:

[0072] Performing data enhancement processing on multiple electronic component image data to obtain multiple enhanced electronic component image data;

[0073] Use regular matching to process the coordinate labels of each electronic component, divide the coordinate labels into sentences, and preprocess each sentence to obtain the preprocessed coordinate labels;

[0074] Use a word segmentation tool to segment the text description of each electronic component to obtain a segmented text description.

[0075] In the embodiment of the present invention, in the data enhancement processing stage of multiple electronic component image data, multiple data enhancement technologies such as rotation, scaling, flipping, color jittering and cropping are used to perform multiple transformations on the original images of the electronic components to generate image pairs with different perspectives and changes, thereby improving the diversity and richness of training samples and enhancing the generalization ability of the model in different environments. The expression is:

[0076]

[0077] Among them, x i represents the original image, T i represents the data enhancement operation, Represents an image after data augmentation;

[0078] The specific transformation process is as follows:

[0079] Rotation: Rotate the image at random angles between 0 and 360 degrees to simulate viewing angles at different angles;

[0080] Scaling: Randomly scale the ratio between 0.8 and 1.2 to generate images of different sizes;

[0081] Flip: Randomly flip horizontally and vertically;

[0082] Color jittering: changing the brightness, contrast, and saturation of the image to enhance the model’s robustness to color changes;

[0083] Cropping: Randomly crop different areas of the image to simulate partial loss of perspective;

[0084] Suppose there are two images I1 and I2 of ordinary diodes, containing positive and negative pole labels. The original image I1 is rotated, scaled, and other image processing combinations to generate A2. Then (I1, A1) and (I2, A2) are training samples generated through different data augmentation operations, thereby improving the diversity of samples.

[0085] In an embodiment of the present invention, the text description is segmented using the jieba word segmentation tool to divide the text description into a sequence of words. The words in the word sequence are then screened according to the stop word list of Harbin Institute of Technology, and the words in the stop word list of Harbin Institute of Technology are deleted and filtered out. Finally, the parts of speech marked by the jieba tool during word segmentation are compared with a custom reserved part-of-speech table, and the words whose parts of speech are not in the reserved part-of-speech table are deleted and filtered out.

[0086] In the embodiment of the present invention, the coordinate label is processed by sentence segmentation using regular matching, and the label text is divided into a set of complete sentences. Then, for each complete sentence, word segmentation, stop word removal, and part-of-speech filtering preprocessing operations similar to the text description are performed.

[0087] Most preferably, the Swin Transformer module is used to extract visual features, and the process of extracting visual features includes:

[0088] For each electronic component image data, the electronic component image data is mapped to a high-dimensional feature space for feature extraction to obtain a shallow visual feature map of each electronic component image data;

[0089] For each shallow visual feature map, the shallow visual feature map is segmented to obtain multiple non-overlapping image window blocks, and each image window block is position-encoded to obtain the encoded visual features;

[0090] For each image window block, the image window block and its internal shallow visual features are calculated to obtain the attention score of each image window block;

[0091] The attention scores of all image window blocks are combined and convolved to obtain the visual features of each electronic component.

[0092] In the embodiment of the present invention, the Swin Transformer module includes multiple convolutional layers and activation functions; for a given electronic component image Among them, H, W and C0 represent the height, width and number of channels of the image respectively; since the electronic component images are photographed and collected in natural scenes, the number of image channels is set to 3, and each pixel point in the pixel matrix of each image has a value of 0-255. If the pixel points are not reduced in dimension, the computational cost and time overhead of the model are particularly high, so the convolution operation is used to perform preliminary feature extraction on the image. By sliding the convolution kernel to cover each area of ​​the pixel matrix, each value in the convolution kernel is multiplied one by one with the pixel value of the corresponding coverage area and summed up to become a pixel point of the new feature map, which effectively reduces the computational overhead of image processing and improves the efficiency of the model in extracting image features; the electronic component image is mapped to a higher-dimensional feature space through multiple convolutional layers, and the expression of the shallow feature map extracted from the electronic component image is:

[0093] T0=Conv i (T e ), i=1,2

[0094] Among them, T0 represents the shallow feature map of electronic components, H, W and C0 represent the height, width and number of channels of the image respectively. Conv i represents the i-th convolution operation, T e Represents an image of an electronic component.

[0095] Since the data in the training set comes from the real world, the distribution of data is usually nonlinear, so an activation function is introduced after image convolution, which can effectively control the flow of information in the model network and improve the network's ability to solve nonlinear problems, thereby enabling the model to solve various complex tasks. For example, the Relu activation function will return all negative values ​​to zero, which can prevent the problem of gradient disappearance to a certain extent, and can maintain the sparsity of the network during training, so that the model network can learn and model complex data patterns and relationships.

[0096] If global attention is performed on the shallow feature map of the image, the model needs to calculate each feature point, which will incur a lot of time cost and computational overhead in the training phase, increasing the training time and parameter amount of the model. Therefore, before entering the multi-head attention layer, the model will divide the shallow feature map T0 of the image into M non-overlapping image window blocks. The size of each image window block is n×n, and attention calculation is performed in each image window block to reduce the overall computational amount of the model. The calculation expression for segmentation is:

[0097]

[0098] Wherein, n represents the width and height of the image window block, and M represents the number of image window blocks.

[0099] Then, a position encoding strategy is used for each image window block to better represent the positional relationship between image blocks. In the initial Swin Transformer module, sine and cosine functions are used to generate a unique code for each position. For a given position pos and a given dimension i, the expression for position encoding of each image window block is:

[0100]

[0101] Among them, PE represents the position encoding, pos represents the position of the element in the sequence, i represents the dimension index of the position encoding, and d model Representing the dimension of the model, this position encoding method can provide the model with absolute information about the position and enable the model to combine the position information with the embedding vector of the input word.

[0102] To ensure that the model can learn and process the position information of the input data, the calculated position encoding vector is added to the input vector through an addition operation. However, the absolute position encoding fixes the maximum length of the sequence, so a learnable position encoding method is adopted on this basis. During the model training process, the model will update the position encoding parameters according to the gradient calculated by the loss function. Through continuous optimization, the position encoding learned by the model will be able to better represent the position information of each element in the sequence, thereby improving the model's effect on sequence data processing.

[0103] In the embodiment of the present invention, for the shallow features in the i-th image window block in the shallow feature map, attention calculation is performed inside each divided image window block to obtain a corresponding attention score. The specific calculation process is as follows:

[0104] Based on the shallow visual features inside the image window block and the shared projection matrix between the image window blocks, a projection matrix is ​​generated, and the expression is:

[0105] Q=T feature(i) T Q

[0106] K=T feature(i) T K

[0107] V=T feature(i) T V

[0108] Among them, T Q 、T K 、T V Both represent the shared projection matrix, T feature(i) Represents the shallow visual features inside the image window block i;

[0109] Attention calculation is performed based on the projection matrix to obtain an attention score, and the expression of the attention score is:

[0110]

[0111] Among them, d represents the feature vector dimension and B represents the learnable positional encoding.

[0112] Introducing a normalization layer after attention calculation can effectively improve model stability and training efficiency, and obtain semantic features, expressed as:

[0113] T LN =LayerNorm(MSA(T feature(i) )+T feature(i) )

[0114] Finally, the local and global information are fused through convolution blocks and residual connections to convert and combine semantic feature information. The calculation formula is as follows:

[0115] Tran=Conv j (MLP(T CN )), j = 1, 2, ..., 4

[0116] Among them, Conv j represents the jth convolution operation, MLP represents a multi-layer perceptron, T CN Represents semantic features;

[0117] The learned deep high-frequency and semantic features are combined through the Swin Transformer module to form the visual features of each electronic component:

[0118] T tran =Tran N (T0), N = 1, 2, ..., 8

[0119] Among them, represents the visual feature vector of electronic components, H, W and C0 represent the height, width and number of channels of the image respectively, and T0 represents the shallow feature map of the electronic components.

[0120] Most preferably, the Transformer module is used to extract text features, and the process of extracting text features is:

[0121] Download the text encoder model and model parameters from the pre-trained neural network model;

[0122] Initialize and freeze the embedding layer parameters in the text encoder model using the model parameters;

[0123] The coordinate label and text description of each electronic component are input into the embedding layer of the text encoder model for transformation to obtain the text features of each electronic component.

[0124] In the embodiment of the present invention, the coordinate label and text description of each electronic component are vectorized by using a text encoder in a pre-trained neural network model (CLIP, Contrastive Language-Image Pre-training), and the text label is decomposed into sub-word units by a word segmenter, mapped to the corresponding word embedding vector to form an input sequence. In order to ensure the order of the character sequence, the position encoding of the traditional Transformer module is adopted, and the position encoding is generated for each pixel through sine and cosine functions, as shown in the formula:

[0125] L pos =Pos(L e )

[0126] After position encoding, the encoded text feature L is obtained pos , and send it to the encoder for sequence feature extraction; in order to speed up the training of the model, multi-head attention is used for parallel calculation of attention scores, and the final text features are obtained by splicing. The input sequence passes through the multi-head attention mechanism to calculate the dependency between each word and other words in the input sequence. The multi-head attention mechanism allows the model to perform attention calculations in different subspaces, thereby capturing richer information. The expression is as follows:

[0127] L att =Attention(L pos )

[0128] Among them, L att Indicates the final text features obtained by concatenation, Attention(·) indicates the attention calculation function;

[0129] In order to ensure the stability of training and speed up the training, L pos Normalization is performed to distribute the data within a specific range, and then the normalized data is fed into the feedforward neural network, so that the network can capture more complex dependencies in the data, help the model learn features at a deeper level, and improve the stability of training. The expression is:

[0130] L LN =FNN(LayerNorm(L att ))

[0131] Wherein, LayerNorm(·) represents the normalization function, and FNN represents the feedforward neural network;

[0132] After being processed by the Transformer module, the generated embedding sequence of each word will be further processed to obtain a deeper text feature L text , the expression is:

[0133] L text =Encoder N (L pos )

[0134] Among them, Encoder N Represents the Nth encoder.

[0135] In the embodiment of the present invention, feature extraction is performed through the Swin Transformer module and the Transformer module to obtain the visual feature T tran and text feature L text Then conduct comparative learning. The specific process is as follows:

[0136] In order to further align the visual features and text features, the similarity of all visual features and text features of each input batch is calculated by the dot product of the vectors, thereby generating an N×N similarity matrix, which represents the size of the batch. Each element M in the matrix ij Represents the similarity between an image and a text in a batch, expressed as:

[0137]

[0138] Among them, I i and L j They represent the feature vectors of an image i and a text j respectively, and |·| represents the norm of the vector.

[0139] Specifically, the embodiment of the present invention constructs the positive and negative sample pairs corresponding to the image by generating a similarity matrix containing all positive and negative sample pairs, where each element M ij Denotes the feature vector I i and L j The cosine similarity between the two, the elements on the diagonal represent positive sample pairs, which are correctly matched image and text pairs; while the elements on the non-diagonal represent negative sample pairs, which are incorrectly matched image and text pairs. Assume I i and I j Represented as the visual feature vector of the two images, L i and L j Represents image I i and I j Corresponding labels, the method of constructing positive and negative sample pairs is as follows:

[0140]

[0141] By constructing positive and negative samples in this way, the time and cost of manual labeling can be reduced and the generalization ability of the model can be improved.

[0142] Specifically, the embodiment of the present invention calculates the similarity between positive and negative sample pairs through a contrast loss function, and improves the recognition of similar features by maximizing the similarity between similar features and minimizing the similarity between different features, so that the image and text pairs of positive samples are closer in the common embedding space, and the image and text pairs of negative samples are farther apart, so as to optimize the model parameters. For a positive sample pair, the similarity between its feature vectors should be maximized; and for a negative sample pair, the similarity between its feature vectors should be minimized. The similarity calculation expression is:

[0143]

[0144] Where τ represents the temperature coefficient, z i and z j is the feature vector in the embedding space.

[0145] Most preferably, Figure 3 As shown, the multimodal fusion module includes:

[0146] Self-attention layer, cross-attention layer, first fusion attention layer, second fusion attention layer, multi-layer perceptron;

[0147] The first input end of the first fused attention layer and the first input end of the second fused attention layer are both connected to the output end of the SwinTransformer module;

[0148] The output of the first fusion attention layer is connected to the first input of the cross attention layer;

[0149] The output of the second fusion attention layer is connected to the input of the multilayer perceptron;

[0150] The output end of the multilayer perceptron is connected to the second input end of the first fusion attention layer and the second input end of the cross attention layer respectively;

[0151] The input end of the self-attention layer is connected to the output end of the Transformer module, and the output end of the self-attention layer is connected to the second input end of the second fusion attention layer;

[0152] The output of the criss-cross attention layer is connected to the input of the classification prediction module.

[0153] Most preferably, the multimodal fusion module is used to fuse visual features and text features, and the fusion process includes:

[0154] For text feature L textPerform self-attention calculation to capture the dependencies and global features between word meta-tokens and obtain the text feature L after self-attention calculation. score ;

[0155] The text features and visual features after self-attention calculation are input into the second fusion attention layer for fusion to obtain the first fusion features;

[0156] In order to fuse the feature information between text and image, the second fusion attention layer is used to calculate text features and visual features, where text features are used as query and visual features are used as key and value, so as to obtain the first fusion feature L cross , the expression is:

[0157]

[0158]

[0159] Among them, Q i The query vector representing the text features, K t and W t Key-value pairs representing image features;

[0160] The first fusion feature L cross Input the multi-layer perceptron for feature conversion and nonlinear activation to obtain the first fusion feature after activation, the expression is:

[0161] Z = ReLU(W1X+b1)

[0162] O=W2Z+b2

[0163] Among them, W1 and W2 represent learnable weight matrices, and b1 and b2 represent learnable bias vectors;

[0164] Input the visual feature and the activated first fusion feature into the first fusion attention layer for fusion, and obtain the second fusion feature;

[0165] For visual features, in order to fuse the feature information between text and image, the image embedding information is passed to the token, and the text features and visual features are calculated through the first fusion attention layer, where the visual features are used as query and the text features are used as key and value, so as to obtain the second fusion feature T cross , thereby aggregating the global image information and token information;

[0166] The activated first fusion feature and the second fusion feature are input into the cross attention layer for fusion to obtain the fusion feature of each electronic component.

[0167] In order to fully integrate the image and token information, the fused L cross and T cross Perform cross attention calculation to obtain the fusion feature T of each electronic component att For the fused L cross and T cross , T cross It comes from the visual feature extraction at the image level, while L cross It is obtained by feature extraction of text sequences. In order to fuse the features between the visual and textual modalities, a cross-attention layer is used for processing and fusion. This allows the model to focus on the related features in the two different modalities and learn how to match the image area with the relevant words in the descriptive text, thereby effectively combining these features to complete the polarity mark recognition task.

[0168] By constructing a learnable weight matrix W q , W k , W v Generate Q, K, and V, where the text feature is used as the query and the visual feature is used as the key and value. Then, the product of each feature vector in the text feature and the visual feature vector in the image is calculated, and the product is normalized using the softmax function to obtain the normalized feature V. score , after obtaining the attention weight, V score Multiply it by value to get the fusion feature T of each electronic component. att , as shown in the formula:

[0169] T att =Cross(I cross , T cross )

[0170] This feature fusion method allows the model to refer to global image information during recognition, flexibly learn complex interdependencies between different modalities, understand and integrate the semantic connections between different modalities at a deeper level, and enhance the model's ability to process text sequences and their positional relationships.

[0171] The classification prediction module in the embodiment of the present invention includes a multi-layer perceptron and a convolutional layer, wherein the input end of the multi-layer perceptron and the input end of the convolutional layer are both connected to the output end of the cross-attention layer, the multi-layer perceptron outputs the polarity mark recognition result, and the convolutional layer outputs the bounding box position of the polarity mark.

[0172] The embodiment of the present invention adds a Dropout (random inactivation) layer after generating the visual feature vector to reduce the overfitting problem during training, so that the final training model has stronger generalization ability. Since the uneven number of samples of each label will affect the training effect, the category loss function is used to assign different weights to the training samples to reduce the impact caused by the uneven number of samples of different labels. The process is as follows:

[0173] In order to improve the classification and migration capabilities of the model, corresponding descriptions are added to all labels and encoded through the CLIP text encoder to form corresponding label vectors. Different from ordinary classification tasks, the product of the feature vector extracted by the model and the label vector is calculated through the category loss to determine the final category.

[0174] In order to evaluate the model's ability to detect and identify polarity marks, the model's loss is calculated using category loss and boundary regression loss to optimize the model parameters and results. The category loss function is expressed as:

[0175]

[0176] where y i is the true label of the i-th category, p i is the predicted probability.

[0177] The expression of the boundary regression loss function is:

[0178]

[0179] Among them, t i are the predicted bounding box parameters, are the corresponding ground-truth bounding box parameters;

[0180] Use stochastic gradient descent and Adam optimizer to update the model structure and parameters.

[0181] Specifically, step 3 includes:

[0182] The acquired target electronic component image data and the text description of the target electronic component are input into the trained polarity identification model for identification, classification and prediction. An ordinary diode is taken as the target electronic component, and the image data of the ordinary diode and the text description of the content of the image data of the ordinary diode are input. For example, the text description is: This is a picture containing the positive and negative polarity identification of an ordinary diode. The polarity identification recognition result is the position coordinates of the positive pole identification of the target electronic component, where the classification result is the positive pole, and the prediction result is the position coordinates corresponding to the polarity identification.

[0183] The model output is converted into result labels according to the threshold of 0.8;

[0184] Calculate the IOU score and F1 Score indicators as indicators for evaluating multi-label classification accuracy:

[0185]

[0186] Among them, A is the predicted bounding box, B is the real bounding box, A∩B represents the intersection area of ​​the predicted bounding box and the real bounding box, and A∪B represents the union area of ​​the predicted bounding box and the real bounding box.

[0187] The embodiment of the present invention obtains multiple electronic component image data, coordinate labels for describing the position coordinates of each electronic component, and text descriptions for describing the content of each electronic component image data; inputs the multiple electronic component image data, the coordinate labels and text descriptions of each electronic component into a polarity identification recognition model to train the polarity identification recognition model, thereby obtaining a trained polarity identification recognition model; inputs the acquired target electronic component image data and the text descriptions for describing the content of the target electronic component image data into the trained polarity identification recognition model for classification and prediction, thereby obtaining classification results and prediction results of the target electronic component, and obtaining polarity identification recognition results based on the classification results and predictions of the target electronic component; the polarity identification recognition model includes a SwinTransformer module for extracting visual features, a Transformer module for extracting text features, a multimodal fusion module for fusing visual features and text features, and a classification prediction module; compared with the prior art, the embodiment of the present invention uses the Swin in the polarity identification recognition model to extract text features, a multimodal fusion module for fusing visual features and text features, and a classification prediction module. The Transformer module and the Transformer module respectively extract the visual features and text features of the image for comparative learning to improve the discrimination of feature extraction; by fusing the visual features and text features through the multimodal fusion module, the model can refer to the global image information during recognition, flexibly learn the complex interdependence between different modalities, and have a deeper understanding and then integrate the semantic connections between different modalities to enhance the feature extraction ability of the model, thereby improving the accuracy of polarity mark recognition.

[0188] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a method for identifying polarity marks of electronic components is implemented.

[0189] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above method embodiment, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the construction device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, disk or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0190] An embodiment of the present invention further provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method for identifying polarity marks of electronic components when executing the computer program.

[0191] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a server, a server cluster, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0192] The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0193] In some embodiments, the memory may be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device. In other embodiments, the memory may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device. Further, the memory may also include both an internal storage unit of the terminal device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (Boot Loader), data and other programs, such as the program code of the computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0194] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the embodiment of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0195] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0196] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for identifying polarity marks of electronic components, characterized in that: include: Step 1, obtaining a plurality of electronic component image data, a coordinate label for describing the position coordinates of each electronic component, and a text description for describing the content of each electronic component image data; Step 2, inputting a plurality of electronic component image data, a coordinate label and a text description of each electronic component into a polarity mark recognition model to train the polarity mark recognition model, thereby obtaining a trained polarity mark recognition model; Step 3, inputting the acquired target electronic component image data and the text description used to describe the content of the target electronic component image data into the trained polarity mark recognition model for classification and prediction, and obtaining the polarity mark recognition result of the target electronic component, wherein the polarity mark recognition result includes a classification result and a prediction result; The polarity mark recognition model includes a Swin Transformer module for extracting visual features, a Transformer module for extracting text features, a multimodal fusion module for fusing visual features and text features, and a classification prediction module; The input end of the Swin Transformer module is used to receive a plurality of electronic component image data, the input end of the Transformer module is used to receive the coordinate label and text description of each electronic component during the training process, and after the training, the input end of the Transformer module is used to receive the text description of each electronic component; The output end of the Swin Transformer module and the output end of the Transformer module are both connected to the input end of the multimodal fusion module; The output end of the multimodal fusion module is connected to the input end of the classification prediction module, and the output end of the classification prediction module is used to output the polarity identification recognition result of each electronic component.

2. The method for identifying polarity marks of electronic components according to claim 1, characterized in that: Before step 2, the method further includes: Performing data enhancement processing on multiple electronic component image data to obtain multiple enhanced electronic component image data; Using regular matching to perform sentence processing on the coordinate label of each electronic component, dividing the coordinate label into sentences, and preprocessing each sentence to obtain a preprocessed coordinate label; Use a word segmentation tool to segment the text description of each electronic component to obtain a segmented text description.

3. The method for identifying polarity marks of electronic components according to claim 2, characterized in that: The SwinTransformer module is used to extract visual features, and the process of extracting the visual features includes: For each electronic component image data, the electronic component image data is mapped to a high-dimensional feature space for feature extraction to obtain a shallow visual feature map of each electronic component image data; For each shallow visual feature map, the shallow visual feature map is segmented to obtain a plurality of non-overlapping image window blocks, and each image window block is position-encoded to obtain encoded visual features; For each image window block, the image window block and the shallow visual features inside it are calculated to obtain the attention score of each image window block; The attention scores of all image window blocks are combined and convolved to obtain the visual features of each electronic component.

4. The method for identifying polarity marks of electronic components according to claim 3, characterized in that: The expression for position encoding of each image window block is: Among them, PE represents the position encoding, pos represents the position of the element in the sequence, i represents the dimension index of the position encoding, and d model Represents the dimensions of the model.

5. The method for identifying polarity marks of electronic components according to claim 4, characterized in that: The visual characteristics of each electronic component are: T tran =Tran N (T0),N=1,2,…,8 Among them, represents the visual feature vector of electronic components, H, W and C0 represent the height, width and number of channels of the image respectively, and T0 represents the shallow feature map of the electronic components.

6. The method for identifying polarity marks of electronic components according to claim 5, characterized in that: The Transformer module is used to extract text features, and the process of extracting the text features is as follows: Download the text encoder model and model parameters from the pre-trained neural network model; Initializing and freezing embedding layer parameters in the text encoder model using the model parameters; The coordinate label and text description of each electronic component are input into the embedding layer of the text encoder model for conversion to obtain the text features of each electronic component.

7. The method for identifying polarity marks of electronic components according to claim 6, characterized in that: The multimodal fusion module comprises: Self-attention layer, cross-attention layer, first fusion attention layer, second fusion attention layer, multi-layer perceptron; The first input end of the first fused attention layer and the first input end of the second fused attention layer are both connected to the output end of the Swin Transformer module; The output end of the first fusion attention layer is connected to the first input end of the cross attention layer; The output end of the second fusion attention layer is connected to the input end of the multilayer perceptron; The output end of the multilayer perceptron is connected to the second input end of the first fusion attention layer and the second input end of the cross attention layer respectively; The input end of the self-attention layer is connected to the output end of the Transformer module, and the output end of the self-attention layer is connected to the second input end of the second fusion attention layer; The output end of the cross attention layer is connected to the input end of the classification prediction module.

8. The method for identifying polarity marks of electronic components according to claim 7, characterized in that: The multimodal fusion module is used to fuse visual features and text features. The fusion process includes: Performing self-attention calculation on the text features to capture dependencies and global features between word units, and obtaining text features after self-attention calculation; Inputting the text features after self-attention calculation and the visual features into the second fusion attention layer for fusion to obtain a first fusion feature; Inputting the first fused feature into the multi-layer perceptron for feature conversion and nonlinear activation to obtain an activated first fused feature; Inputting the visual feature and the activated first fused feature into the first fused attention layer for fusion to obtain a second fused feature; The activated first fusion feature and the second fusion feature are input into the cross attention layer for fusion to obtain the fusion feature of each electronic component.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for identifying the polarity mark of an electronic component as claimed in any one of claims 1 to 8 is implemented.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for identifying the polarity mark of an electronic component according to any one of claims 1 to 8 is implemented.

Images