Chip screen printing information identification method and device, electronic equipment and storage medium
Through the ResNet and Transformer model combining the cross attention mechanism and CLIP text encoder, the chip silk screen information recognition model is optimized, which solves the problem of cumbersome identification process and low accuracy in the existing technology, and improves the recognition efficiency and accuracy of chip silk screen information recognition .
Patent Information
- Application Number
- CN202411040076.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-31
AI Technical Summary
The process of identifying existing chip silk screen printing information is cumbersome, and low resolution images lead to low recognition efficiency and accuracy, affecting chip inventory management and production scheduling.
The ResNet and Transformer model are combined with the cross attention mechanism, and through feature extraction and masking technology, visual and language features are integrated, the silk screen information recognition model is optimized, and similar words are expanded using CLIP text encoder to improve recognition accuracy.
Reduce the dependence of manually defining templates, improve the recognition efficiency and accuracy of chip silk screen printing information, especially in low-resolution scenarios.
Smart Images

Figure CN118941816A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of Internet technology, and in particular to a chip silk screen information recognition method, device, electronic device and storage medium. Background Art
[0002] In the field of smart chip manufacturing, chip silk screen information can be used to quickly identify and classify chips of different models and batches, improve production efficiency and automation level, and help achieve efficient inventory management and production scheduling.
[0003] However, the existing technology mainly detects the position information of each character in the chip image through text detection and segmentation technology, segments each character in turn according to the position information, and finally performs character query and matching through manually defined templates to predict and generate a single predicted character. When faced with low-resolution chip images, since the character shapes in the chip image are usually fuzzy and there is a certain image occlusion between the characters, it will not only interfere with the accuracy of text detection, but also affect the classification and prediction generation of characters in the recognition. Therefore, the recognition process of the existing chip silk screen information is cumbersome, which is not conducive to improving the recognition efficiency and recognition accuracy of the chip silk screen information. Summary of the invention
[0004] The embodiments of the present application provide a chip silk screen information recognition method, device, electronic device and storage medium to solve the technical problem that the above-mentioned existing chip silk screen information recognition process is cumbersome and is not conducive to improving the recognition efficiency and recognition accuracy of chip silk screen information.
[0005] In a first aspect, an embodiment of the present application provides a method for identifying chip silk screen information, which is applied to an electronic device, wherein the electronic device stores a silk screen information recognition model, wherein the silk screen information recognition model includes a ResNet model and a Transformer model, and wherein the method for identifying chip silk screen information includes:
[0006] Obtaining a preset chip image and preset text description information corresponding to the preset chip image;
[0007] Using the ResNet model and the Transformer model, feature extraction is performed on the preset chip image to obtain visual features corresponding to the preset chip image; using the Transformer model, feature extraction is performed on the preset text description information to obtain language features corresponding to the preset chip image;
[0008] Based on the cross attention mechanism, the visual feature and the language feature are fused to obtain a fused feature, a sequence of the fused features is obtained, a position in the sequence is randomly selected, and the visual feature before the position is used as the visual feature that needs to be masked;
[0009] Classifying the masked visual features to generate a character sequence, mapping the character sequence into a text vector through a text encoder of a pre-trained model, and obtaining similar words generated by the pre-trained model based on the text vector;
[0010] Optimizing the silk screen information recognition model through the CLIP text encoder and the similar words;
[0011] Using the optimized silk screen information recognition model, the predicted silk screen information is obtained, and according to the cross entropy loss value between the predicted silk screen information and the real silk screen information, the silk screen information recognition model is trained;
[0012] When the cross entropy loss value is less than the preset loss value, the silk screen information recognition result output by the trained silk screen information recognition model is obtained.
[0013] In a possible implementation manner of the first aspect, based on the cross attention mechanism, the visual feature and the language feature are fused to obtain a fused feature, a sequence of the fused features is acquired, a position in the sequence is randomly selected, and the visual feature before the position is used as the visual feature that needs to be masked, including:
[0014] Based on the cross attention mechanism, the visual feature and the language feature are fused to obtain the fused feature, a sequence of the fused features is obtained, and a position in the sequence is randomly selected;
[0015] An attention score map is obtained, and according to the attention score map, the visual features before the position are used as the visual features that need to be masked.
[0016] In a possible implementation manner of the first aspect, classifying the masked visual features to generate a character sequence, mapping the character sequence into a text vector through a text encoder of a pre-trained model, and obtaining a similar word generated by the pre-trained model based on the text vector includes:
[0017] Replacing the feature value of the visual feature to be masked with the selected mark to obtain the masked visual feature;
[0018] Sending the masked visual features to a linear layer, and classifying the masked visual features by the classification function to generate a character sequence corresponding to the masked visual features;
[0019] The character sequence is mapped into a text vector through a text encoder of a pre-trained model, and similar words generated by the pre-trained model based on the text vector are obtained.
[0020] In a possible implementation manner of the first aspect, optimizing the silk screen information recognition model by using a CLIP text encoder and the similar words includes:
[0021] Encode the similar words by using the CLIP text encoder to obtain text features corresponding to the similar words, use the cosine similarity formula to calculate the similarity between the visual features and the text features, and construct a similarity matrix by using multiple similarities;
[0022] In the similarity matrix, the visual features and the text features whose similarity is higher than a preset value are regarded as positive sample pairs, and the visual features and the text features whose similarity is not higher than the preset value are regarded as negative sample pairs;
[0023] The silk screen information recognition model is optimized by using a contrast loss function with the goal of minimizing the distance between the positive sample pairs and maximizing the distance between the negative sample pairs.
[0024] In a possible implementation manner of the first aspect, when the cross entropy loss value is less than a preset loss value, obtaining the silk screen information recognition result output by the trained silk screen information recognition model includes:
[0025] When the cross entropy loss value is less than the preset loss value, stop training the silk screen information recognition model, and save the trained silk screen information recognition model;
[0026] Obtaining the recognition accuracy of the trained silk screen information recognition model on the validation set;
[0027] When the recognition accuracy is greater than the preset accuracy, the current chip image and the current text description information corresponding to the current chip image are obtained, the current chip image and the current text description information are input into the trained silk screen information recognition model, and the silk screen information recognition result output by the trained silk screen information recognition model is obtained.
[0028] In a possible implementation manner of the first aspect, when the cross entropy loss value is less than a preset loss value, after obtaining the silk screen information recognition result output by the trained silk screen information recognition model, the chip silk screen information recognition method includes:
[0029] A display window is created, and the current text description information is displayed in the display window.
[0030] In a possible implementation manner of the first aspect, the pre-trained model includes one or a combination of a Word2Vec model and a BRET model.
[0031] In a second aspect, an embodiment of the present application provides a chip silk screen information recognition device, which is applied to an electronic device, wherein the electronic device stores a silk screen information recognition model, and the silk screen information recognition model includes a ResNet model and a Transformer model, including:
[0032] A first acquisition module, used to acquire a preset chip image and preset text description information corresponding to the preset chip image;
[0033] An extraction module, used to perform feature extraction on the preset chip image using the ResNet model and the Transformer model to obtain visual features corresponding to the preset chip image, and to perform feature extraction on the preset text description information using the Transformer model to obtain language features corresponding to the preset chip image;
[0034] A selection module is used to fuse the visual feature and the language feature based on a cross attention mechanism to obtain a fused feature, obtain a sequence of the fused features, randomly select a position in the sequence, and use the visual feature before the position as the visual feature that needs to be masked;
[0035] A second acquisition module is used to classify the masked visual features to generate a character sequence, map the character sequence into a text vector through a text encoder of a pre-trained model, and obtain similar words generated by the pre-trained model based on the text vector;
[0036] An optimization module, used for optimizing the silk screen information recognition model through a CLIP text encoder and the similar words;
[0037] A third acquisition module is used to use the optimized silk screen information recognition model to obtain predicted silk screen information, and train the silk screen information recognition model according to the cross entropy loss value between the predicted silk screen information and the real silk screen information;
[0038] The recognition module is used to obtain the silk screen information recognition result output by the trained silk screen information recognition model when the cross entropy loss value is less than a preset loss value.
[0039] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the chip silk screen information recognition method of the first aspect when executing the computer program.
[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the chip silk screen information recognition method of the first aspect is implemented.
[0041] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the chip silk screen information recognition method of the first aspect.
[0042] The beneficial effects of the embodiments of the present application lie in two aspects. On the one hand, when the cross entropy loss value is less than the preset loss value, the silk screen information recognition result output by the trained silk screen information recognition model is obtained. Since there is no need to manually define the template, the recognition time of the chip silk screen information is reduced, which is beneficial to improving the recognition efficiency of the chip silk screen information. On the other hand, the visual features are masked by feature masking technology, and the silk screen information recognition model is trained to recognize silk screen information within limited visual features. Similarly, similar word expansion is used to provide multiple possible candidates for each predicted character sequence, thereby improving the prediction ability of the silk screen information recognition model in low-resolution scenarios, thereby improving the recognition accuracy of the chip silk screen information by the silk screen information recognition model. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0044] Figure 1 An application scenario diagram of the chip silk screen information recognition method provided in an embodiment of the present application;
[0045] Figure 2 It is a flow chart of a method for identifying chip silk screen information provided in an embodiment of the present application;
[0046] Figure 3 A flowchart for obtaining current text description information provided in an embodiment of the present application;
[0047] Figure 4 A schematic block diagram of a chip silk screen information recognition device provided in an embodiment of the present application;
[0048] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0050] In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0051] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0052] The chip silk screen information recognition method provided in the embodiment of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific type of electronic devices.
[0053] See also Figure 1 , Figure 1 The application scenario diagram of the chip silk screen information recognition method provided in the embodiment of the present application is described in detail as follows:
[0054] The electronic device is connected to a database via a preset network, and obtains a preset chip image and preset text description information corresponding to the preset chip image from the database.
[0055] The preset network includes one or a combination of Ethernet, WIFI network, 4G network, and 5G network.
[0056] Among them, WIFI network is a wireless fidelity network.
[0057] Among them, 4G network is the fourth generation mobile communication technology network.
[0058] Among them, 5G network is the fifth generation mobile communication technology network.
[0059] In an embodiment of the present application, the electronic device can be connected to a database to obtain the labeled data of the target chip from the database, which is conducive to improving the efficiency of obtaining the labeled data.
[0060] See also Figure 2 , Figure 2 It is a flow chart of the chip silk screen information recognition method provided in an embodiment of the present application. The method can be applied to an electronic device, wherein the electronic device stores a silk screen information recognition model, and the silk screen information recognition model includes a ResNet model and a Transformer model.
[0061] Among them, the ResNet model is a model based on ResNet.
[0062] Among them, ResNet (Residual Network) is a deep neural network architecture.
[0063] Among them, the Transformer model is a neural network model based on the self-attention mechanism.
[0064] Among them, the silk screen information recognition model is a recognition model of silk screen information.
[0065] like Figure 2 As shown, the chip silk screen information recognition method provided in the embodiment of the present application includes the following steps, which are described in detail as follows:
[0066] S201, obtaining a preset chip image and preset text description information corresponding to the preset chip image;
[0067] Exemplarily, obtaining a preset chip image and preset text description information corresponding to the preset chip image includes:
[0068] Sample data of a data set is obtained, and in the sample data, the preset chip image and preset text description information corresponding to the preset chip image are obtained.
[0069] Wherein, before obtaining sample data of a data set and obtaining the preset chip image and preset text description information corresponding to the preset chip image in the sample data, the chip silk screen information recognition method includes:
[0070] Use cameras to capture multiple angles and views of different chip models under different backgrounds and lighting conditions to ensure coverage of all possible industrial production scenarios and to ensure the integrity of the captured images;
[0071] According to various label definitions in the label system, the silk-screen content in the preset chip image is marked with a suitable silk-screen information label, and the silk-screen information label and its coordinate position are combined into a preset text description information;
[0072] The preset chip image and the preset text description information corresponding to the preset chip image are taken as a sample data, and a plurality of sample data are combined into a data set.
[0073] S202, using the ResNet model and the Transformer model, performing feature extraction on the preset chip image to obtain visual features corresponding to the preset chip image, and using the Transformer model, performing feature extraction on the preset text description information to obtain language features corresponding to the preset chip image;
[0074] Among them, the preset chip image and the preset text description information are preprocessed, and the ResNet model and the Transformer model are used to extract features of the preprocessed preset chip image, and the Transformer model is used to extract features of the preprocessed preset text description information to obtain the language features corresponding to the preset chip image.
[0075] The preprocessing of the preset chip image and the preset text description information includes:
[0076] Perform multiple transformations on the preset chip images to generate image pairs with different perspectives and changes to improve the generalization ability of the model;
[0077]
[0078] Among them, x i represents the original image, T represents the image enhancement operation, Represents the image after image enhancement.
[0079] The image enhancement operation is as follows:
[0080] Rotate: Rotates the image at random angles between 0 and 360 degrees, simulating viewing angles at different angles.
[0081] Scaling: Randomly scale the image with a factor between 0.8 and 1.5 to generate images of different sizes.
[0082] Flip: Perform random flips in horizontal and vertical directions.
[0083] Cropping: Randomly crop different areas of the image to simulate partial loss of perspective;
[0084] Among them, for the text description information, sentence processing is first performed, and then each sentence is segmented, stop words are removed, and part-of-speech filtering is performed. The word segmentation tool is used for word segmentation and part-of-speech filtering, and the stop word list of Harbin Institute of Technology is used to remove stop words.
[0085] The ResNet model and the Transformer model are used to extract visual sequence features of a preset chip image. Specifically, the chip image is convolved by two-dimensional convolution, and while reducing the image size, the preset chip image is mapped to a higher-dimensional space to facilitate the reading of deep features.
[0086] After multiple convolution operations, the input result and the result after convolution are added to obtain the image features of the chip image, thereby alleviating the problem of gradient disappearance and explosion caused by too many model layers.
[0087] After passing through the ResNet module, preliminary visual features are obtained, and some pixels in the feature map are selected by a certain random probability for random occlusion. The random probability is 0.65-0.85, and its value is set to 0. By introducing two random probabilities, the model's recognition ability is trained when visual features are insufficient, thereby improving the generalization ability of the model.
[0088] In order to extract the sequence features of the preset chip image, the visual feature size after occlusion is transposed and merged into a three-dimensional channel, which is kept consistent with the input dimension of the Transformer and input into the encoder of the Transformer to extract the features of the image sequence.
[0089] For visual features, in order to ensure the order of the character sequence in the preset chip image, the traditional Transformer position encoding is used to generate a position code for each pixel through sine and cosine functions.
[0090] After position encoding, the encoded visual features are obtained and sent to the encoder for sequence feature extraction, as shown in the formula. In order to speed up the training of the model, multi-head attention is used to perform parallel calculation of attention scores, and the final attention score is obtained by splicing.
[0091] In order to ensure the stability of training and speed up the training, the encoded visual features will be normalized so that the data is distributed within a specific range. The normalized data will then be sent to the feedforward neural network, allowing the feedforward neural network to capture more complex dependencies in the data and improve the generalization prediction ability of the silk screen information recognition model.
[0092] After passing through multiple Transformer encoders, the silk screen information recognition model obtains deeper visual sequence features.
[0093] S203, based on a cross attention mechanism, fusing the visual feature and the language feature to obtain a fused feature, obtaining a sequence of the fused features, randomly selecting a position in the sequence, and using the visual feature before the position as the visual feature that needs to be masked;
[0094] The method of fusing the visual features and the language features based on the cross attention mechanism to obtain a fused feature, obtaining a sequence of the fused features, randomly selecting a position in the sequence, and using the visual feature before the position as the visual feature that needs to be masked includes:
[0095] Based on the cross attention mechanism, the visual feature and the language feature are fused to obtain the fused feature, a sequence of the fused features is obtained, and a position in the sequence is randomly selected;
[0096] An attention score map is obtained, and according to the attention score map, the visual features before the position are used as the visual features that need to be masked.
[0097] For the sake of illustration, an example is given below:
[0098] For the visual feature V text and language features L text , V text It comes from the visual feature extraction at the image level, while L text It is obtained by extracting features from text sequences. In order to fuse the features between the visual and textual modalities, the cross-attention mechanism is used for processing and fusion. This allows the model to focus on the related features in the two different modalities and learn how to match image areas with related words in the descriptive text, thereby effectively combining these features to complete the text recognition task.
[0099] In order to perform the cross attention mechanism, the visual features and language features are projected to the corresponding dimensions respectively, and a learnable weight matrix W is constructed. q , W k , W v , generate three vectors: Query, Key, and Value.
[0100] Among them, Query, Key and Value can be understood as query, key and value in Chinese respectively.
[0101] Among them, text is used as query and visual features are used as key and value.
[0102] Then, by calculating the product of each feature vector in the text and the visual feature vector in the image, the softmax function is used to normalize the product result to obtain the attention score V score , after obtaining the attention weight, V score Multiply it by value to get the final attention result V cross , as shown in the formula.
[0103] V cross =Cross(V text ,L text ).
[0104] Among them, the visual feature is V text , the language feature is L text , V cross For attention results.
[0105] S204, classifying the masked visual features to generate a character sequence, mapping the character sequence into a text vector through a text encoder of a pre-trained model, and obtaining similar words generated by the pre-trained model based on the text vector;
[0106] The pre-trained model includes one or a combination of a Word2Vec model and a BRET model.
[0107] The step of classifying the masked visual features to generate a character sequence, mapping the character sequence into a text vector through a text encoder of a pre-trained model, and obtaining similar words generated by the pre-trained model based on the text vector includes:
[0108] Replacing the feature value of the visual feature to be masked with the selected mark to obtain the masked visual feature;
[0109] Among them, random probability is used to obtain the index of the visual feature that needs to be masked, and then replaced with the corresponding mark. For example, if the character sequence selects the ninth position, the first k visual features will be found in descending order of the attention score of the ninth position, and then the visual feature index is obtained by random probability to replace the visual feature.
[0110] This method randomly blocks part of the image features, allowing the model to learn more robust feature representations and extract key image information in the presence of noisy backgrounds, thereby improving the model's anti-interference and generalization capabilities. Since the mask strategy is only performed during the training phase, it does not bring additional time overhead to the testing and reasoning phases of the silk screen information recognition model.
[0111] Sending the masked visual features to a linear layer, and classifying the masked visual features by the classification function to generate a character sequence corresponding to the masked visual features;
[0112] The character sequence is mapped into a text vector through a text encoder of a pre-trained model, and similar words generated by the pre-trained model based on the text vector are obtained.
[0113] For the sake of illustration, an example is given below:
[0114] The masked visual feature V mask The characters are sent to the linear layer and classified by softmax to generate the corresponding character probability scores. The characters with the highest probability scores are selected and combined to generate the predicted character sequence seq t ;
[0115] Using the pre-trained Word2Vec model, the predicted character sequence seq t Mapped to text vector seq c , capturing the semantic similarity between characters, and generating similar words S of multiple characters through the BRET encoder t Perform expansion between sequences;
[0116] Similar words S of multiple character sequences 1 ,S 2 ,S 3 ,S 4 …S t , the text vector is encoded through the CLIP text encoder to obtain the corresponding text feature C 1 ,C 2 ,C 3 ,C 4 …C t ;
[0117] S205, optimizing the silk screen information recognition model through the CLIP text encoder and the similar words;
[0118] Among them, CLIP text encoder, the full name is Contrastive Language-Image Pre-Training text encoder.
[0119] The step of optimizing the silk screen information recognition model by using the CLIP text encoder and the similar words includes:
[0120] Encode the similar words by using the CLIP text encoder to obtain text features corresponding to the similar words, use the cosine similarity formula to calculate the similarity between the visual features and the text features, and construct a similarity matrix by using multiple similarities;
[0121] In the similarity matrix, the visual features and the text features whose similarity is higher than a preset value are regarded as positive sample pairs, and the visual features and the text features whose similarity is not higher than the preset value are regarded as negative sample pairs;
[0122] The silk screen information recognition model is optimized by using a contrast loss function with the goal of minimizing the distance between the positive sample pairs and maximizing the distance between the negative sample pairs.
[0123] For the sake of illustration, an example is given below:
[0124] For each input batch n, in order to further transform the visual features V mask And text feature seq c Alignment, calculate the similarity of all visual features and text features of the batch by the cosine of the vector, and generate an n×n similarity matrix s.
[0125] The similarity calculation model is as follows:
[0126]
[0127] Among them, sim(v i ,s j ) indicates v i ,s j The similarity between i represents the visual features of the i-th chip image, v j represents the visual features of the jth chip image, s i and j Represents the image features v i and v j The corresponding text features.
[0128] The positive and negative sample pairs corresponding to the image are constructed by generating a similarity matrix s, where each element s ij Represents the visual feature vector v i and the text feature vector s j The cosine similarity between them, the elements on the diagonal represent positive sample pairs, which are correctly matched image and text feature pairs;
[0129] The non-diagonal elements are represented as negative sample pairs, which are incorrectly matched image and text feature pairs.
[0130] v i represents the visual features of the i-th chip image, v j represents the visual features of the jth chip image, s i and j Represents the image features v i and vj Corresponding text features, the method of constructing positive and negative sample pairs is as follows:
[0131]
[0132] This construction method can reduce the time and cost of manual labeling and improve the generalization ability of the model.
[0133] In the base, the contrast loss function is used to calculate the similarity between positive and negative sample pairs. By maximizing the similarity between similar features and minimizing the similarity between different features, the image visual features and text features of the positive sample are closer in the common embedding space, and the image visual features and text features of the negative sample are farther away, thereby effectively optimizing the learning ability of the silk screen information recognition model.
[0134] S206, using the optimized silk screen information recognition model to obtain predicted silk screen information, and training the silk screen information recognition model according to a cross entropy loss value between the predicted silk screen information and the real silk screen information;
[0135] Exemplarily, using the optimized silk screen information recognition model to obtain predicted silk screen information, and training the silk screen information recognition model according to the cross entropy loss value between the predicted silk screen information and the real silk screen information, including:
[0136] Use the optimized silk screen information recognition model to obtain the predicted silk screen information corresponding to the masked visual features, obtain the cross entropy loss value between the predicted silk screen information and the real silk screen information through the cross entropy loss function, and train the silk screen information recognition model based on the cross entropy loss value.
[0137] The correspondence between the preset chip image and the real silk-screen information is one-to-one, and different preset chip images correspond to different real silk-screen information.
[0138] Among them, the real silk-screen information is the correct silk-screen information.
[0139] S207, when the cross entropy loss value is less than the preset loss value, obtaining the silk screen information recognition result output by the trained silk screen information recognition model.
[0140] Exemplarily, when the cross entropy loss value is less than a preset loss value, obtaining the silk screen information recognition result output by the trained silk screen information recognition model includes:
[0141] When the cross entropy loss value is less than the preset loss value, the current chip image and the current text description information are input into the trained silk screen information recognition model to obtain the silk screen information recognition result output by the trained silk screen information recognition model.
[0142] Wherein, when the cross entropy loss value is less than the preset loss value, after obtaining the silk screen information recognition result output by the trained silk screen information recognition model, the chip silk screen information recognition method includes:
[0143] A display window is created, and the current text description information is displayed in the display window.
[0144] Compared with the existing chip silk screen information recognition method, the advantages of this application are as follows:
[0145] First, this application extracts the visual features and language features of the image through comparative learning and pre-trained silk screen information recognition model, which reduces the cost and time investment of manual labeling, better captures the details and features of the chip silk screen information, and improves the overall recognition performance.
[0146] Second, through feature fusion and masking mechanisms, this application can effectively establish associations between features of different modalities, train the silk screen information recognition model to perform text recognition in limited visual features, and improve the recognition accuracy of the silk screen information recognition model in complex and low-resolution scenes.
[0147] Finally, this application utilizes the prior knowledge of the language silk screen information recognition model to expand the predicted character sequence with similar words, provide multiple possible candidates for each predicted character sequence, and select the most appropriate result to guide the generation of correct recognition results based on the degree of match between the text language features of these candidates and the visual features of the image, thereby improving the recognition accuracy and generalization ability of the silk screen information recognition model in various complex scenarios.
[0148] See also Figure 3 , Figure 3 The flowchart for obtaining the current text description information provided in the embodiment of the present application is described in detail as follows:
[0149] S301, when the cross entropy loss value is less than the preset loss value, stop training the silk screen information recognition model, and save the trained silk screen information recognition model;
[0150] S302, obtaining the recognition accuracy of the trained silk screen information recognition model on the validation set;
[0151] S303, when the recognition accuracy is greater than the preset accuracy, obtain the current chip image and the current text description information corresponding to the current chip image, input the current chip image and the current text description information into the trained silk screen information recognition model, and obtain the silk screen information recognition result output by the trained silk screen information recognition model.
[0152] Among them, the recognition effect of the silk screen information of the model is evaluated on the validation set to ensure that it performs well in extracting image features, and according to the evaluation results, the model parameters and training strategies are dynamically adjusted.
[0153] For ease of explanation, the following is an example:
[0154] Search from high to low in a set of candidate values τ ∈ {0.05, 0.1, 0.2, 0.5, 0.75, 1.0}. As the training progresses, τ is dynamically adjusted according to the change of the loss. Through the performance metrics of the validation set, a suitable temperature coefficient τ is selected.
[0155] During the training process of the model, a step decay strategy is adopted. After every fixed number of training times, the learning rate is reduced by a fixed ratio. This method can smooth the training process and avoid premature convergence and oscillation phenomena.
[0156]
[0157] Among them, η 0 represents the initial learning rate, d is the decay factor, t represents the decay ratio, s represents the decay step, and η t represents the current learning rate.
[0158] Among them, 0 < d < 1. As t / s increases, the current learning rate will gradually decrease.
[0159] Among them, the decay factor determines the decay speed of the initial learning rate.
[0160] In the embodiment of the present application, when the recognition accuracy rate is greater than the preset accuracy rate, the current chip image and the current text description information corresponding to the current chip image are obtained, and the current chip image and the current text description information are input into the trained silk screen information recognition model, and the silk screen information recognition result output by the trained silk screen information recognition model is obtained. Since the preset accuracy rate is based on a custom function, the trained silk screen information recognition model can meet the recognition needs of users.
[0161] Corresponding to the chip silk screen information recognition method described in the above embodiment, please refer to Figure 4 , Figure 4 is a schematic block diagram of a chip silk screen information recognition device provided by an embodiment of the present application. Figure 4 The shown chip silk screen information recognition device 400 can be applied to an electronic device in the application scenario diagram shown in Figure 1 As shown. Taking the electronic device as an example, for Figure 4The chip silk screen information recognition device 400 shown is described in detail. The chip silk screen information recognition device 400 may include a first acquisition module 401, an extraction module 402, a selection module 403, a second acquisition module 404, an optimization module 405, a third acquisition module 406, and an identification module 407.
[0162] A first acquisition module 401 is used to acquire a preset chip image and preset text description information corresponding to the preset chip image;
[0163] An extraction module 402 is used to perform feature extraction on the preset chip image using the ResNet model and the Transformer model to obtain visual features corresponding to the preset chip image, and to perform feature extraction on the preset text description information using the Transformer model to obtain language features corresponding to the preset chip image;
[0164] A selection module 403 is used to fuse the visual feature and the language feature based on a cross attention mechanism to obtain a fused feature, obtain a sequence of the fused features, randomly select a position in the sequence, and use the visual feature before the position as the visual feature that needs to be masked;
[0165] A second acquisition module 404 is used to classify the masked visual features to generate a character sequence, map the character sequence into a text vector through a text encoder of a pre-trained model, and obtain similar words generated by the pre-trained model based on the text vector;
[0166] An optimization module 405 is used to optimize the silk screen information recognition model through a CLIP text encoder and the similar words;
[0167] A third acquisition module 406 is used to use the optimized silk screen information recognition model to obtain predicted silk screen information, and train the silk screen information recognition model according to the cross entropy loss value between the predicted silk screen information and the real silk screen information;
[0168] The recognition module 407 is used to obtain the silk screen information recognition result output by the trained silk screen information recognition model when the cross entropy loss value is less than a preset loss value.
[0169] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0170] The beneficial effects of the embodiments of the present application lie in two aspects. On the one hand, when the cross entropy loss value is less than the preset loss value, the silk screen information recognition result output by the trained silk screen information recognition model is obtained. Since there is no need to manually define the template, the recognition time of the chip silk screen information is reduced, which is beneficial to improving the recognition efficiency of the chip silk screen information. On the other hand, the visual features are masked by feature masking technology, and the silk screen information recognition model is trained to recognize silk screen information within limited visual features. Similarly, similar word expansion is used to provide multiple possible candidates for each predicted character sequence, thereby improving the prediction ability of the silk screen information recognition model in low-resolution scenarios, thereby improving the recognition accuracy of the chip silk screen information by the silk screen information recognition model.
[0171] See also Figure 5 , Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0172] like Figure 5 As shown, Figure 5 The electronic device 2 includes: at least one processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, and when the processor 20 executes the computer program 22, the steps in any of the above-mentioned method embodiments are implemented.
[0173] The electronic device 2 may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will appreciate that Figure 5 It is only an example of the electronic device 2 and does not constitute a limitation on the electronic device 2. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.
[0174] The processor 20 is used to run the computer program 22 stored in the memory 21, and implement the following steps when executing the computer program 22:
[0175] Obtaining a preset chip image and preset text description information corresponding to the preset chip image;
[0176] Using the ResNet model and the Transformer model, feature extraction is performed on the preset chip image to obtain visual features corresponding to the preset chip image; using the Transformer model, feature extraction is performed on the preset text description information to obtain language features corresponding to the preset chip image;
[0177] Based on the cross attention mechanism, the visual feature and the language feature are fused to obtain a fused feature, a sequence of the fused features is obtained, a position in the sequence is randomly selected, and the visual feature before the position is used as the visual feature that needs to be masked;
[0178] Classifying the masked visual features to generate a character sequence, mapping the character sequence into a text vector through a text encoder of a pre-trained model, and obtaining similar words generated by the pre-trained model based on the text vector;
[0179] The silk screen information recognition model is optimized through the CLIP text encoder and the similar words; the optimized silk screen information recognition model is used to obtain predicted silk screen information, and the silk screen information recognition model is trained according to the cross entropy loss value between the predicted silk screen information and the real silk screen information; when the cross entropy loss value is less than the preset loss value, the silk screen information recognition result output by the trained silk screen information recognition model is obtained.
[0180] In some embodiments, the processor 20 is configured to implement:
[0181] Based on the cross attention mechanism, the visual feature and the language feature are fused to obtain the fused feature, a sequence of the fused features is obtained, and a position in the sequence is randomly selected;
[0182] An attention score map is obtained, and according to the attention score map, the visual features before the position are used as the visual features that need to be masked.
[0183] In some embodiments, the processor 20 is configured to implement:
[0184] Replacing the feature value of the visual feature to be masked with the selected mark to obtain the masked visual feature;
[0185] Sending the masked visual features to a linear layer, and classifying the masked visual features by the classification function to generate a character sequence corresponding to the masked visual features;
[0186] The character sequence is mapped into a text vector through a text encoder of a pre-trained model, and similar words generated by the pre-trained model based on the text vector are obtained.
[0187] In some embodiments, the processor 20 is configured to implement:
[0188] Encode the similar words by using the CLIP text encoder to obtain text features corresponding to the similar words, use the cosine similarity formula to calculate the similarity between the visual features and the text features, and construct a similarity matrix by using multiple similarities;
[0189] In the similarity matrix, the visual features and the text features whose similarity is higher than a preset value are regarded as positive sample pairs, and the visual features and the text features whose similarity is not higher than the preset value are regarded as negative sample pairs;
[0190] The silk screen information recognition model is optimized by using a contrast loss function with the goal of minimizing the distance between the positive sample pairs and maximizing the distance between the negative sample pairs.
[0191] In some embodiments, the processor 20 is configured to implement:
[0192] When the cross entropy loss value is less than the preset loss value, stop training the silk screen information recognition model, and save the trained silk screen information recognition model;
[0193] Obtaining the recognition accuracy of the trained silk screen information recognition model on the validation set;
[0194] When the recognition accuracy is greater than the preset accuracy, the current chip image and the current text description information corresponding to the current chip image are obtained, the current chip image and the current text description information are input into the trained silk screen information recognition model, and the silk screen information recognition result output by the trained silk screen information recognition model is obtained.
[0195] In some embodiments, the processor 20 is configured to implement:
[0196] A display window is created, and the current text description information is displayed in the display window.
[0197] In some embodiments, the processor 20 is configured to implement:
[0198] The pre-trained model includes one or a combination of a Word2Vec model and a BRET model.
[0199] The processor 20 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0200] In some embodiments, the memory 21 may be an internal storage unit of the electronic device 2, such as a hard disk or memory of the electronic device 2. In other embodiments, the memory 21 may also be an external storage device of the electronic device 2, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 2. Further, the memory 21 may also include both an internal storage unit of the electronic device 2 and an external storage device. The memory 21 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, such as the program code of the computer program, etc. The memory 21 may also be used to temporarily store data that has been output or is to be output.
[0201] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0202] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0203] The computer-readable storage medium stores program codes, which can be called by a processor to execute the chip silk screen information recognition method described in the above method embodiment.
[0204] The computer-readable storage medium has a storage space for program codes.
[0205] The program code includes the code of any step in the chip silk screen information recognition method described in the above method embodiment.
[0206] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0207] Among them, the computer-readable storage medium can also be an external storage device of the chip silk-screen information identification device or the electronic device, for example, a plug-in hard disk equipped on the chip silk-screen information identification device or the electronic device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), a non-volatile computer-readable medium (non-transitory computer-readable storage medium), etc.
[0208] Since the computer program stored in the computer-readable storage medium can execute any one of the chip silk screen information recognition methods provided in the embodiments of the present application, the computer-readable storage medium can achieve the beneficial effects that can be achieved by any one of the chip silk screen information recognition methods provided in the embodiments of the present application. Please see the previous embodiments for details and will not be repeated here.
[0209] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the above-mentioned chip silk screen information recognition method.
[0210] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
[0211] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0212] Based on such understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to an electronic device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.
[0213] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0214] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A method for identifying chip silk screen information, It is characterized in that Applied to electronic equipment, the electronic equipment stores a silk screen information recognition model, the silk screen information recognition model includes a ResNet model and a Transformer model, and the chip silk screen information recognition method includes: Obtaining a preset chip image and preset text description information corresponding to the preset chip image; Using the ResNet model and the Transformer model, feature extraction is performed on the preset chip image to obtain visual features corresponding to the preset chip image; using the Transformer model, feature extraction is performed on the preset text description information to obtain language features corresponding to the preset chip image; Based on the cross attention mechanism, the visual feature and the language feature are fused to obtain a fused feature, a sequence of the fused features is obtained, a position in the sequence is randomly selected, and the visual feature before the position is used as the visual feature that needs to be masked; Classifying the masked visual features to generate a character sequence, mapping the character sequence into a text vector through a text encoder of a pre-trained model, and obtaining similar words generated by the pre-trained model based on the text vector; Optimizing the silk screen information recognition model through the CLIP text encoder and the similar words; Using the optimized silk screen information recognition model, the predicted silk screen information is obtained, and according to the cross entropy loss value between the predicted silk screen information and the real silk screen information, the silk screen information recognition model is trained; When the cross entropy loss value is less than the preset loss value, the silk screen information recognition result output by the trained silk screen information recognition model is obtained.
2. The chip silk screen information recognition method according to claim 1, It is characterized in that The method of fusing the visual feature and the language feature based on the cross attention mechanism to obtain a fused feature, obtaining a sequence of the fused features, randomly selecting a position in the sequence, and using the visual feature before the position as the visual feature that needs to be masked includes: Based on the cross attention mechanism, the visual feature and the language feature are fused to obtain the fused feature, a sequence of the fused features is obtained, and a position in the sequence is randomly selected; An attention score map is obtained, and according to the attention score map, the visual features before the position are used as the visual features that need to be masked.
3. The chip silk screen information recognition method according to claim 1, It is characterized in that The step of classifying the masked visual features to generate a character sequence, mapping the character sequence into a text vector through a text encoder of a pre-trained model, and obtaining similar words generated by the pre-trained model based on the text vector includes: Replacing the feature value of the visual feature to be masked with the selected mark to obtain the masked visual feature; Sending the masked visual features to a linear layer, and classifying the masked visual features by the classification function to generate a character sequence corresponding to the masked visual features; The character sequence is mapped into a text vector through a text encoder of a pre-trained model, and similar words generated by the pre-trained model based on the text vector are obtained.
4. The chip silk screen information recognition method according to claim 1, It is characterized in that The step of optimizing the silk screen information recognition model by using the CLIP text encoder and the similar words includes: Encode the similar words by using the CLIP text encoder to obtain text features corresponding to the similar words, use the cosine similarity formula to calculate the similarity between the visual features and the text features, and construct a similarity matrix by using multiple similarities; In the similarity matrix, the visual features and the text features whose similarity is higher than a preset value are regarded as positive sample pairs, and the visual features and the text features whose similarity is not higher than the preset value are regarded as negative sample pairs; The silk screen information recognition model is optimized by using a contrast loss function with the goal of minimizing the distance between the positive sample pairs and maximizing the distance between the negative sample pairs.
5. The chip silk screen information recognition method according to claim 1, It is characterized in that When the cross entropy loss value is less than a preset loss value, obtaining the silk screen information recognition result output by the trained silk screen information recognition model comprises: When the cross entropy loss value is less than the preset loss value, stop training the silk screen information recognition model, and save the trained silk screen information recognition model; Obtaining the recognition accuracy of the trained silk screen information recognition model on the validation set; When the recognition accuracy is greater than the preset accuracy, the current chip image and the current text description information corresponding to the current chip image are obtained, the current chip image and the current text description information are input into the trained silk screen information recognition model, and the silk screen information recognition result output by the trained silk screen information recognition model is obtained.
6. The chip silk screen information recognition method according to claim 1, It is characterized in that When the cross entropy loss value is less than the preset loss value, after obtaining the silk screen information recognition result output by the trained silk screen information recognition model, the chip silk screen information recognition method includes: A display window is created, and the current text description information is displayed in the display window.
7. The chip silk screen information recognition method according to any one of claims 1 to 6, It is characterized in that The pre-trained model includes one or a combination of a Word2Vec model and a BRET model.
8. A chip silk screen information recognition device, It is characterized in that Applied to electronic equipment, the electronic equipment stores a silk screen information recognition model, the silk screen information recognition model includes a ResNet model and a Transformer model, including: A first acquisition module, used to acquire a preset chip image and preset text description information corresponding to the preset chip image; An extraction module, used to perform feature extraction on the preset chip image using the ResNet model and the Transformer model to obtain visual features corresponding to the preset chip image, and to perform feature extraction on the preset text description information using the Transformer model to obtain language features corresponding to the preset chip image; A selection module is used to fuse the visual feature and the language feature based on a cross attention mechanism to obtain a fused feature, obtain a sequence of the fused features, randomly select a position in the sequence, and use the visual feature before the position as the visual feature that needs to be masked; A second acquisition module is used to classify the masked visual features to generate a character sequence, map the character sequence into a text vector through a text encoder of a pre-trained model, and obtain similar words generated by the pre-trained model based on the text vector; An optimization module, used for optimizing the silk screen information recognition model through a CLIP text encoder and the similar words; A third acquisition module is used to use the optimized silk screen information recognition model to obtain predicted silk screen information, and train the silk screen information recognition model according to the cross entropy loss value between the predicted silk screen information and the real silk screen information; The recognition module is used to obtain the silk screen information recognition result output by the trained silk screen information recognition model when the cross entropy loss value is less than a preset loss value.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the chip silk screen information recognition method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the chip silk screen information recognition method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Natural scene text recognition method and device based on bidirectional feature language model
CN112733768A
Natural scene text detection method and system based on attention mechanism feature fusion and enhancement
CN114255456A
Video detection method and device based on multi-modal information fusion and computer equipment
CN116524404A
Chinese named entity recognition method based on InterFormer-BERT model and insertion of special mark
CN116776876A
Multi-modal action recognition model training method, action recognition method and related equipment
CN117636473A
Cited By
Chip serial number identification method for Bar wafers
CN120976938A
Chip serial number identification method for Bar strip wafer
CN120976938B
