Electronic component zero-sample recognition model and method based on visual information and semantic attributes

Abstract

The application discloses an electronic component zero-sample recognition model and method based on visual information and semantic attributes, and the recognition model comprises a visual classification network model, a Skip-gram model and a CBOW model. According to the visual features of the electronic components, a semantic attribute table of visible class and invisible class electronic components is designed, so that the semantic attributes between the visible class and the invisible class are shared. By combining the visual information and the semantic attributes of the labels, the semantic attributes of the labels learned from the text data are used as the supervision signals, and the visual classification network is trained in combination with the seen class images, and then zero-sample learning is realized. During the training, the visible class is converted into the semantic attribute class through the Skip-gram model, so that the training process of the visual classification network becomes the training between the shared semantic attribute class and the input image. In the prediction process, the visual classification network predicts the shared semantic attribute class, the CBOW model converts the semantic attribute class into the class label, the prediction of the visible class and the invisible class is realized, and the zero-sample recognition of the electronic components is completed.

Description

Technical Field

[0001] This invention belongs to the field of intelligent manufacturing, and in particular relates to a zero-sample identification method for electronic components based on visual information and semantic attributes. Background Technology

[0002] Zero-shot image classification aims to enable deep learning models to identify unseen categories that have not been trained on. Without requiring labeled target category data, it learns from visible classes and shared semantic knowledge, achieving human-like reasoning abilities and transferring knowledge from visible to invisible classes to identify them.

[0003] In 2013, Andrea Frome et al. proposed a deep visual semantic embedding model by combining feature vectors obtained from a visual model with semantic vectors obtained from a Word2Vec model to achieve zero-shot learning. This model utilizes semantic information collected from labeled image data and unlabeled text to train and recognize visual objects. The model primarily targets 1000 classes and semantically similar classes in the ImageNet public dataset.

[0004] In 2017, Elyor Kodirov et al. proposed a zero-shot learning method based on a learned semantic autoencoder (SAE). Employing an encoder-decoder approach, the encoder functions similarly to common zero-shot learning methods, aiming to project visual feature vectors into the semantic space. However, the decoder adds an extra constraint to reconstruct the original visual features. This additional reconstruction constraint allows the projection function learned from the visible class to better generalize to the invisible class. The model achieved good recognition results primarily on the six benchmark datasets described in their paper.

[0005] Currently, zero-shot image classification algorithms are rarely used in specific scenarios, and research and application of zero-shot image classification algorithms for electronic components are still limited. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a zero-shot identification model and method for electronic components based on visual information and semantic attributes. This aims to enable the zero-shot algorithm to identify invisible classes of electronic components in specific scenarios. The invention provides a zero-shot identification method for electronic components based on visual information and semantic attributes.

[0007] The technical solution adopted in this invention is as follows:

[0008] Zero-shot identification models for electronic components based on visual information and semantic attributes include:

[0009] A visual classification network model is provided, comprising a visual feature extraction module, a semantic attribute label classification module, a classification scoring module, and a semantic attribute label supervision module. The visual feature extraction module extracts feature maps from input electronic component images and inputs them into the semantic attribute label classification module. The semantic attribute label classification module predicts classification results based on the input feature maps. The semantic attribute label supervision module supervises the visual classification network model based on the semantic attribute class labels corresponding to the category labels output by the Skip-gram model. The classification scoring module calculates classification scores based on the classification results predicted by the semantic attribute label classification module and outputs the three semantic attribute class labels with the highest scores.

[0010] The Skip-gram model takes the category labels of electronic components as input and outputs the semantic attribute class labels corresponding to the category labels. The Skip-gram model inputs the output to the semantic attribute label supervision module, where the visual classification network model is supervised by a multi-class cross-entropy loss function.

[0011] The CBOW model takes semantic attribute class labels output by the classification scoring module as input and outputs the category label corresponding to the predicted electronic component image as output.

[0012] Furthermore, the visual feature extraction module is composed of the conv1 layer, conv2_x layer, conv3_x layer, conv4_x layer and conv5_x layer of the ResNet101 classification network connected in sequence.

[0013] Furthermore, the semantic attribute label classification module consists of an adaptive average pooling layer and a fully connected layer; the feature map extracted by the visual feature extraction module is then processed by adaptive average pooling and input into the fully connected layer to predict the classification result.

[0014] Furthermore, the classification scoring module calculates the classification score based on the predicted classification results using the Softmax activation function.

[0015] Furthermore, the Skip-gram model includes an input layer, a hidden layer, and an output layer connected in sequence. The input layer is the word vector X of the center word encoded in one-hot encoding, and the output layer outputs the word vectors of the context words encoded in one-hot encoding.

[0016] Furthermore, the CBOW model includes an input layer, a hidden layer, and an output layer connected in sequence. The input layer is the word vector of the context words encoded in one-hot encoding, and the output layer is the word vector of the middle words encoded in one-hot encoding.

[0017] Furthermore, the method for training the zero-shot recognition model of electronic components based on visual information and semantic attributes is as follows:

[0018] S2.1, Electronic components are divided into visible and invisible categories. A semantic attribute table of electronic components is constructed based on the visual characteristics of the visible and invisible electronic components. The semantic attribute table includes category labels and semantic attribute class labels corresponding to the electronic components. The category labels are the names of the electronic components, and the semantic attribute class labels are the visual characteristics corresponding to the electronic components.

[0019] The category label words and semantic attribute class label words appearing in the semantic attribute table are counted. The category label is placed in the middle as the middle word and the semantic attribute class label is placed on both sides of the category label as the context word to form a vocabulary list. The vocabulary list is then expanded to obtain the training text.

[0020] The Skip-gram model is trained using category label words and semantic attribute label words corresponding to visible electronic components, and the CBOW model is trained using category label words and semantic attribute label words corresponding to both visible and invisible electronic components. The model parameters of the Skip-gram model and the CBOW model are obtained.

[0021] S2.2, Based on the trained Skip-gram model, input the category labels of visible electronic components into the Skip-gram model, and the Skip-gram model outputs semantic attribute class labels;

[0022] Images of visible electronic components are input into a visual classification network to obtain the classification results from the semantic attribute label classification module.

[0023] The cross-entropy loss function of multi-class classification is used to calculate the cost loss between the classification result of the semantic attribute label classification module and the semantic attribute class output by the Skip-gram model. The visual classification network model is iteratively updated according to the cost loss to complete the training of the zero-shot recognition model of electronic components based on visual information and semantic attributes.

[0024] Furthermore, the cross-entropy loss function for the multi-class classification is expressed as:

[0025]

[0026] Among them, y i It is sample x i The true probability of belonging to the i-th category, f θ (x i ) is sample x iThe predicted score for the i-th category, where n is the total number of categories, and S() is the Softmax activation function.

[0027] Furthermore, the visual characteristics of the electronic components include the color, shape, and pins of the electronic components.

[0028] A zero-shot identification method for electronic components based on visual information and semantic attributes includes the following steps:

[0029] S1. Input the image of the electronic component to be identified into the visual feature extraction module, and the visual feature extraction module extracts features from the input image;

[0030] S2, the semantic attribute label classification module performs classification prediction on the extracted features, and the classification scoring module calculates the classification score on the classification prediction results;

[0031] S3. Filter out the semantic attribute class labels with the highest prediction scores according to color, shape and pin respectively;

[0032] S4. Initialize the CBOW model using the trained model parameters;

[0033] S5. Input the three semantic attribute classes with the highest prediction scores into the CBOW model and output the final recognition result.

[0034] The beneficial effects of this invention are:

[0035] (1) Based on the functions and principles of visual classification network models, Skip-gram models, and CBOW models, this invention combines the advantages of these three models to design a zero-shot identification method for electronic components based on visual information and semantic attributes. This method is beneficial for solving the zero-shot identification problem of electronic components in specific scenarios.

[0036] (2) Based on the visual features of various categories in the electronic component dataset, this invention links the semantic attributes of visible and invisible categories to form a semantic attribute table for each category of electronic components. By manually designing semantic attributes to link visible and invisible categories in specific scenarios, it is more conducive to zero-shot recognition of invisible categories in specific scenarios, and helps to improve the recognition accuracy of zero-shot algorithms for invisible categories of electronic components.

[0037] (3) Based on the visual features of each category in the electronic component dataset, a semantic attribute table for each category of electronic components is constructed, enabling the semantic attributes of visible and invisible classes to be shared. During training, the visible classes are transformed into semantic attribute classes using the Skip-gram model, making the training process of the visual classification network a training between the shared semantic attribute classes and the input image. During prediction, the visual classification network predicts the shared semantic attribute classes, and the semantic attribute classes are transformed into visible and invisible classes using the CBOW model, achieving the recognition of visible and invisible classes. This enables zero-shot recognition of electronic components. This invention applies the zero-shot method to a specific scenario, which is beneficial for exploring the application of zero-shot classification algorithms in specific scenarios. Attached Figure Description

[0038] Figure 1 This is a network structure diagram of the zero-sample identification algorithm for electronic components of this invention;

[0039] Figure 2 This is a network structure diagram of the visual feature extraction module described in this invention;

[0040] Figure 3 This refers to the Skip-gram model described in this invention;

[0041] Figure 4 The CBOW model described in this invention;

[0042] Figure 5 This is a semantic attribute table for various categories of electronic components described in this invention;

[0043] Figure 6 This is the vocabulary of the Skip-gram model and CBOW model described in this invention;

[0044] Figure 7 This is the semantic attribute category table of the visual classification network described in this invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0046] This application designs a zero-shot recognition model for electronic components based on visual information and semantic attributes, such as... Figure 1 The diagram illustrates a zero-shot recognition network model for electronic components, constructed based on visual information and semantic attributes. The overall network model consists of three sub-models: a visual classification network model, a Skip-gram model, and a CBOW model (e.g., [model name missing]). Figure 1(As shown). The visual classification network model learns the semantic attribute labels of electronic components and classifies and scores the input images according to multiple shared semantic attribute labels, predicting the higher-scoring semantic attribute label. The Skip-gram model converts category labels into semantic attribute labels (taking an I-shaped inductor as an example, converting its category label "I-shaped inductor" into its corresponding semantic attribute label "black, I-shaped, and single-sided pin"). The CBOW model converts the semantic attribute labels output by the visual classification network model into category labels (e.g., converting the semantic attribute labels "black, I-shaped, and single-sided pin" into the category label "I-shaped inductor").

[0047] The specific structures of each sub-model are as follows:

[0048] 1. Visual classification network model

[0049] The visual classification network model consists of a visual feature extraction module, a semantic attribute label classification module, a classification scoring module, and a semantic attribute label supervision module.

[0050] The visual feature extraction module consists of the conv1, conv2_x, conv3_x, conv4_x, and conv5_x layers of the ResNet101 classification network (e.g., ...). Figure 2 As shown in the figure, the input image is processed through these five layers in sequence, and the output image features are then extracted to complete the feature extraction of the input image and obtain the feature map.

[0051] The semantic attribute label classification module consists of an adaptive average pooling layer and a fully connected layer. The feature maps extracted by the visual feature extraction module are processed by adaptive average pooling and then input into the fully connected layer to predict the classification results. The classification results of each category predicted by the fully connected layer are output in the form of an array. For example, if there are 5 categories, the output results are [0.55, -0.56, 0.71, -0.28, -0.35].

[0052] The classification scoring module calculates the classification score based on the predicted classification results using the Softmax activation function; the mathematical expression for the Softmax activation function is as follows:

[0053]

[0054] Among them, z i z is the predicted output value for the i-th category; cThis is the predicted output value for the c-th category, where c = 1, 2, ..., C, and C is the number of categories. In this application, since the classification results predicted by the semantic attribute label classification module are output in the form of an array, the final predicted probability of each category is calculated using the Softmax activation function, and the three semantic attribute class labels with the highest predicted probabilities are taken as the final output.

[0055] The semantic attribute label supervision module supervises the visual classification network model using a multi-class cross-entropy loss function.

[0056] 2. Skip-gram model

[0057] The Skip-gram model predicts the words preceding and following it by using an intermediate word. For example, in the sentence "I drove my car to the company", if the input is the word "car", then the output would be {"I", "drove", "my", "to", "the", "company"}.

[0058] Specifically, such as Figure 3 As shown, the Skip-gram model consists of an input layer, a hidden layer, and an output layer. The input layer is the word vector X of the one-hot encoded center word, and V is the dimension of the word vector space. In this invention, V is 25, which is the vocabulary of the Skip-gram model and the CBOW model (e.g., ...). Figure 6 The number of words in the given text (as shown). The hidden layer is an N-dimensional vector h, where N is a parameter that can be adjusted according to the model's performance; in this invention, it is set to 300. The output layer outputs the one-hot encoded word vectors {Y1, Y2, ..., Y} of the context words. c}, where C is the number of context words, which is set to 3 in this invention, representing the number of semantic attribute class labels corresponding to each type of electronic component. In the Skip-gram model, the word vectors input from the input layer are processed by the weight matrix W. v×N Connected to the hidden layer, the hidden layer is connected through the weight matrix W′ N×V Connect to the output layer.

[0059] 3. CBOW Model

[0060] The CBOW model predicts the intermediate word by using the words in the context. Specifically, as follows... Figure 4 As shown, the CBOW model consists of an input layer, hidden layers, and an output layer. The input layer is the one-hot encoded word vectors {X1, X2, ..., X} of the context words. cThe hidden layer is an N-dimensional vector h. The output layer outputs the word vector Y, which is the one-hot encoded intermediate word. Similar to the Skip-gram model, the parameters V, C, and N are set to 25, 3, and 300. In the CBOW model, the word vectors input from the input layer are processed by the weight matrix W. V×N Connected to the hidden layer, the hidden layer is connected through the weight matrix W′ N×V Connect to the output layer.

[0061] The established zero-shot identification model for electronic components based on visual information and semantic attributes is trained. The training process includes the following steps:

[0062] S1. First, train the Skip-gram model and the CBOW model separately to ensure that the Skip-gram model can convert category labels into semantic attribute labels, and the CBOW model can convert semantic attribute labels into category labels.

[0063] Labels. Specific training includes the following steps:

[0064] ① First, some electronic components are designated as the visible class, and different electronic components that share common visual characteristics with the visible electronic components are designated as the invisible class. Based on the visual characteristics of the visible and invisible electronic components, a semantic attribute table is constructed for each category of electronic components (e.g., ...). Figure 5 (As shown); where the category label is the name of the electronic component, and the semantic attribute label is the visual feature corresponding to the electronic component, such as color, shape, and pins. In this application, the visual feature relationship between visible and invisible categories is also considered, so that the semantic attribute categories between visible and invisible categories are linked, that is, common words between different electronic components are selected as much as possible. Thus, a semantic attribute table for various categories of electronic components is formed. In this application, the semantic attribute table contains 8 visible electronic components (including I-shaped inductors, color-ring resistors, magnetic rod inductors, ring varistors, white pins, rectifier diodes, red pins, and relays) and 4 invisible electronic components (including horizontal I-shaped inductors, magnetic ring inductors, cement resistors, and black pins).

[0065] ② Statistically analyze the category label words and semantic attribute class label words appearing in the constructed semantic attribute table. Arrange the category labels in the middle as middle words and the semantic attribute class labels on both sides as context words to form the vocabulary of the Skip-gram model and the CBOW model (e.g., Figure 6 (as shown);

[0066] ③ Expand the vocabulary according to the above arrangement of category labels and semantic attribute labels, and construct training text. Specifically, expand each category 5000 times. For example, to expand the training text for I-shaped inductors, copy the text "black, I-shaped, I-shaped inductor, single-sided pin" 5000 times. After expanding the data for all categories, merge them to form the training text;

[0067] ④ Finally, the Skip-gram model and the CBOW model were trained using training texts to obtain the model parameters of the Skip-gram model and the CBOW model, respectively. The training texts used to train the Skip-gram model consisted only of category label words and semantic attribute label words for visible electronic components. The training texts used to train the CBOW model contained category label words and semantic attribute label words for both visible and invisible electronic components.

[0068] S2, combined with a pre-trained Skip-gram model, trains a visual classification network. For example... Figure 1 The training section includes the following steps:

[0069] ① Initialize the Skip-gram model using the model parameters already trained in S2.1;

[0070] ② Input the category labels of visible electronic components into the Skip-gram model, and the Skip-gram model outputs the semantic attribute class labels corresponding to the category labels;

[0071] Combined with the semantic attribute table in this embodiment (e.g.) Figure 1 A total of 13 semantic attribute class tags were formed (such as...). Figure 7 The semantic attribute category table of the visual classification network is shown below.

[0072] ③ Input the image of the visible electronic component into the visual classification network. The visual feature extraction module extracts features from the input image, and the semantic attribute label classification module performs classification prediction of semantic attribute labels to obtain the predicted classification result.

[0073] ④ In the semantic attribute label supervision module, the cost loss between the classification result of the semantic attribute label classification module and the semantic attribute class label output by the Skip-gram model is calculated using the multi-class cross-entropy loss function. The specific formula is as follows:

[0074]

[0075] The visual classification network model is updated by backpropagation based on cost loss, and the iteration continues until the set iteration termination condition is reached, thus completing the training of the visual classification network model. The iteration termination condition can be the set number of iterations.

[0076] Among them, y i It is sample x i The true probability of belonging to the i-th category, f θ (x i ) is sample x i The predicted score for the i-th category, where n is the total number of categories, and S() is the Softmax activation function. The mathematical expression for S() is as follows:

[0077]

[0078] Among them, z i z is the predicted output value for the i-th category; c It is the predicted output value of the c-th category, where c = 1, 2, ..., C, and C is the number of categories.

[0079] Based on the zero-shot electronic component recognition model constructed in this application, which is based on visual information and semantic attributes, this application can use this recognition model to achieve zero-shot electronic component recognition, as shown in the following process (as follows). Figure 1 The prediction section (as shown) includes the following steps:

[0080] (1) Input the image of the electronic component to be identified into the visual feature extraction module, and the visual feature extraction module extracts features from the input image; the image of the electronic component to be identified can be a visible electronic component or an invisible electronic component.

[0081] (2) The semantic attribute label classification module performs classification prediction on the extracted features, and the classification scoring module calculates the classification score based on the classification prediction results using the Softmax activation function. This yields the predicted score for each semantic attribute class label.

[0082] (3) Filter out the semantic attribute class labels with the highest prediction scores by color, shape and pin respectively;

[0083] (4) Initialize the CBOW model using the trained model parameters;

[0084] (5) Input the three semantic attribute classes with the highest prediction scores into the CBOW model, and output the final recognition result (including visible and invisible classes). For example... Figure 1 As shown in the prediction section.

[0085] The embodiments described above are preferred embodiments of the present invention, but the present invention is not limited to the above.

[0086] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for constructing a zero-shot recognition model for electronic components based on visual information and semantic attributes, characterized in that, include: A visual classification network model is provided, comprising a visual feature extraction module, a semantic attribute label classification module, a classification scoring module, and a semantic attribute label supervision module. The visual feature extraction module extracts feature maps from input electronic component images and inputs them into the semantic attribute label classification module. The semantic attribute label classification module predicts classification results based on the input feature maps. The semantic attribute label supervision module supervises the visual classification network model based on the semantic attribute class labels corresponding to the category labels output by the Skip-gram model. The classification scoring module calculates classification scores based on the classification results predicted by the semantic attribute label classification module and outputs the three semantic attribute class labels with the highest scores. The Skip-gram model takes the category labels of electronic components as input and outputs the semantic attribute class labels corresponding to the category labels. The Skip-gram model inputs the output to the semantic attribute label supervision module, where the visual classification network model is supervised by a multi-class cross-entropy loss function. The CBOW model takes semantic attribute class labels output by the classification scoring module as input and outputs the category label corresponding to the predicted electronic component image as output.

2. The method for constructing a zero-shot identification model for electronic components based on visual information and semantic attributes according to claim 1, characterized in that, The visual feature extraction module is composed of the conv1 layer, conv2_x layer, conv3_x layer, conv4_x layer and conv5_x layer of the ResNet101 classification network connected in sequence.

3. The method for constructing a zero-shot identification model for electronic components based on visual information and semantic attributes according to claim 1, characterized in that, The semantic attribute label classification module consists of an adaptive average pooling layer and a fully connected layer; the feature map extracted by the visual feature extraction module is processed by adaptive average pooling and then input into the fully connected layer to predict the classification result.

4. The method for constructing a zero-shot identification model for electronic components based on visual information and semantic attributes according to claim 1, characterized in that, The classification scoring module calculates the classification score based on the predicted classification results using the Softmax activation function.

5. The method for constructing a zero-shot identification model for electronic components based on visual information and semantic attributes according to any one of claims 1-4, characterized in that, The Skip-gram model comprises an input layer, a hidden layer, and an output layer connected in sequence. The input layer is the word vector X of the center word encoded in one-hot encoding, and the output layer outputs the word vectors of the context words encoded in one-hot encoding.

6. The method for constructing a zero-shot identification model for electronic components based on visual information and semantic attributes according to claim 5, characterized in that, The CBOW model comprises an input layer, a hidden layer, and an output layer connected in sequence. The input layer consists of word vectors of one-hot encoded context words, and the output layer consists of word vectors of one-hot encoded intermediate words.

7. The method for constructing a zero-shot identification model for electronic components based on visual information and semantic attributes according to claim 6, characterized in that, The method for training the zero-shot recognition model of electronic components based on visual information and semantic attributes is as follows: S2.1, Electronic components are divided into visible and invisible categories. A semantic attribute table of electronic components is constructed based on the visual characteristics of the visible and invisible electronic components. The semantic attribute table includes category labels and semantic attribute class labels corresponding to the electronic components. The category labels are the names of the electronic components, and the semantic attribute class labels are the visual characteristics corresponding to the electronic components. The category label words and semantic attribute class label words appearing in the semantic attribute table are counted. The category label is placed in the middle as the middle word and the semantic attribute class label is placed on both sides of the category label as the context word to form a vocabulary list. The vocabulary list is then expanded to obtain the training text. The Skip-gram model is trained using category label words and semantic attribute label words corresponding to visible electronic components, and the CBOW model is trained using category label words and semantic attribute label words corresponding to both visible and invisible electronic components. The model parameters of the Skip-gram model and the CBOW model are obtained. S2.2, Based on the trained Skip-gram model, input the category labels of visible electronic components into the Skip-gram model, and the Skip-gram model outputs semantic attribute class labels; Images of visible electronic components are input into a visual classification network to obtain the classification results from the semantic attribute label classification module. The cross-entropy loss function of multi-class classification is used to calculate the cost loss between the classification result of the semantic attribute label classification module and the semantic attribute class output by the Skip-gram model. The visual classification network model is iteratively updated according to the cost loss to complete the training of the zero-shot recognition model of electronic components based on visual information and semantic attributes.

8. The method for constructing a zero-shot identification model for electronic components based on visual information and semantic attributes according to claim 7, characterized in that, The cross-entropy loss function for the multi-class classification is expressed as: Among them, y i It is sample x i The true probability of belonging to the i-th category, f θ (x i ) is sample x i The predicted score for the i-th category, where n is the total number of categories, and S() is the Softmax activation function.

9. The method for constructing a zero-shot identification model for electronic components based on visual information and semantic attributes according to claim 7, characterized in that, The visual characteristics of the electronic components include their color, shape, and pins.

10. A zero-sample identification method for electronic components, characterized in that, Based on the zero-shot electronic component recognition model constructed using the method of claim 1, which is based on visual information and semantic attributes, the recognition steps are as follows: S1. Input the image of the electronic component to be identified into the visual feature extraction module, and the visual feature extraction module extracts features from the input image; S2, the semantic attribute label classification module performs classification prediction on the extracted features, and the classification scoring module calculates the classification score on the classification prediction results; S3. Filter out the semantic attribute class labels with the highest prediction scores according to color, shape and pin respectively; S4. Initialize the CBOW model using the trained model parameters; S5. Input the three semantic attribute classes with the highest prediction scores into the CBOW model and output the final recognition result.

Images