Method for recognizing visual concepts oriented to multi-modal knowledge graph construction

Abstract

The application provides a method for identifying visual concepts for multi-modal knowledge graph construction, for the first time explicitly proposes the problem of identifying visual concepts in the process of multi-modal knowledge graph construction, models the problem as a binary classification problem, screens and designs feature data required to solve the classification problem, thereby constructing a multi-modal classification model containing a BERT model and an image classification model, which is used to encode and fuse text features and image features of a to-be-tested concept, so that the purpose of distinguishing visual concepts and non-visual concepts is achieved. Meanwhile, the multi-modal classification model adopts a two-stage PULearning training method, realizes the initialization of training data in a fully automated manner, effectively expands the training data set in an iterative manner by using the self-training idea, and completely avoids manual means in the expansion process and improves the accuracy of the classification network.

Description

Technical Field

[0001] This invention belongs to the field of multimodal knowledge graphs, specifically relating to a method for recognizing visual concepts in the construction of multimodal knowledge graphs. Background Technology

[0002] As data modalities become increasingly diverse, the construction of knowledge graphs is also gradually incorporating additional modalities. A common approach is to add modal information beyond text, such as images, to entities and concepts in the graph. However, not all entities in a knowledge graph are suitable for representation using images. For example, words like "mind" do not have a direct visual image, and these entities should be filtered out during the construction of a multimodal knowledge graph.

[0003] Currently, there are no specific research methods for visual concept recognition. Only a few multimodal knowledge graph construction methods implicitly consider this problem. Two naive methods for distinguishing visual concepts can be summarized. The first method directly classifies all hypotheses of "abstract" in the WordNet dataset as non-visual concepts, while treating all other words in WordNet as visual concepts. The second method adopts a learning approach: it assumes that images of visual concepts are more easily distinguished from "background" images. A binary classifier based on support vector machines is trained for each entity, comparing the concept image and the "background" image. If the accuracy of this classifier reaches a threshold, the entity is considered a visual concept; otherwise, it is a non-visual concept.

[0004] Currently, both methods are relatively rudimentary and lack quantitative evaluation of their accuracy. Testing on manually labeled test sets yielded accuracies of only 0.61 and 0.54, respectively. Analysis reveals that Method 1 suffers from the problem of confusing non-visual concepts with non-abstract entities. Equating visual concepts with non-abstract entities is problematic; for example, "texture" is not an abstract entity, but it also cannot be considered a visual concept because no suitable image can be found to match it. Method 2 involves two issues: first, the assumptions about visual concepts are unverified; second, the concept images used for classification are crawled from the web without disambiguation, easily introducing confusion due to entities with the same name. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a method for targeted visual concept identification during the construction of multimodal knowledge graphs. The technical solution adopted in this invention is as follows:

[0006] This invention provides a method for recognizing visual concepts in the construction of multimodal knowledge graphs, characterized by the following steps: Step S1, constructing a binary classification model that distinguishes between visual and non-visual concepts, and iteratively training the model using self-training to obtain a multimodal visual concept classification model; Step S2, inputting the concept to be tested into the multimodal visual concept classification model to obtain the probability that the concept to be tested is a visual concept, wherein the multimodal visual concept classification model has a BERT sub-model, an image classification sub-model, and a binary classification network.

[0007] The method for recognizing visual concepts based on multimodal knowledge graph construction provided by this invention may also have the following technical features, wherein step S2 includes the following sub-steps: step S2-1, obtaining the text representation of the concept to be tested by the BERT sub-model and obtaining the image representation of the concept to be tested by the image classification sub-model; step S2-2, concatenating the text representation and the image representation to form a vector as the multimodal representation of the concept to be tested; step S2-3, inputting the multimodal representation into a binary classification network to obtain the probability that the concept to be tested belongs to a visual concept.

[0008] The method for recognizing visual concepts for constructing multimodal knowledge graphs provided by this invention may also have the following technical features: the multimodal visual concept classification model is trained using a two-stage PU Learning strategy. The training process is as follows: in the first stage, a reliable negative set of the same size as the positive set is selected from the unlabeled data using a network-supervised initialization method. The reliable negative set is then combined with the labeled positive set to form an initial training set. In the second stage, the binary classification model is iteratively trained using the initial training set until the accuracy of the binary classification model no longer improves.

[0009] The method for recognizing visual concepts for constructing multimodal knowledge graphs provided by this invention may also have the following technical features, wherein the process of obtaining a reliable negative set is as follows: Step T1, randomly select a portion of data from the unlabeled data as "pseudo-negative examples"; Step T2, mix the "pseudo-negative examples" and positive examples and use them to train a binary classification model; Step T3, use the trained binary classification model to predict the unlabeled data to obtain several candidate negative examples; Step T4, by evaluating the diversity of the concept network image of the unlabeled data, select reliable negative examples from the candidate negative examples; Step T5, calculate the confidence of the reliable negative examples, and take the predetermined number of candidate negative examples with the highest confidence as the reliable negative set.

[0010] The method for recognizing visual concepts for constructing multimodal knowledge graphs provided by this invention may also have the following technical features: in iterative training, after each iteration, the remaining unlabeled data is simply classified using the current binary classification model to obtain a coarsely screened set of positive and negative data, and k candidate sets are selected from these sets and added to the training set. The new training set is then used for the next round of training.

[0011] Invention Function and Effect

[0012] This invention, regarding a method for recognizing visual concepts in multimodal knowledge graph construction, explicitly addresses the problem of targeted visual concept recognition during multimodal knowledge graph construction for the first time. This problem is modeled as a binary classification problem, and the feature data required to solve this classification problem is selected and designed. This results in the construction of a two-stream multimodal classification model incorporating both a BERT model and an image classification model, used to encode and fuse the textual and image features of the concept to be tested, thereby achieving the goal of distinguishing between visual and non-visual concepts. Furthermore, this invention's multimodal classification model employs a two-stage PU Learning training method, achieving fully automated initialization of training data. It uses a self-training approach to automatically and effectively expand the training dataset during iterations, completely avoiding manual intervention and improving the accuracy of the classification network. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the structure of the multimodal visual concept classification model in an embodiment of the present invention;

[0014] Figure 2 This is a flowchart of the two-stage PU Learning training process in an embodiment of the present invention. Detailed Implementation

[0015] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following describes in detail the method for recognizing visual concepts for constructing multimodal knowledge graphs according to this invention, in conjunction with embodiments and accompanying drawings.

[0016] <Example>

[0017] This invention employs a multimodal visual concept classification model to identify whether the concept under test belongs to a visual concept or a non-visual concept, and outputs the probability that the concept under test belongs to a visual concept.

[0018] Figure 1 This is a schematic diagram of the structure of the multimodal visual concept classification model in an embodiment of the present invention.

[0019] like Figure 1As shown, the multimodal visual concept classification model in this embodiment has a BERT sub-model, an image classification sub-model, and a binary classification network.

[0020] The BERT sub-model is used to obtain the textual representation of the concept to be tested. Specifically:

[0021] First, define the concept d. i The input format processed for BERT is: "[CLS]d i The [SEP] tag is then fed into the BERT model. The embedding of the [CLS] tag is further encoded into a 256-dimensional vector through a fully connected layer. The output of this fully connected layer is the final embedding of the text. That is, text representation.

[0022] The image classification sub-model is used to obtain an image representation of the concept to be tested. Specifically:

[0023] In this embodiment, several pre-trained image classification models (such as InceptionV3, ResNet50, and VGG16) are used to obtain image representations. M images are input into the pre-trained image classification model, and M feature vectors are obtained from the output of the final pooling layer. d CNN This represents the dimension of the average pooling layer in a CNN model (image classification model), and it varies depending on the CNN model. For example, in InceptionV3, d... CNN It's 2048. These M vectors are concatenated together and fed into a fully connected layer to obtain the final visual feature vector. That is, image representation.

[0024] The binary classification network consists of a 64-dimensional hidden layer and an output layer, used to identify the multimodal representation of the test concept and output the probability that the test concept belongs to a visual concept. The multimodal representation is a 512-dimensional vector formed by concatenating the text and image representations of the test concept.

[0025] In this embodiment, the multimodal visual concept classification model is obtained by iteratively training a binary classification model that distinguishes between visual and non-visual concepts using a self-training strategy. Specifically:

[0026] To train a multimodal classifier using supervised learning, we first create a partially labeled dataset. Currently, there is no dataset that directly identifies which concepts are non-visual, but the labels in existing image datasets clearly have corresponding manually labeled images; these concepts can be considered visual concepts.

[0027] Therefore, this embodiment trains on the WordNet noun set, which contains 74,374 concepts. The relevant visual concepts are then labeled using the ILSVRC image dataset (a subset of ImageNet) built on a WordNet-based structure, leaving the remaining concepts in WordNet unlabeled. Although ILSVRC contains far fewer concepts than ImageNet, it is a benchmark dataset for many tasks in computer vision, and its quality has been extensively tested, making it more reliable than ImageNet.

[0028] For each concept, data from two modalities were collected as features for training: the definition text from WordNet and web images retrieved from the Google Image Search engine. The search query format was "cd", where "c" is the name of a concept and "d" is its definition in WordNet, used to disambiguate concepts with the same name during retrieval. Furthermore, some concepts in WordNet had duplicate definitions, so these were removed from the training set, resulting in a positive sample set of 999 and an unlabeled dataset containing 73,183 samples. Simultaneously, 600 concepts were randomly selected from WordNet and manually labeled to form a test set, which included 322 positive samples and 278 negative samples.

[0029] As shown above, since there are only about 1,000 positive examples labeled and a lack of negative examples of the same size, this embodiment adopts a two-stage PU Learning strategy. That is, firstly, in order to supplement the binary classification training dataset, a network-supervised initialization step is designed to select a reliable negative set of the same size as the positive set from the unlabeled data, and then a self-training strategy is used to iteratively train the binary classification model.

[0030] Figure 2 This is a flowchart of the two-stage PU Learning training process in an embodiment of the present invention.

[0031] like Figure 2 As shown in this embodiment, the process of self-training and iteratively training the binary classification model using a two-stage PU Learning strategy is as follows:

[0032] First, complete the negative examples needed to form a binary classification training dataset. Typically, this step requires manual annotation; however, this embodiment proposes a network-supervised automatic method to construct a high-quality negative set without manual annotation. The method includes the following steps:

[0033] Step T1: Randomly select a portion of data from the unlabeled data as "pseudo-negative samples";

[0034] Step T2: Mix the "pseudo-negative examples" and positive examples to train the binary classification model;

[0035] Step T3: Use the trained binary classification model to predict the unlabeled data to obtain several candidate negative samples;

[0036] Step T4: Reliable negative examples are selected from the candidate negative examples by evaluating the diversity of the concept network images of unlabeled data;

[0037] Step T5: Calculate the confidence level of reliable negative samples, and take the predetermined number of candidate negative samples with the highest confidence level as the reliable negative set.

[0038] In the process of obtaining the reliable negative set, the more cluttered the network image of a concept, the richer its diversity, and the more likely that concept is a non-visual concept. The diversity calculation formula is as follows:

[0039]

[0040] In the formula, It refers to the output features of the i-th image of the current concept c through the pre-trained image classification model, and n is the number of images crawled from the Internet, which is 16 in this embodiment.

[0041] By comprehensively considering the probability output by the classification network and the diversity calculated using the above formula, the confidence level of the candidate concept as a non-visual entity can be obtained:

[0042] Conf(c)=softmax(1-p(c))+softmax(D(c))

[0043] In the formula, p(c) is the probability that the concept output by the classification network is a visual concept.

[0044] Then, the 999 candidate negative samples with the highest confidence are selected as reliable negative samples and combined with the previously labeled positive samples to form an initial training set Tr0 to start the second stage of iterative training.

[0045] In the second stage of iterative training, after each iteration, the remaining unlabeled data is simply classified using the current binary classification model to obtain a coarse set of positive and negative data. From this set, k candidate sets are selected and added to the training set. The new training set is then used for the next round of training, iterating until the accuracy of the classification network no longer improves.

[0046] Meanwhile, this embodiment employs two techniques to avoid noise that may arise from the automated extraction of negative samples in Phase 1. Specifically:

[0047] Random sampling: Taking negative data as an example, instead of using the top k candidates as pseudo-labels, k candidates are randomly sampled based on their confidence scores. For example, if the confidence score of concept c is 0.8, then the probability of it being selected is 0.8.

[0048] Additional visual information: Reusing prior knowledge, i.e., network images of non-visual concepts have higher diversity (measured by their standard deviation). The diversity score is added to the confidence score calculation.

[0049] Several simple conjectures about visual entities have been proposed in existing recognition methods, and methods for recognizing visual entities based on these conjectures have been put forward without verification. This embodiment demonstrates, through validation on a test set, that these approaches are too crude and cannot yield accurate results. Meanwhile, the method proposed in this embodiment for classifying visual and non-visual concepts using text and image features has been rigorously validated on a test set and proven to be correct and effective.

[0050] In this embodiment, a trained multimodal visual concept classification model was used to identify visual entities, and its recognition accuracy was quantitatively analyzed, achieving an accuracy of 0.83, far exceeding existing concept recognition technologies. Therefore, the method for recognizing visual concepts based on multimodal knowledge graph construction of this invention not only ensures recognition accuracy but also allows the recognition results to be extended to more downstream applications.

[0051] Functions and effects of the embodiments

[0052] The method for recognizing visual concepts for multimodal knowledge graph construction provided in this embodiment is the first to explicitly address the problem of targeted recognition of visual concepts during the construction of multimodal knowledge graphs. This problem is modeled as a binary classification problem, and the feature data required to solve this classification problem is selected and designed. In this way, a two-stream multimodal classification model containing a BERT model and an image classification model is constructed to encode and fuse text features and image features, thereby achieving the purpose of distinguishing between visual concepts and non-visual concepts.

[0053] In this embodiment, when labeled data is lacking, a two-stage PU Learning training method is employed. Positive examples are labeled using the inherent characteristics of the existing image dataset, and reliable negative examples are selected through network image supervision. This method initializes the training data in a completely automated manner, and automatically and effectively expands the training dataset iteratively using a self-training approach, completely avoiding manual intervention during the expansion process.

[0054] In the embodiment, to address the noise that may be introduced by the automated extraction of negative samples in the first stage, random sampling and additional visual information are designed to avoid noise, thereby improving the accuracy of the classification network.

[0055] The above embodiments are only used to illustrate specific implementations of the present invention, and the present invention is not limited to the scope of the description of the above embodiments.

Claims

1. A method for recognizing visual concepts for multi-modal knowledge graph construction, characterized in that, The method comprises the following steps: Step S1, a binary classification model distinguishing visual concepts and non-visual concepts is constructed, and self-training is used for iterative training, so that a multi-modal visual concept classification model is obtained; Step S2, a to-be-tested concept is input into the multi-modal visual concept classification model, so that a probability that the to-be-tested concept is a visual concept is obtained, The multi-modal visual concept classification model has a BERT sub-model, an image classification sub-model, and a binary classification network, The multi-modal visual concept classification model is trained by using a two-stage PU Learning learning strategy, and the training process is as follows: In the first stage, a network supervised initialization method is used to select a reliable negative set with the same size as the positive set from unlabeled data, and the reliable negative set and the labeled positive set are combined into a preliminary training set, In the second stage, the binary classification model is iteratively trained by using the preliminary training set until the accuracy of the binary classification model no longer improves, The reliable negative set is obtained as follows: Step T1, a part of data is randomly extracted from unlabeled data as pseudo-negative samples; Step T2, the pseudo-negative samples and positive samples are mixed to train the binary classification model; Step T3, the trained binary classification model is used to predict the unlabeled data to obtain a plurality of candidate negative samples; Step T4, the diversity of the concept network image of the unlabeled data is evaluated, and reliable negative samples are selected from the candidate negative samples; Step T5, the confidence of the reliable negative samples is calculated, and a predetermined number of candidate negative samples with the highest confidence are taken as the reliable negative set, The calculation formula of the diversity is as follows: wherein is the output feature of the ith image of concept c by the pre-trained image classification model, n is the number of images crawled from the web, The calculation formula of the confidence is as follows: Conf(c)=softmax(1-p(c))+softmax(D(c)) In the formula, p(c) is the probability that the classification network outputs that the concept is a visual concept.

2. The method for identifying visual concepts based on a multi-modal knowledge graph according to claim 1, wherein: wherein Step S2 comprises the following sub-steps: Step S2-1, a text representation of the to-be-tested concept is obtained by the BERT sub-model, and an image representation of the to-be-tested concept is obtained by the image classification model; Step S2-2, the text representation and the image representation are connected to form a vector as a multi-modal representation of the to-be-tested concept; Step S2-3, the multi-modal representation is input into the binary classification network to obtain the probability that the to-be-tested concept belongs to a visual concept.

3. The method for identifying visual concepts based on a multi-modal knowledge graph according to claim 1, wherein: wherein, In the iterative training, after each iteration, the remaining unlabeled data is simply classified by using the current binary classification model to obtain a coarsely screened positive and negative set, k candidate sets are selected from the coarsely screened positive and negative set and added to the training set, and the next round of training is performed by using the new training set.

Images