Medical-oriented few-sample multi-modal fine-grained entity classification method and system

Abstract

The invention relates to the technical field of electric digital data processing, in particular to a medical-oriented few-sample multi-modal fine-grained entity classification method and system, and the method comprises the following steps: constructing a few-sample training set, a few-sample verification set and a few-sample test set, and expanding text information to obtain an expanded training set; obtaining a text representation of the sentence; obtaining global visual representation, local object representation and cross-modal semantic representation of the image; obtaining an optimized single mode representation; obtaining a multi-modal fusion representation; calculating the prediction probability and binary cross entropy loss of each fine-grained entity category; iterative optimization training is carried out until the set number of times is reached, performance verification is carried out on the few-sample verification set, and final effect evaluation is carried out on the few-sample test set. According to the method and the system provided by the invention, the performance of fine-grained classification is improved by effectively extracting hierarchical multi-modal features, and a more accurate and reliable entity classification technology can be provided for the medical field.

Description

Technical Field

[0001] The present invention relates to the field of electronic digital data processing technology, and in particular to a medical-oriented few-sample multi-modal fine-grained entity classification method and system. Background Art

[0002] In today's era of rapid development of digital information, the amount of data is showing an explosive growth trend, and a large amount of data exists in the form of text. According to statistics, the global Internet generates about 2.5 trillion bytes of data every day, of which more than 80% is unstructured text. Faced with this situation, information extraction technology has become the core means to extract structured knowledge from massive text. In the field of natural language processing, entity classification, as a basic task, has evolved from symbolic rule systems to statistical learning models. However, the massive text data generated in the Internet era presents highly granular and specialized characteristics. Traditional methods cannot distinguish fine-grained entities in the medical field, which puts higher requirements on entity classification technology and gives rise to the research direction of fine-grained entity classification.

[0003] Fine-grained entity classification is a fundamental and important task for extracting knowledge from unstructured text. It aims to assign context-sensitive fine-grained semantic types to entities in context. Fine-grained entity classification serves many downstream natural language processing tasks, such as relation extraction and entity linking, so it is the basis for building knowledge graphs. In Internet media and newspapers, text often appears with images, and images also contain rich semantic information, which provides additional help for fine-grained entity classification tasks. Therefore, multimodal fine-grained entity classification uses the rich fine-grained visual cues contained in images to assist in fine-grained entity classification.

[0004] However, in real-world application scenarios, medical care, as a specific field, requires experts to spend a lot of time labeling to obtain a large amount of high-quality multimodal data. In addition, training with such a large-scale dataset requires considerable time and computing resources. Therefore, this is a task that is both labor-intensive and resource-intensive. When high-quality labeled multimodal data is insufficient, performance will drop significantly. Therefore, how to perform accurate multimodal fine-grained entity classification with limited labeled data is an urgent problem to be solved. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a medical-oriented few-sample multimodal fine-grained entity classification method and system, which improves the accuracy of fine-grained entity classification by performing information enhancement from the text and image perspectives respectively and using a hierarchical cross-modal semantic alignment network to fuse text information with image information from a hierarchical multi-granularity visual semantic pyramid module.

[0006] The present invention is achieved through the following technical solutions: A medical-oriented few-sample multimodal fine-grained entity classification method comprises the following steps: S1: Based on the medical FIGER dataset, a few-shot training set, a few-shot validation set, and a few-shot test set are constructed. The text in the few-shot training set is expanded with text information through semantic interpolation to obtain an expanded training set. S2: Arrange the sentences in each sample in the expanded training set, the few-shot validation set, and the few-shot test set to obtain the representation of the sentence, and then input the representation of the sentence into the text feature extraction model to obtain the text representation of the sentence; S3: Integrate the images in each sample in the expanded training set, few-shot validation set, and few-shot test set through a hierarchical multi-granularity visual semantic pyramid module to obtain the global visual representation, local object representation, and cross-modal semantic representation of the image; S4: The text representation of the sentence, the global visual representation of the image, the local object representation and the cross-modal semantic representation are optimized in a unimodal form using a multi-head self-attention mechanism to obtain an optimized unimodal representation; S5: Input the optimized unimodal representations into the hierarchical cross-modal semantic alignment network for multimodal feature fusion to obtain text cross-object representation, text cross-modal semantic representation, and text cross-visual representation, and concatenate the text cross-object representation, text cross-modal semantic representation, and text cross-visual representation to obtain a multimodal fusion representation; S6: Input the multimodal fusion representation into the classifier to calculate the prediction probability and binary cross entropy loss of each fine-grained entity category; S7: Perform iterative optimization training on the expanded training set using binary cross entropy loss until the set number of times is reached. After determining the relevant parameters, perform performance verification on the few-shot validation set and conduct final effect evaluation on the few-shot test set to quantify the performance of the model in the few-shot multimodal fine-grained entity classification task.

[0007] Furthermore, in step S1, the following method is used to construct a few-sample training set, a few-sample validation set, and a few-sample test set based on the medical FIGER dataset: S11: Based on a mapping file from Wikipedia titles to knowledge graph unique identifiers, retrieve the knowledge graph unique identifier corresponding to the entity mention in each sentence context of the medical FIGER dataset; S12: Based on a mapping file from knowledge graph entities to Wikidata entities, according to the unique identifier of the knowledge graph corresponding to the entity mention in each sentence context, obtain the Wikidata identifier corresponding to the entity mention in the sentence context; S13: obtaining an image from a Wikidata webpage according to a Wikidata identifier corresponding to the entity mention in the sentence context, thereby obtaining a corresponding image for the entity mention in the sentence context, so that a sentence and a corresponding image constitute a sample; S14: Divide the constructed samples into a training set, a validation set, and a test set according to a set ratio, and screen the samples in the training set, delete the categories corresponding to the fine-grained entity classifications with fewer samples than the set number, and randomly delete the samples in the retained categories that exceed the set number to form a few-sample training set, and at the same time delete the samples corresponding to the corresponding categories in the validation set and the test set, and randomly delete the samples in the categories retained in the validation set that exceed the set number to form a few-sample validation set and a few-sample test set.

[0008] Furthermore, the method for obtaining the text representation of the sentence in step S2 is as follows: S21: Arrange the sentences in each sample in the expanded training set, the few-shot validation set, and the few-shot test set according to formula (1) to obtain the representation of the sentence: (1); in: The expression of the sentence, Indicates the classification mark, Indicates entity mention, Indicates a separator mark. Represents the textual context of the sentence; S22: The representation of the sentence is processed by the word segmenter of the text feature extraction model to obtain the input word sequence, and then the core architecture of the text feature extraction model is used to obtain the first The contextualized word representation of the input word unit is used, and the contextualized word representation of the last hidden layer is used as the text representation of the entire sentence: (2); in: Indicates The contextualized word representation of the input tokens, represents the text feature extraction model, Indicates Input word, Represents the parameters of the text feature extraction model, Represents the dimension of the hidden state of the text feature extraction model, The dimension representing the hidden state of the text feature extraction model is The vector space of .

[0009] Furthermore, in step S3, the global visual representation, local object representation and cross-modal semantic representation of the image are obtained by the following method: S31: inputting the image of each sample in the expanded training set, the few-shot validation set and the few-shot test set into the hierarchical multi-granularity visual semantic pyramid module, the global visual encoder of the hierarchical multi-granularity visual semantic pyramid module adjusts the image to the set pixel, and uses the last hidden layer state of the global visual encoder as the global visual representation of the image; S32: The local visual encoder of the hierarchical multi-granularity visual semantic pyramid module processes the image according to formula (3) to obtain the local object representation of the image: (3); in: represents the local object representation of the image, represents the target detection model, Indicates the sequence number of the image. Indicates the number of detected targets, represents the feature dimension of the detected target, express dimensional vector space; S33: The additional visual encoder of the hierarchical multi-granularity visual semantic pyramid module processes the image according to formula (4) to obtain the natural language text describing the image, adds a classification tag in front of the natural language text describing the image, adds a separation tag after the natural language text describing the image, and then inputs it into the text feature extraction model. The text feature extraction model processes according to formula (5) to obtain the cross-modal semantic representation of the image: (4); (5); in: represents the natural language text describing the image, represents the image description generation model that constitutes the additional visual encoder, Represents the cross-modal semantic representation of images, represents the text feature extraction model, represents the length of the sequence of natural language text describing the image, Represents the dimension of the hidden state of the text feature extraction model, express dimensional vector space, Indicates the classification mark, Indicates a delimiter mark.

[0010] In step S4, the text representation of the sentence, the global visual representation of the image, the local object representation, and the cross-modal semantic representation are optimized in a unimodal form by applying a multi-head self-attention mechanism according to formula (6) to obtain the optimized unimodal representation: (6); in: represents the optimized unimodal representation, represents the multi-head self-attention mechanism, represents the query matrix of the multi-head self-attention mechanism, represents the key matrix of the multi-head self-attention mechanism, represents the value matrix of the multi-head self-attention mechanism, Represents a splicing operation, Indicates The output of an attention head is represents the number of attention heads, Represents the linear transformation weight matrix.

[0011] Further, in step S5, the optimized unimodal representations are respectively input into the hierarchical cross-modal semantic alignment network for multimodal feature fusion to obtain text cross-object representation, text cross-modal semantic representation, and text cross-visual representation, and the text cross-object representation, text cross-modal semantic representation, and text cross-visual representation are spliced ​​to obtain a multimodal fusion representation: S51: The hierarchical cross-modal semantic alignment network uses the global visual representation in the optimized unimodal representation as the query matrix, and the text representation in the optimized unimodal representation as the key matrix and the value matrix to obtain the image-perceived text representation. The text representation in the optimized unimodal representation is used as the query matrix, and the image-perceived text representation is used as the key matrix and the value matrix to obtain the refined image-perceived text representation. The text representation in the optimized unimodal representation is used as the query matrix, and the global visual representation in the optimized unimodal representation is used as the key matrix and the value matrix to finally obtain the refined text-perceived image representation. The refined image-perceived text representation and the refined text-perceived image representation are then gated to obtain the text cross-visual representation. S52: The hierarchical cross-modal semantic alignment network uses the local object representation in the optimized unimodal representation as the query matrix, and the text representation in the optimized unimodal representation as the key matrix and the value matrix to obtain the local object-aware text representation, uses the text representation in the optimized unimodal representation as the query matrix, and the local object representation in the optimized unimodal representation as the key matrix and the value matrix to obtain the refined local object-aware text representation, and then uses the text representation in the optimized unimodal representation as the query matrix, and the local object representation in the optimized unimodal representation as the key matrix and the value matrix to finally obtain the refined text-aware local object representation, and then uses the refined local object-aware text representation and the refined text-aware local object representation through the gate control function to obtain the text cross-object representation; S53: The hierarchical cross-modal semantic alignment network uses the cross-modal semantic representation in the optimized unimodal representation as the query matrix, and the text representation in the optimized unimodal representation as the key matrix and the value matrix to obtain the cross-modal perceived text representation, uses the text representation in the optimized unimodal representation as the query matrix, and the cross-modal semantic representation in the optimized unimodal representation as the key matrix and the value matrix to obtain the refined cross-modal perceived text representation, and then uses the text representation in the optimized unimodal representation as the query matrix, and the cross-modal semantic representation in the optimized unimodal representation as the key matrix and the value matrix to finally obtain the refined text-aware cross-modal semantic representation, and then uses the refined cross-modal perceived text representation and the refined text-aware cross-modal semantic representation through the gate control function to obtain the text cross-modal semantic representation; S54: splicing text cross-object representation, text cross-modal semantic representation, and text cross-visual representation to obtain a multi-granular cross-modal representation of text perception; S55: Concatenate the text representation in the optimized unimodal representation with the text-aware multi-granularity cross-modal representation to obtain a multimodal fusion representation.

[0012] Optimally, in step S6, the following method is used to input the multimodal fusion representation into the classifier to calculate the prediction probability and binary cross entropy loss of each fine-grained entity category: S61: The classifier calculates the predicted probability of each fine-grained entity category according to formula (7): (7); in: Indicates the predicted vector length type vector, represents the linear transformation matrix, represents the text representation in the optimized unimodal representation, Representing text-aware multi-granular cross-modal representations, represents multimodal fusion representation, Indicates The predicted probability of fine-grained entity categories, represents the total number of fine-grained entity categories in the expanded training set, Represents the predicted probability of the last fine-grained entity category; S62: Calculate the binary cross entropy loss according to formula (8): (8); in: represents the binary cross entropy loss, Indicates Sample category labels.

[0013] Optimized, in step S7, the following method is used to perform iterative optimization training on the expanded training set through binary cross entropy loss: During the training process, the binary cross entropy loss is monitored in real time. If the binary cross entropy loss does not decrease after three consecutive iterations, the early stopping strategy is used to terminate the training. Otherwise, the training is terminated after the set number of iterations.

[0014] Furthermore, in the process of performance verification on the few-sample validation set and final effect evaluation on the few-sample test set, if If the predicted probability of a fine-grained entity category is greater than 0.5, the fine-grained entity is classified as the type corresponding to the fine-grained entity category. If the predicted probabilities of multiple fine-grained entity categories are greater than 0.5, the fine-grained entities are classified into the types corresponding to these fine-grained entity categories. The fine-grained entity categories have multiple type labels. If the prediction probabilities of all fine-grained entity categories are less than or equal to 0.5, the type corresponding to the value of the fine-grained entity category with the highest prediction probability is selected as the final prediction type.

[0015] A medical-oriented few-sample multimodal fine-grained entity classification system, used to execute a medical-oriented few-sample multimodal fine-grained entity classification method as described in any one of the above, comprising a medical-based FIGER dataset, a context-aware semantic difference module, a text feature extraction model, a hierarchical multi-granularity visual semantic pyramid module, a unimodal representation optimization module, a hierarchical cross-modal semantic alignment network and a classifier; The medical-based FIGER dataset is used to provide a fine-grained entity classification dataset for a medical-oriented few-sample multimodal fine-grained entity classification system; The context-aware semantic difference module is used to expand text information for the text in the few-sample training set to obtain an expanded training set; The text feature extraction model is used to obtain text representation of sentences from each sentence in the expanded training set, the few-sample validation set and the few-sample test set; The hierarchical multi-granularity visual semantic pyramid module is used to integrate the images in each sample in the expanded training set, the few-shot validation set and the few-shot test set to obtain the global visual representation, local object representation and cross-modal semantic representation of the image, wherein the global visual encoder in the hierarchical multi-granularity visual semantic pyramid module is used to obtain the global visual representation of the image, the local visual encoder is used to obtain the local object representation, and the additional visual encoder is used to obtain the cross-modal semantic representation; The unimodal representation optimization module is used to optimize the text representation of the sentence, the global visual representation of the image, the local object representation and the cross-modal semantic representation in a unimodal form by applying a multi-head self-attention mechanism to obtain an optimized unimodal representation; The hierarchical cross-modal semantic alignment network is used to fuse the optimized single-modal representation with multi-modal features to obtain a multi-modal fusion representation; The classifier is used to calculate the prediction probability and binary cross entropy loss of each fine-grained entity category, and classify medical-oriented few-sample multimodal fine-grained entities.

[0016] Beneficial effects of the invention: The present invention provides a medical-oriented few-sample multimodal fine-grained entity classification method and system, which has the following advantages: The present invention innovatively proposes a multi-view information enhancement framework, which uses a context-aware semantic interpolation method to enhance text information from a text perspective, and uses a hierarchical multi-granularity visual semantic pyramid to enhance image information from an image perspective. Only a small amount of multimodal samples are used for training. By effectively extracting hierarchical multimodal features, the performance of fine-grained classification can be further improved, and a more accurate and reliable entity classification technology can be provided for the medical field. It has broad prospects in practical applications and contains huge commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the process of the present invention.

[0018] Figure 2 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0019] A medical-oriented few-sample multimodal fine-grained entity classification method includes the following steps, and its flowchart is as follows: Figure 1 As shown: S1: Based on the medical FIGER dataset, a few-shot training set, a few-shot validation set, and a few-shot test set are constructed. The text in the few-shot training set is expanded with text information through semantic interpolation to obtain an expanded training set. FIGER is a dataset for fine-grained entity classification based on medical data, with a two-level hierarchy derived from the public domain knowledge graph Freebase. The training set consists of Wikipedia articles automatically annotated using a distant supervision method that encodes information from anchor text links. The test set is manually annotated. The Multimodal Fine-Grained Entity Classification (MFIGER) dataset is built on FIGER for multimodal fine-grained entity classification.

[0020] Specifically, the following method can be used to construct a few-sample training set, a few-sample validation set, and a few-sample test set based on the medical FIGER dataset: S11: Based on a mapping file from Wikipedia titles to knowledge graph (Freebase) unique identifiers (mid), retrieve the unique identifiers of the knowledge graph corresponding to the entity mentions in each sentence context of the medical FIGER dataset; S12: Based on a mapping file from knowledge graph entities to Wikidata entities, according to the unique identifier of the knowledge graph corresponding to the entity mention in each sentence context, obtain the Wikidata identifier corresponding to the entity mention in the sentence context; S13: obtaining an image from a Wikidata webpage according to a Wikidata identifier corresponding to the entity mention in the sentence context, thereby obtaining a corresponding image for the entity mention in the sentence context, so that a sentence and a corresponding image constitute a sample; S14: Divide the constructed samples into a training set, a validation set, and a test set according to a set ratio, and screen the samples in the training set, delete the categories corresponding to the fine-grained entity classifications with fewer samples than the set number, and randomly delete the samples in the retained categories that exceed the set number to form a few-sample training set, and at the same time delete the samples corresponding to the corresponding categories in the validation set and the test set, and randomly delete the samples in the categories retained in the validation set that exceed the set number to form a few-sample validation set and a few-sample test set.

[0021] Specifically, it can be divided into training set, validation set and test set in a ratio of 7:1:2.

[0022] When screening the samples in the training set, you can set 5 training samples for each category. First, screen out categories with no less than 5 samples from the training set, then retain only samples of these categories in the training set, validation set, and test set, and then randomly delete samples with more than 5 samples in the retained categories of the training set and validation set, thereby forming a few-sample training set, a few-sample validation set, and a few-sample test set. Finally, the number of samples in each category in the few-sample training set and the few-sample validation set is only 5.

[0023] Considering that entity mentions belonging to the same category have similar textual contexts, these sentences may contain the same or similar words or phrases. Therefore, the present invention proposes a context-aware semantic interpolation method to expand text information by replacing the context of entities belonging to the same category. Specifically, when there are only 5 training samples in each category in the few-shot training set, by replacing the context of entity mentions belonging to the same category, the training data scale of the few-shot training set is expanded to 5 times, and an expanded training set is obtained, thereby enhancing the text information of the few-shot training set.

[0024] sample Generated augmented training samples The formula is as follows: ; in: Indicates Sample entity mentions, Indicates Sample contexts, It is Sample pictures, Indicates Sample category labels, Indicates that the two samples have the same type set, is the context of the jth sample, Indicates the number of samples set for each category in the few-shot training set.

[0025] S2: Arrange the sentences in each sample in the expanded training set, the few-shot validation set, and the few-shot test set to obtain the representation of the sentence, and then input the representation of the sentence into the text feature extraction model to obtain the text representation of the sentence; Specifically, the text representation of the sentence can be obtained by the following method: S21: Arrange the sentences in each sample in the expanded training set, the few-shot validation set, and the few-shot test set according to formula (1) to obtain the representation of the sentence: (1); in: Indicates the expression of the sentence, Indicates the classification mark, Indicates entity mention, Indicates a separator mark. Represents the textual context of the sentence; and There are two special tags in the text feature extraction model. [CLS] (Classification Token) is placed at the beginning of the input text. Its corresponding final hidden state is often used as the feature representation of the entire text for classification tasks. [SEP] (Separator Token) is used to separate different text fragments, such as distinguishing two sentences when processing sentence pairs.

[0026] S22: The representation of the sentence is processed by the word segmenter of the text feature extraction model to obtain the input word sequence, and then the core architecture of the text feature extraction model is used to obtain the first The contextualized word representation of the input word unit is used, and the contextualized word representation of the last hidden layer is used as the text representation of the entire sentence: (2); in: Indicates The contextualized word representation of the input tokens, represents the text feature extraction model, Indicates Input word, Represents the parameters of the text feature extraction model, Represents the dimension of the hidden state of the text feature extraction model, The dimension representing the hidden state of the text feature extraction model is The vector space of .

[0027] After the above steps, the text information is enhanced, and the text representation is extracted using the text feature extraction model. Finally, a deeply contextualized text representation is obtained after information enhancement. More effective information is extracted from a small amount of text samples, providing a basis for subsequent multimodal fusion and fine-grained classification.

[0028] S3: Integrate the images in each sample in the expanded training set, few-shot validation set, and few-shot test set through a hierarchical multi-granularity visual semantic pyramid module to obtain the global visual representation, local object representation, and cross-modal semantic representation of the image; Specifically, the following methods can be used to obtain the global visual representation, local object representation, and cross-modal semantic representation of the image: S31: inputting the image of each sample in the expanded training set, the few-shot validation set and the few-shot test set into the hierarchical multi-granularity visual semantic pyramid module, the global visual encoder of the hierarchical multi-granularity visual semantic pyramid module adjusts the image to the set pixel, and uses the last hidden layer state of the global visual encoder as the global visual representation of the image; The global visual encoder here is a pre-trained Vision Transformer model (ViT). The global visual representation from the Vision Transformer model (ViT) provides global semantic features of the entire image, which helps to enhance the model's ability to grasp the comprehensive content and contextual information of the image.

[0029] The set pixel here is 224×224 pixels.

[0030] The global visual representation is as follows: ; in: represents the global visual representation, Indicates the sequence number of the image. Indicates that the content in brackets is processed by the Vision Transformer model. Represents the parameters of the Vision Transformer model, express dimensional vector space, represents the number of image tags, Represents the dimension of the hidden state of the Vision Transformer model.

[0031] S32: The local visual encoder of the hierarchical multi-granularity visual semantic pyramid module processes the image according to formula (3) to obtain the local object representation of the image: (3); in: represents the local object representation of the image, represents the target detection model, Indicates the number of detected targets. represents the feature dimension of the detected target, express dimensional vector space; Local object representations from object detection can capture fine-grained semantic information embedded in the visual context. Since few-shot multimodal fine-grained entity classification is a multi-class multi-label classification problem, the fine-grained types are organized into a certain hierarchy, so it is necessary to extract not only fine-grained semantic information from the text context, but also fine-grained visual cues from the visual context. The objects detected from the image provide powerful visual cues for predicting the set of fine-grained types of entity mentions.

[0032] The local visual encoder here can use the object detector of the object detection model (VinVL).

[0033] The cross-modal semantic representation from image description provides a high-level understanding and semantic interpretation of the image content. Image description enables the neural network model to understand the image and generate natural language descriptions to describe the content, scenes or objects in the image. The image description generation model (OFA) can be used as an additional visual encoder. Specifically, the following method can be used to obtain the cross-modal semantic representation of the image.

[0034] S33: The additional visual encoder of the hierarchical multi-granularity visual semantic pyramid module processes the image according to formula (4) to obtain the natural language text describing the image, adds a classification tag in front of the natural language text describing the image, adds a separation tag after the natural language text describing the image, and then inputs it into the text feature extraction model. The text feature extraction model processes according to formula (5) to obtain the cross-modal semantic representation of the image: (4); (5); in: represents the natural language text describing the image, represents the image description generation model that constitutes the additional visual encoder, Represents the cross-modal semantic representation of images, represents the text feature extraction model, represents the length of the sequence of natural language text describing the image, Represents the dimension of the hidden state of the text feature extraction model, express dimensional vector space, Indicates the classification mark, Indicates a delimiter mark.

[0035] The above-mentioned hierarchical multi-granularity visual semantic pyramid module extracts the global semantic perception, local object relations and cross-modal semantic interpretation features of the image. Through a multi-level and multi-granularity feature integration mechanism, it injects more comprehensive and detailed information into the image representation, thereby improving the accuracy and semantic richness of subsequent multimodal information interaction.

[0036] S4: The text representation of the sentence, the global visual representation of the image, the local object representation and the cross-modal semantic representation are optimized in a unimodal form using a multi-head self-attention mechanism to obtain an optimized unimodal representation; Specifically, the text representation of the sentence, the global visual representation of the image, the local object representation, and the cross-modal semantic representation are optimized in a unimodal form using a multi-head self-attention mechanism according to formula (6) to obtain the optimized unimodal representation: (6); in: represents the optimized unimodal representation, It can be an optimized text representation, an optimized global visual representation, an optimized local object representation of an image, or an optimized cross-modal semantic representation of an image.

[0037] represents the multi-head self-attention mechanism, represents the query matrix of the multi-head self-attention mechanism, represents the key matrix of the multi-head self-attention mechanism, represents the value matrix of the multi-head self-attention mechanism, Represents a splicing operation, represents the linear transformation weight matrix, Indicates The output of an attention head Represents the number of attention heads.

[0038] Here is The output of an attention head Indicates the number of attention heads There is the following formula: ; in: represents the attention mechanism, Indicates The attention heads are used to transform the weight matrix of the query matrix, Indicates The attention heads are used to transform the weight matrix of the key matrix, Indicates The attention heads are used to transform the weight matrix of the value matrix.

[0039] After the above operations, the representation of the single modality can be further optimized and strengthened to comprehensively capture the semantic associations within the modality, laying the foundation for subsequent multimodal fusion.

[0040] S5: Input the optimized unimodal representations into the hierarchical cross-modal semantic alignment network for multimodal feature fusion to obtain text cross-object representation, text cross-modal semantic representation, and text cross-visual representation, and concatenate the text cross-object representation, text cross-modal semantic representation, and text cross-visual representation to obtain a multimodal fusion representation; Specifically, the following method can be used to input the optimized unimodal representations into the hierarchical cross-modal semantic alignment network for multimodal feature fusion to obtain text cross-object representation, text cross-modal semantic representation, and text cross-visual representation, and then concatenate the text cross-object representation, text cross-modal semantic representation, and text cross-visual representation to obtain a multimodal fusion representation: S51: The hierarchical cross-modal semantic alignment network uses the global visual representation in the optimized unimodal representation as the query matrix, and the text representation in the optimized unimodal representation as the key matrix and the value matrix to obtain the image-perceived text representation. The text representation in the optimized unimodal representation is used as the query matrix, and the image-perceived text representation is used as the key matrix and the value matrix to obtain the refined image-perceived text representation. The text representation in the optimized unimodal representation is used as the query matrix, and the global visual representation in the optimized unimodal representation is used as the key matrix and the value matrix to finally obtain the refined text-perceived image representation. The refined image-perceived text representation and the refined text-perceived image representation are then gated to obtain the text cross-visual representation. The specific control formula is as follows: ; Among them: represents the gating signal, represents the Sigmoid activation function, represents the trainable parameter matrix used to transform the refined image-aware text representation when computing the gating signal, , represents the trainable parameter matrix used to transform the refined text-aware image representation when computing the gating signal, , express dimensional vector space, represents the refined image-aware text representation, represents the refined text-aware image representation, represents the training parameters, , represents the trainable bias vector, represents the cross-visual representation of text, , express dimensional vector space; Through the operation of this step, the global semantic features of text and image can be integrated.

[0041] S52: The hierarchical cross-modal semantic alignment network uses the local object representation in the optimized unimodal representation as the query matrix, and the text representation in the optimized unimodal representation as the key matrix and the value matrix to obtain the local object-aware text representation, uses the text representation in the optimized unimodal representation as the query matrix, and the local object representation in the optimized unimodal representation as the key matrix and the value matrix to obtain the refined local object-aware text representation, and then uses the text representation in the optimized unimodal representation as the query matrix, and the local object representation in the optimized unimodal representation as the key matrix and the value matrix to finally obtain the refined text-aware local object representation, and then uses the refined local object-aware text representation and the refined text-aware local object representation through the gate control function to obtain the text cross-object representation; The specific control formula is shown in the control formula of text cross-visual representation.

[0042] Through the operation of this step, text and local object features can be fused.

[0043] S53: The hierarchical cross-modal semantic alignment network uses the cross-modal semantic representation in the optimized unimodal representation as the query matrix, and the text representation in the optimized unimodal representation as the key matrix and the value matrix to obtain the cross-modal perceived text representation, uses the text representation in the optimized unimodal representation as the query matrix, and the cross-modal semantic representation in the optimized unimodal representation as the key matrix and the value matrix to obtain the refined cross-modal perceived text representation, and then uses the text representation in the optimized unimodal representation as the query matrix, and the cross-modal semantic representation in the optimized unimodal representation as the key matrix and the value matrix to finally obtain the refined text-aware cross-modal semantic representation, and then uses the refined cross-modal perceived text representation and the refined text-aware cross-modal semantic representation through the gate control function to obtain the text cross-modal semantic representation; The specific control formula is shown in the control formula of text cross-visual representation.

[0044] Through the operation of this step, text and cross-modal semantic features can be fused.

[0045] S54: splicing text cross-object representation, text cross-modal semantic representation, and text cross-visual representation to obtain a multi-granular cross-modal representation of text perception; The specific formula is as follows: ; in: Representing text-aware multi-granular cross-modal representations, express Activation function, represents the linear transformation matrix, represents the cross-object representation of text, Represents the cross-modal semantic representation of text, It means concatenating text cross-visual representation, text cross-object representation, and text cross-modal semantic representation.

[0046] S55: Concatenate the text representation in the optimized unimodal representation with the text-aware multi-granularity cross-modal representation to obtain a multimodal fusion representation.

[0047] The present invention adopts a hierarchical cross-modal semantic alignment network to grasp the intrinsic connection between textual modality and visual modality. That is, a hierarchical cross-modal semantic alignment network composed of stacked cross-modal Transformers is used to perform hierarchical multimodal feature fusion on the text representation in the optimized unimodal representation, the global visual representation in the optimized unimodal representation, the local object representation in the optimized unimodal representation, and the cross-modal semantic representation in the optimized unimodal representation.

[0048] S6: Input the multimodal fusion representation into the classifier to calculate the prediction probability and binary cross entropy loss of each fine-grained entity category; Optimally, in step S6, the following method is used to input the multimodal fusion representation into the classifier to calculate the prediction probability and binary cross entropy loss of each fine-grained entity category: S61: The classifier calculates the predicted probability of each fine-grained entity category according to formula (7): (7); in: Indicates the predicted vector length type vector, represents the linear transformation matrix, represents the text representation in the optimized unimodal representation, express, represents multimodal fusion representation, Indicates The predicted probability of fine-grained entity categories, represents the total number of fine-grained entity categories in the expanded training set, Represents the predicted probability of the last fine-grained entity category; S62: Calculate the binary cross entropy loss according to formula (8): (8); in: represents the binary cross entropy loss, Indicates Sample category labels.

[0049] S7: Perform iterative optimization training on the expanded training set using binary cross entropy loss until the set number of times is reached. After determining the relevant parameters, perform performance verification on the few-shot validation set and conduct final effect evaluation on the few-shot test set to quantify the performance of the model in the few-shot multimodal fine-grained entity classification task.

[0050] Specifically, the following method can be used to perform iterative optimization training on the expanded training set through binary cross entropy loss: During the training process, the binary cross entropy loss is monitored in real time. If the binary cross entropy loss does not decrease after three consecutive iterations, the early stopping strategy is used to terminate the training. Otherwise, the training is terminated after the set number of iterations.

[0051] Furthermore, in the process of performance verification on the few-sample validation set and final effect evaluation on the few-sample test set, if If the predicted probability of a fine-grained entity category is greater than 0.5, the fine-grained entity is classified as the type corresponding to the fine-grained entity category. If the predicted probabilities of multiple fine-grained entity categories are greater than 0.5, the fine-grained entities are classified into the types corresponding to these fine-grained entity categories. The fine-grained entity categories have multiple type labels. If the prediction probabilities of all fine-grained entity categories are less than or equal to 0.5, the type corresponding to the value of the fine-grained entity category with the highest prediction probability is selected as the final prediction type.

[0052] The effectiveness of the proposed method is verified by performing a few-shot multimodal fine-grained entity classification task on a few-shot validation set. The proposed method is compared with five few-shot fine-grained entity classification methods: Fine-tuning, Prompt-based MLM, PLET, ALIGNIE, FiveFine, two multimodal named entity recognition methods: UMT, MGCMT, and one multimodal fine-grained entity classification method: MOVCNet.

[0053] The performance comparison data of the present invention and other methods on the few-sample multimodal fine-grained entity classification task is shown in Table 1: Table 1

[0054] Experiments show that: both as a whole and at the coarse-grained or fine-grained level, this method surpasses all single-text and multimodal methods and achieves the best classification performance, effectively proving the rationality and effectiveness of the multi-view information enhanced few-sample multimodal fine-grained entity classification method proposed in the present invention.

[0055] A medical-oriented few-sample multimodal fine-grained entity classification system is used to execute a medical-oriented few-sample multimodal fine-grained entity classification method as described in any one of the above. The system structure diagram is as follows Figure 2 As shown, it includes the medical-based FIGER dataset, context-aware semantic difference module, text feature extraction model (BERT), hierarchical multi-granularity visual semantic pyramid module, unimodal representation optimization module, hierarchical cross-modal semantic alignment network and classifier; The medical FIGER dataset is used to provide a fine-grained entity classification dataset for the medical-oriented few-sample multimodal fine-grained entity classification system; The context-aware semantic difference module is used to expand the text information of the text in the few-shot training set to obtain an expanded training set; The text feature extraction model is used to obtain the text representation of sentences in each sample of the expanded training set, the few-shot validation set and the few-shot test set; The hierarchical multi-granularity visual semantic pyramid module is used to integrate the images in each sample in the expanded training set, the few-shot validation set and the few-shot test set to obtain the global visual representation, local object representation and cross-modal semantic representation of the image. The global visual encoder in the hierarchical multi-granularity visual semantic pyramid module is used to obtain the global visual representation of the image, the local visual encoder is used to obtain the local object representation, and the additional visual encoder is used to obtain the cross-modal semantic representation. The unimodal representation optimization module is used to optimize the text representation of the sentence, the global visual representation of the image, the local object representation and the cross-modal semantic representation in a unimodal form using a multi-head self-attention mechanism to obtain the optimized unimodal representation; The hierarchical cross-modal semantic alignment network is used to fuse the optimized unimodal representation with multimodal features to obtain a multimodal fusion representation. The classifier is used to calculate the prediction probability and binary cross entropy loss of each fine-grained entity category, and classify medical-oriented few-shot multimodal fine-grained entities.

[0056] In summary, the present invention provides a medical-oriented few-sample multimodal fine-grained entity classification method and system, which only uses a small number of multimodal samples for training. By effectively extracting hierarchical multimodal features, the performance of fine-grained classification can be further improved, and a more accurate and reliable entity classification technology can be provided for the medical field. It has broad prospects in practical applications and contains huge commercial value.

[0057] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A medical-oriented few-sample multimodal fine-grained entity classification method, characterized by: The steps include: S1: Based on the medical FIGER dataset, a few-shot training set, a few-shot validation set, and a few-shot test set are constructed. The text in the few-shot training set is expanded with text information through semantic interpolation to obtain an expanded training set. S2: Arrange the sentences in each sample in the expanded training set, the few-shot validation set, and the few-shot test set to obtain the representation of the sentence, and then input the representation of the sentence into the text feature extraction model to obtain the text representation of the sentence; S3: Integrate the images in each sample in the expanded training set, few-shot validation set, and few-shot test set through a hierarchical multi-granularity visual semantic pyramid module to obtain the global visual representation, local object representation, and cross-modal semantic representation of the image; S4: The text representation of the sentence, the global visual representation of the image, the local object representation and the cross-modal semantic representation are optimized in a unimodal form using a multi-head self-attention mechanism to obtain an optimized unimodal representation; S5: Input the optimized unimodal representations into the hierarchical cross-modal semantic alignment network for multimodal feature fusion to obtain text cross-object representation, text cross-modal semantic representation, and text cross-visual representation, and concatenate the text cross-object representation, text cross-modal semantic representation, and text cross-visual representation to obtain a multimodal fusion representation; S6: Input the multimodal fusion representation into the classifier to calculate the prediction probability and binary cross entropy loss of each fine-grained entity category; S7: Perform iterative optimization training on the expanded training set using binary cross entropy loss until the set number of times is reached. After determining the relevant parameters, perform performance verification on the few-shot validation set and conduct final effect evaluation on the few-shot test set to quantify the performance of the model in the few-shot multimodal fine-grained entity classification task.

2. According to claim 1, a medical-oriented few-sample multimodal fine-grained entity classification method is characterized by: In step S1, the following method is used to construct a few-sample training set, a few-sample validation set, and a few-sample test set based on the medical FIGER dataset: S11: Based on a mapping file from Wikipedia titles to knowledge graph unique identifiers, retrieve the knowledge graph unique identifier corresponding to the entity mention in each sentence context of the medical FIGER dataset; S12: Based on a mapping file from knowledge graph entities to Wikidata entities, according to the unique identifier of the knowledge graph corresponding to the entity mention in each sentence context, obtain the Wikidata identifier corresponding to the entity mention in the sentence context; S13: obtaining an image from a Wikidata webpage according to a Wikidata identifier corresponding to the entity mention in the sentence context, thereby obtaining a corresponding image for the entity mention in the sentence context, so that a sentence and a corresponding image constitute a sample; S14: Divide the constructed samples into a training set, a validation set, and a test set according to a set ratio, and screen the samples in the training set, delete the categories corresponding to the fine-grained entity classifications with fewer samples than the set number, and randomly delete the samples in the retained categories that exceed the set number to form a few-sample training set, and at the same time delete the samples corresponding to the corresponding categories in the validation set and the test set, and randomly delete the samples in the categories retained in the validation set that exceed the set number to form a few-sample validation set and a few-sample test set.

3. The medical-oriented few-sample multimodal fine-grained entity classification method according to claim 1, characterized in that: The method for obtaining the text representation of the sentence in step S2 is as follows: S21: Arrange the sentences in each sample in the expanded training set, the few-shot validation set, and the few-shot test set according to formula (1) to obtain the representation of the sentence: （1）； in: Indicates the expression of the sentence, Indicates the classification mark, Indicates entity mention, Indicates a separator mark. Represents the textual context of the sentence; S22: The representation of the sentence is processed by the word segmenter of the text feature extraction model to obtain the input word sequence, and then the core architecture of the text feature extraction model is used to obtain the first The contextualized word representation of the input word unit is used, and the contextualized word representation of the last hidden layer is used as the text representation of the entire sentence: （2）； in: Indicates The contextualized word representation of the input tokens, represents the text feature extraction model, Indicates Input word, Represents the parameters of the text feature extraction model, Represents the dimension of the hidden state of the text feature extraction model, The dimension representing the hidden state of the text feature extraction model is The vector space of .

4. The medical-oriented few-sample multimodal fine-grained entity classification method according to claim 1, characterized in that: In step S3, the global visual representation, local object representation and cross-modal semantic representation of the image are obtained by the following method: S31: inputting the image of each sample in the expanded training set, the few-shot validation set and the few-shot test set into the hierarchical multi-granularity visual semantic pyramid module, the global visual encoder of the hierarchical multi-granularity visual semantic pyramid module adjusts the image to the set pixel, and uses the last hidden layer state of the global visual encoder as the global visual representation of the image; S32: The local visual encoder of the hierarchical multi-granularity visual semantic pyramid module processes the image according to formula (3) to obtain the local object representation of the image: （3）； in: represents the local object representation of the image, represents the target detection model, Indicates the sequence number of the image. Indicates the number of detected targets. represents the feature dimension of the detected target, express dimensional vector space; S33: The additional visual encoder of the hierarchical multi-granularity visual semantic pyramid module processes the image according to formula (4) to obtain the natural language text describing the image, adds a classification tag in front of the natural language text describing the image, adds a separation tag after the natural language text describing the image, and then inputs it into the text feature extraction model. The text feature extraction model processes according to formula (5) to obtain the cross-modal semantic representation of the image: （4）； （5）； in: represents the natural language text describing the image, represents the image description generation model that constitutes the additional visual encoder, Represents the cross-modal semantic representation of images, represents the text feature extraction model, represents the length of the sequence of natural language text describing the image, Represents the dimension of the hidden state of the text feature extraction model, express dimensional vector space, Indicates the classification mark, Indicates a delimiter mark.

5. The medical-oriented few-sample multimodal fine-grained entity classification method according to claim 1, characterized in that: In step S4, the text representation of the sentence, the global visual representation of the image, the local object representation, and the cross-modal semantic representation are optimized in a unimodal form using a multi-head self-attention mechanism according to formula (6) to obtain the optimized unimodal representation: （6）； in: represents the optimized unimodal representation, represents the multi-head self-attention mechanism, represents the query matrix of the multi-head self-attention mechanism, represents the key matrix of the multi-head self-attention mechanism, represents the value matrix of the multi-head self-attention mechanism, Represents a splicing operation, Indicates The output of an attention head is represents the number of attention heads, Represents the linear transformation weight matrix.

6. The medical-oriented few-sample multimodal fine-grained entity classification method according to claim 1, characterized in that: In step S5, the optimized unimodal representations are respectively input into the hierarchical cross-modal semantic alignment network for multimodal feature fusion using the following method to obtain text cross-object representation, text cross-modal semantic representation, and text cross-visual representation. The text cross-object representation, text cross-modal semantic representation, and text cross-visual representation are concatenated to obtain a multimodal fusion representation: S51: The hierarchical cross-modal semantic alignment network uses the global visual representation in the optimized unimodal representation as the query matrix, and the text representation in the optimized unimodal representation as the key matrix and the value matrix to obtain the image-perceived text representation. The text representation in the optimized unimodal representation is used as the query matrix, and the image-perceived text representation is used as the key matrix and the value matrix to obtain the refined image-perceived text representation. The text representation in the optimized unimodal representation is used as the query matrix, and the global visual representation in the optimized unimodal representation is used as the key matrix and the value matrix to finally obtain the refined text-perceived image representation. The refined image-perceived text representation and the refined text-perceived image representation are then gated to obtain the text cross-visual representation. S52: The hierarchical cross-modal semantic alignment network uses the local object representation in the optimized unimodal representation as the query matrix, and the text representation in the optimized unimodal representation as the key matrix and the value matrix to obtain the local object-aware text representation, uses the text representation in the optimized unimodal representation as the query matrix, and the local object representation in the optimized unimodal representation as the key matrix and the value matrix to obtain the refined local object-aware text representation, and then uses the text representation in the optimized unimodal representation as the query matrix, and the local object representation in the optimized unimodal representation as the key matrix and the value matrix to finally obtain the refined text-aware local object representation, and then uses the refined local object-aware text representation and the refined text-aware local object representation through the gate control function to obtain the text cross-object representation; S53: The hierarchical cross-modal semantic alignment network uses the cross-modal semantic representation in the optimized unimodal representation as the query matrix, and the text representation in the optimized unimodal representation as the key matrix and the value matrix to obtain the cross-modal perceived text representation, uses the text representation in the optimized unimodal representation as the query matrix, and the cross-modal semantic representation in the optimized unimodal representation as the key matrix and the value matrix to obtain the refined cross-modal perceived text representation, and then uses the text representation in the optimized unimodal representation as the query matrix, and the cross-modal semantic representation in the optimized unimodal representation as the key matrix and the value matrix to finally obtain the refined text-aware cross-modal semantic representation, and then uses the refined cross-modal perceived text representation and the refined text-aware cross-modal semantic representation through the gate control function to obtain the text cross-modal semantic representation; S54: splicing text cross-object representation, text cross-modal semantic representation, and text cross-visual representation to obtain a multi-granular cross-modal representation of text perception; S55: Concatenate the text representation in the optimized unimodal representation with the text-aware multi-granularity cross-modal representation to obtain a multimodal fusion representation.

7. The medical-oriented few-sample multimodal fine-grained entity classification method according to claim 1, characterized in that: In step S6, the following method is used to input the multimodal fusion representation into the classifier to calculate the prediction probability and binary cross entropy loss of each fine-grained entity category: S61: The classifier calculates the predicted probability of each fine-grained entity category according to formula (7): （7）； in: Indicates the predicted vector length type vector, represents the linear transformation matrix, represents the text representation in the optimized unimodal representation, Representing text-aware multi-granular cross-modal representations, represents multimodal fusion representation, Indicates The predicted probability of fine-grained entity categories, represents the total number of fine-grained entity categories in the expanded training set, Represents the predicted probability of the last fine-grained entity category; S62: Calculate the binary cross entropy loss according to formula (8): （8）； in: represents the binary cross entropy loss, Indicates Sample category labels.

8. The medical-oriented few-sample multimodal fine-grained entity classification method according to claim 7, characterized in that: In step S7, the following method is used to perform iterative optimization training on the expanded training set using binary cross entropy loss: During the training process, the binary cross entropy loss is monitored in real time. If the binary cross entropy loss does not decrease after three consecutive iterations, the early stopping strategy is used to terminate the training. Otherwise, the training is terminated after the set number of iterations.

9. The medical-oriented few-sample multimodal fine-grained entity classification method according to claim 7, characterized in that: During the performance verification on the few-shot validation set and the final effect evaluation on the few-shot test set, if If the predicted probability of a fine-grained entity category is greater than 0.5, the fine-grained entity is classified as the type corresponding to the fine-grained entity category. If the predicted probabilities of multiple fine-grained entity categories are greater than 0.5, the fine-grained entities are classified into the types corresponding to these fine-grained entity categories. The fine-grained entity categories have multiple type labels. If the prediction probabilities of all fine-grained entity categories are less than or equal to 0.5, the type corresponding to the value of the fine-grained entity category with the highest prediction probability is selected as the final prediction type.

10. A medical-oriented few-sample multimodal fine-grained entity classification system, used to execute a medical-oriented few-sample multimodal fine-grained entity classification method as claimed in any one of claims 1 to 9, characterized in that: Including medical-based FIGER dataset, context-aware semantic difference module, text feature extraction model, hierarchical multi-granularity visual semantic pyramid module, single-modal representation optimization module, hierarchical cross-modal semantic alignment network and classifier; The medical-based FIGER dataset is used to provide a fine-grained entity classification dataset for a medical-oriented few-sample multimodal fine-grained entity classification system; The context-aware semantic difference module is used to expand text information for the text in the few-sample training set to obtain an expanded training set; The text feature extraction model is used to obtain text representation of sentences from each sentence in the expanded training set, the few-sample validation set and the few-sample test set; The hierarchical multi-granularity visual semantic pyramid module is used to integrate the images in each sample in the expanded training set, the few-shot validation set and the few-shot test set to obtain the global visual representation, local object representation and cross-modal semantic representation of the image, wherein the global visual encoder in the hierarchical multi-granularity visual semantic pyramid module is used to obtain the global visual representation of the image, the local visual encoder is used to obtain the local object representation, and the additional visual encoder is used to obtain the cross-modal semantic representation; The unimodal representation optimization module is used to optimize the text representation of the sentence, the global visual representation of the image, the local object representation and the cross-modal semantic representation in a unimodal form by applying a multi-head self-attention mechanism to obtain an optimized unimodal representation; The hierarchical cross-modal semantic alignment network is used to fuse the optimized single-modal representation with multi-modal features to obtain a multi-modal fusion representation; The classifier is used to calculate the prediction probability and binary cross entropy loss of each fine-grained entity category, and classify medical-oriented few-sample multimodal fine-grained entities.

Images