Multi-modal relation extraction method and system based on meta-learning

Abstract

The invention provides a multi-modal relation extraction method and system based on meta-learning, and belongs to the technical field of text entity recognition, and the method comprises the steps: obtaining a to-be-predicted text image pair; and processing the obtained text image pair by using a pre-trained relationship prediction model to obtain a multi-modal relationship between the entities. According to the method, a multi-modal relation extraction framework based on meta-learning is provided and is suitable for any relation classification model based on gradient descent; initial parameters are learned from a head relation class through meta learning, then model parameters in a tail item feature learning process are finely adjusted, and task learning from easy to difficult is realized by utilizing priori knowledge learned from head data; the feature representation of the tail class is enhanced, the feature gap between the head class and the tail class is bridged, and the relation identification performance is improved; during rare category training, the weight of the positive gradient is dynamically improved, the weight of the negative gradient is dynamically reduced, and the problem of tail data sparsity is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of text entity recognition, and in particular to a multimodal relationship extraction method and system based on meta-learning. Background Art

[0002] Multimodal relation extraction is a key task in the field of multimedia and knowledge graphs. It aims to detect and classify semantic relations between entities using text and visual cues, and can be widely used in downstream tasks such as visual question answering and reasoning. Although this research field has made some progress after years of research and development. However, this task is often plagued by the long-tail problem, that is, the training data is mainly concentrated on a few types of relations, lacking sufficient labels for the remaining types of relations. Due to the uneven distribution of training data categories, the model tends to predict the most frequent labels at the head and ignores the recognition of the tail relations.

[0003] Existing multimodal relation extraction methods mainly achieve relation understanding and recognition through effective alignment between image target object regions and text entities. For example, Zheng et al. attempted an effective graph alignment method to perform multimodal fusion based on the structural and semantic consistency between the visual scene graph and the grammatical dependency tree to extract fine-grained object and entity information. Chen et al. attempted to remove irrelevant image objects using a hierarchical multimodal fusion framework to reduce the noise interference of irrelevant images and achieve better multimodal fusion. Wang et al. used a retrieval method to collect relevant text from the entire Wikipedia to improve MNER and MRE performance. Subsequent work further retrieved images related to objects and text from NewsCLIPping to better enhance retrieval. Existing research mainly focuses on studying how to better achieve the fusion alignment of image and text feature representations and bridge the semantic gap between images and texts. The evaluation criteria are the overall Micro F1 performance of all classes. However, due to the long tail of the training data, the recognition accuracy of some classes at the tail (such as "per / per / relatives") is 0%. This shows that the model only improves the overall performance by fitting more head relations of the annotated data, while ignoring the recognition of tail relations. There is currently no research that specifically proposes a solution to the long-tail problem of this task.

[0004] In summary, existing multimodal relation extraction methods mainly rely on a large number of annotated samples for model optimization, to achieve effective alignment between image target object regions and text entities, or to further combine related images from external databases to help with relation recognition. Using the Micro F1 score to measure the average performance of all classes ignores the imbalance of dataset samples (i.e., long-tail distribution), which leads to the method tending to fit the head relation data with a large number of annotated samples, while ignoring the recognition accuracy of the tail relation. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal relationship extraction method and system based on meta-learning, so as to realize the extraction of relationships from image-text data pairs and solve the problem of sparsity of tail relationship data, so as to meet the needs of class imbalance in actual scenarios and solve at least one technical problem existing in the above-mentioned background technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a multimodal relationship extraction method based on meta-learning, comprising:

[0008] Get the text image pair to be predicted;

[0009] The acquired text-image pairs are processed by using a pre-trained relationship prediction model to obtain a multimodal relationship between entities; wherein, the training of the relationship prediction model includes: extracting modality-specific features of the input image and text data by using different encoders; extracting the position features of each object in the image based on the image features; performing a hierarchical interaction of multimodal information on the obtained visual features and text features, splicing the visual features and text features, and obtaining preliminary fusion features; performing a token-wise layer cross-modal interaction by calculating the similarity between the text token and the image feature, and obtaining the final fusion feature ; Input the fused features into the linear layer of the backbone network for feature learning; Calculate the cross entropy loss; Add up the losses of each sample to get the total loss value; Perform backpropagation from the final calculated loss value to automatically calculate the gradient values ​​of the parameters in the model; Use the gradient values ​​of the parameters to use the optimizer to update the learnable parameter values ​​of the model algorithm; Stop the model training process after the number of training rounds to obtain the model training parameters; Load the model training parameters, use the remaining tail class data for supervised fine-tuning training, replace the cross entropy loss with the gradient-guided reweighted loss, and finally obtain the model weight parameters that perform well in all classes.

[0010] Furthermore, the first layer of ViT is used as a visual encoder and the first layer of BERT is used as a text encoder to encode the image and text data respectively.

[0011] Furthermore, each relation Ri is regarded as a task, and multi-class classifiers are trained for all relations in a multi-task manner. The categories are arranged according to the frequency of occurrence, and all categories are learned from easy to difficult. For all relations R from the distribution p(R) i , optimize the following objects:

[0012]

[0013] in is the loss on relation Ri, f θ is a relation classification model, and eventually learns the best model parameters with good prediction performance for all relations

[0014] Furthermore, the first layer of ViT is used as a visual encoder to extract image features H v , and the first layer of BERT is used as a text encoder; using the token <o>Wrap the obtained object title Scap and mark it with <s>Mark the corresponding entities in the sentence and then stitch them together with [SEP]; the above sentence will be used as the input of the text encoder BERT to extract the text feature H T ;

[0015] W cap =ClipCap(o k )

[0016]

[0017] Among them k represents the object to be predicted, x i represents the i-th entity in the sentence, MHA, FFN and LN represent multi-head attention, feedforward network and Layernorm operator respectively.

[0018] Furthermore, we use the depth estimation model S2RDepthNet to obtain the depth map of the entire image. Then we obtain a separate depth image from the area corresponding to each object, denoted as V D , the RGB features and depth features of all objects are connected to generate the visual feature H V :

[0019]

[0020] Furthermore, a modality fusion encoder M-Encoder is used for hierarchical interaction of multimodal information; the encoding strategy of the lth layer is as follows:

[0021]

[0022] in and Representing text and visual feature representations respectively;

[0023] The PGI module redefines the text and visual and Self-attention calculation:

[0024]

[0025] In a second aspect, the present invention provides a multimodal relationship extraction system based on meta-learning, comprising:

[0026] An acquisition module, used to acquire a text image pair to be predicted;

[0027] The processing module is used to process the acquired text image pairs using a pre-trained relationship prediction model to obtain the multimodal relationship between entities; wherein, the training of the relationship prediction model includes: extracting modality-specific features of the input image and text data using different encoders; extracting the position features of each object in the image based on the image features; performing a hierarchical interaction of multimodal information on the obtained visual features and text features, splicing the visual features and text features to obtain preliminary fusion features; and performing a token-wise cross-modal interaction by calculating the similarity between the text token and the image feature to obtain the final Fuse features; input the fused features into the linear layer of the backbone network for feature learning; calculate the cross entropy loss; add up the losses of each sample to get the total loss value; perform backpropagation from the final calculated loss value to automatically calculate the gradient values ​​of the parameters in the model; use the gradient values ​​of the parameters to use the optimizer to update the learnable parameter values ​​of the model algorithm; stop the model training process after the number of training rounds to obtain the model training parameters; load the model training parameters, use the remaining tail class data for supervised fine-tuning training, replace the cross entropy loss with the gradient-guided reweighted loss, and finally obtain the model weight parameters that perform well in all classes.

[0028] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the multimodal relationship extraction method based on meta-learning as described in the first aspect is implemented.

[0029] In a fourth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the multimodal relationship extraction method based on meta-learning as described in the first aspect.

[0030] In a fifth aspect, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the multimodal relationship extraction method based on meta-learning as described in the first aspect.

[0031] Terminology explanation:

[0032] Multi-modal Relation Extraction: Integrates visual information to enrich textual relationship analysis, uses text and visual cues to detect and classify semantic relationships between entities, and for each potential relationship r i ∈R, assign confidence score p. Relation set R = {r1,…,r c ,None} include predefined relationship types, where "None" indicates the absence of a specific relationship.

[0033] Meta-learning: also known as "learning to learn", the core concept is to learn prior knowledge, emphasizing the process of improving learning algorithms in multiple learning stages. The basic learning stage solves tasks through internal learning algorithms, while the meta-learning stage updates the internal learning algorithm through external algorithms to improve its learning model.

[0034] Beneficial effects of the present invention: Compared with the existing methods, it fills the gap in the research of long-tail problems in the field of multimodal relationship extraction, pays more attention to the performance improvement of each class rather than the overall performance, and is more in line with the phenomenon of class imbalance in actual application scenarios. A multimodal relationship extraction framework based on meta-learning is proposed, which is applicable to any relation classification model based on gradient descent. The initial parameters are learned from the head relationship class through meta-learning, and then the model parameters in the tail feature learning process are fine-tuned. It utilizes the prior knowledge learned from the head data to achieve task learning from easy to difficult. Multimodal context learning is introduced to combine with a large language model (ChatGPT) to generate a language description of the tail class. This strategy provides additional fine-grained attribute descriptions for the tail class, further enhances the feature representation of the tail class, bridges the feature gap between the head class and the tail class, and improves the performance of relationship recognition. In the fine-tuning stage, a gradient-guided reweighted loss function is introduced. When training rare categories, the weight of the positive gradient is dynamically increased and the weight of the negative gradient is reduced to solve the problem of sparse tail data.

[0035] Additional advantages of the present invention will be more clearly given in the following description or learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0037] Figure 1 This is an overall framework diagram of the MetaMRE model described in an embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram of a template for a multimodal context prompt solution based on ChatGPT according to an embodiment of the present invention.

[0039] Figure 3 This is a flow chart of multimodal relationship extraction based on meta-learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below by the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0041] It should be understood by those skilled in the art that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0042] It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have a meaning consistent with that in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless as defined herein.

[0043] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention means the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or groups thereof.

[0044] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. Different embodiments or examples described in this specification and features of different embodiments or examples may be combined and combined by those skilled in the art without contradiction.

[0045] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0046] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0047] Relation extraction is one of the basic tasks of information extraction, and its purpose is to identify the relationship between entities in text, which is a key technical step in building knowledge graphs. With the enrichment of multimedia data resources such as network images and videos, many researchers have explored a series of methods for multimodal information extraction. These methods play a vital role in building multimodal knowledge graphs. Although such methods have improved performance and provided valuable insights into the field of fusion of image and text modality representations. However, existing methods rely on effective alignment between image target object regions and text entities, or further combine related images from external databases to enhance retrieval to help relationship identification. Generally, these methods require a large number of annotated samples for model optimization. However, in actual application scenarios, training data often exhibits an unbalanced distribution, resulting in a long tail phenomenon. This results in the model tending to fit most of the relationship data while ignoring the tail relationship. In order to solve the sparsity problem of tail relations, the present invention proposes a multimodal relationship extraction method and device (MetaMRE) based on meta-learning to extract relationship triplets from images and corresponding texts. Specifically, meta-learning is used to obtain initial model parameters from a rich set of head relations. Subsequently, ChatGPT’s multimodal context learning is used to enhance the feature representation of the tail class. Finally, the model parameters are fine-tuned based on these tail features. This method adopts a progressive learning process from simple (head class) to complex (tail class), which is closely integrated with the human cognitive process.

[0048] Example 1

[0049] In this embodiment 1, a multimodal relationship extraction system based on meta-learning is first provided, including: an acquisition module, used to acquire a text-image pair to be predicted. A processing module, used to process the acquired text-image pair using a pre-trained relationship prediction model to obtain a multimodal relationship between entities; wherein, training the relationship prediction model includes: extracting modality-specific features of the input image and text data using different encoders; extracting position features of each object in the image based on image features; performing a hierarchical interaction of multimodal information between the obtained visual features and text features, and splicing the visual features and text features to obtain preliminary fusion features; and performing cross-modal interaction of the token-wise layer by calculating the similarity between the text token and the image feature to obtain the final Fuse features; input the fused features into the linear layer of the backbone network for feature learning; calculate the cross entropy loss; add up the losses of each sample to get the total loss value; perform backpropagation from the final calculated loss value to automatically calculate the gradient values ​​of the parameters in the model; use the gradient values ​​of the parameters to use the optimizer to update the learnable parameter values ​​of the model algorithm; stop the model training process after the number of training rounds to obtain the model training parameters; load the model training parameters, use the remaining tail class data for supervised fine-tuning training, replace the cross entropy loss with the gradient-guided reweighted loss, and finally obtain the model weight parameters that perform well in all classes.

[0050] In this embodiment, the above-mentioned system is used to implement a multimodal relationship extraction method based on meta-learning, including: obtaining a text-image pair to be predicted; processing the obtained text-image pair using a pre-trained relationship prediction model to obtain a multimodal relationship between entities; wherein training the relationship prediction model includes: extracting modality-specific features of the input image and text data using different encoders; extracting position features of each object in the image based on image features; performing hierarchical interaction of multimodal information on the obtained visual features and text features, splicing the visual features and text features, and obtaining preliminary fusion features; and executing tok by calculating the similarity between the text token and the image feature. The cross-modal interaction of the en-wise layer is used to obtain the final fused features; the fused features are input into the linear layer of the backbone network for feature learning; the cross entropy loss is calculated; the loss of each sample is added up to obtain the total loss value; back propagation is performed from the final calculated loss value to automatically calculate the gradient values ​​of the parameters in the model; the gradient values ​​of the parameters are used to update the learnable parameter values ​​of the model algorithm using the optimizer; the model training process is stopped after the number of training rounds to obtain the model training parameters; the model training parameters are loaded, and the remaining tail class data is used for supervised fine-tuning training, and the cross entropy loss is replaced by the gradient-guided reweighted loss, and finally the model weight parameters that perform well in all classes are obtained.

[0051] In this case, each relation Ri is regarded as a task, and multi-class classifiers are trained for all relations in a multi-task manner. The categories are arranged according to the frequency of occurrence, and all categories are learned from easy to difficult. For all relations R from the distribution p(R), i , optimize the following objects:

[0052]

[0053] in is the loss on relation Ri, f θ is a relation classification model, and eventually learns the best model parameters with good prediction performance for all relations

[0054] Use the first layer of ViT as a visual encoder to extract image features H v , and the first layer of BERT is used as a text encoder; using the token <o>Wrap the obtained object title Scap and mark it with <s>Mark the corresponding entities in the sentence and then stitch them together with [SEP]; the above sentence will be used as the input of the text encoder BERT to extract the text feature H T ;

[0055] W cap =ClipCap(o k )

[0056]

[0057] Among them k represents the object to be predicted, x i represents the i-th entity in the sentence, MHA, FFN and LN represent multi-head attention, feedforward network and Layernorm operator respectively.

[0058] We use the depth estimation model S2RDepthNet to obtain the depth map of the entire image. Then we obtain a separate depth image from the area corresponding to each object, denoted as V D , the RGB features and depth features of all objects are connected to generate the visual feature H V :

[0059]

[0060] A modality fusion encoder M-Encoder is used for hierarchical interaction of multimodal information. The encoding strategy of the lth layer is as follows:

[0061]

[0062] in and Representing text and visual feature representations respectively;

[0063] The PGI module redefines the text and visual and Self-attention calculation:

[0064]

[0065] In order to model the importance of objects, the position features of each object are fused. k The position feature can be represented by a 5-dimensional vector:

[0066]

[0067] where w box 、h box and box They are o k The width, height and area of ​​​​them are normalized to the range of [0,1], and then the position features Fusion with the RGB-D visual features of the object.

[0068] Finally, a modality fusion encoder M-Encoder is used for the hierarchical interaction of multimodal information. The encoding strategy of the lth layer is as follows:

[0069]

[0070] in and Represents the feature representation of text and vision respectively. The PGI module redefines the and The self-attention calculation is as follows:

[0071]

[0072] The CAF module performs cross-modal interactions at the token-wise level. The module is defined as follows:

[0073] Sim=x t (x v ) T ,

[0074] Agg i (x v )=softmax(Sim i )x v ,(1≤i <n),

[0075] Agg(x v )=[Agg1(xv); ..., Agg n (x v )],

[0076] FFN(x t ) = ReLU(x t W1+b1+Agg(x v )W3)W2+b2.

[0077] Where [,] represents vector connection, and Sim represents the token similarity calculation between text and image. t and x v denote text and visual vectors respectively, and W* and b* are trainable parameters.

[0078] Finally, the relationship is predicted by fusing attribute-aware text encoding, depth-aware visual encoding, and position-fused multimodal encoding, as shown in the following formula:

[0079]

[0080] Example 2

[0081] like Figure 1 , Figure 3 As shown, in this embodiment 2, a meta-learning-based multimodal relationship extraction algorithm and device, MetaMRE, is provided, which mainly makes three improvements: the first is the introduction of a model-agnostic meta-learning framework; the second is the use of large model multimodal context to generate fine-grained language descriptions of tail classes; the third is a gradient-based reweighted loss function. The final proposed MetaMRE can achieve easy-to-difficult learning through a meta-learning strategy: first learn common relations (head data), and then based on prior knowledge of common classes, recognize rare tail categories to achieve impressive long-tail relationship extraction performance. The following is a specific explanation of the methods and modules designed and used in the present invention.

[0082] In the multimodal relation extraction task, the relationship between entities is predicted by given text and image pairs as input. The meta-learning-based multimodal relation extraction algorithm described in this embodiment is built on the most popular model-agnostic meta-learning (Reptile) algorithm. Reptile trains the meta-model on a set of source tasks so that the meta-model provides good parameter initialization for the target task. Intuitively, Reptile encourages the model to learn model parameters that can be adjusted quickly to achieve the best prediction performance in all tasks with as few gradient descent steps as possible. Therefore, the head relation data with rich annotated data is fully utilized to train the model to learn a good joint parameter initialization for the sparse tail relation. Then fine-tune it through limited supervision of the training instances of the tail relation.

[0083] In this embodiment, each relation Ri is regarded as a task, and a multi-class classifier is trained for all relations in a multi-task manner. The categories are arranged according to the frequency of occurrence, and all categories are learned from easy to difficult. For all relations R from the distribution p(R) i , this method directly optimizes the following objects:

[0084]

[0085] in is the loss on relation Ri, f θ is a relation classification model, and eventually learns the best model parameters with good prediction performance for all relations

[0086] The algorithm has two stages, first a meta-learning stage, then a fine-tuning learning stage, which fine-tunes the meta-learning parameters, both of which are performed on the relation classification model. In the meta-learning stage, the categories are arranged according to their frequency of occurrence, and the head relation categories are sampled from p(R) from easy to difficult in each iteration. Then, for each relation, a batch (70%) of supervised instances D are sampled from its training set as SupportSet, and the remaining instances are used as QureySet. Then, the adaptive model parameters on the relation are obtained by first calculating the gradient of the training loss on the sampled relation instance and back-propagating the gradient using a gradient-based optimization algorithm (SGD). At the end of the learning iteration, the adaptive parameters of each sampled relation in the batch are averaged, and the model parameters θ are updated:

[0087]

[0088] Among them, α is the meta-step size parameter.

[0089] In the fine-tuning learning phase, the model parameters are initialized using the parameters learned during meta-training. Then, standard supervised learning is continued to fine-tune the model parameters by performing multiple gradient descent steps using supervised examples of the tail relation categories.

[0090] In the feature extraction stage, following the current mainstream method, the first layer of ViT is used as a visual encoder to extract the image features H v , and the first layer of BERT is used as a text encoder. And the popular image caption model ClipCap is used to obtain attribute information about specific objects in the image. <o>Wrap the obtained object title Scap and mark it with <s>Mark the corresponding entities in the sentence and then stitch them together with [SEP]. The above sentence will be used as the input of the text encoder BERT to extract the text feature H T .

[0091] W cap =ClipCap(o k ),

[0092]

[0093] Among them k represents the object to be predicted, x i represents the i-th entity in the sentence, MHA, FFN and LN represent multi-head attention, feedforward network and Layernorm operator respectively.

[0094] Since the depth map contains the hierarchical information of the image objects, it is important for multi-object disambiguation. Therefore, the depth estimation model S2RDepthNet is used to obtain the depth map of the entire image. Then a separate depth image is obtained from the area corresponding to each object, denoted as V D , the RGB features and depth features of all objects are connected to generate the visual feature H V :

[0095]

[0096] In order to model the importance of objects, the position features of each object are fused. k The position feature can be represented by a 5-dimensional vector:

[0097]

[0098] where w box 、h box and box They are o k The width, height and area of ​​. We normalize them to the range [0,1] and then use the position feature Fusion with the RGB-D visual features of the object.

[0099] Finally, a modality fusion encoder M-Encoder is used for the hierarchical interaction of multimodal information. The encoding strategy of the lth layer is as follows:

[0100]

[0101] in and Represents the feature representation of text and vision respectively. The PGI module redefines the and The self-attention calculation is as follows:

[0102]

[0103] The CAF module performs cross-modal interactions at the token-wise level. The module is defined as follows:

[0104] Sim=x t (x v ) T ,

[0105] Agg i (x v )=softmax(Sim i )x v ,(1≤i <n),

[0106] Agg(x v )=[Agg1(x v ); ..., Agg n (x v )],

[0107] FFN(x t ) = ReLU(x t W1+b1+Agg(x v )W3)W2+b2. (10)

[0108] Where [*] represents vector connection, and Sim represents the token similarity calculation between text and image. t and x v denote text and visual vectors respectively, and W* and b* are trainable parameters.

[0109] Finally, the relationship is predicted by fusing attribute-aware text encoding, depth-aware visual encoding, and position-fused multimodal encoding, as shown in Formula 11:

[0110]

[0111] Since the text modality usually contains a lot of high-level and abstract information, and the text description can summarize the expert's prior knowledge, this may be useful when there are not enough samples to learn rare class representations for recognition. With the popularity of large models such as ChatGPT, some studies have shown that pre-trained LLMs can benefit from very few contextual demonstrations. In this embodiment, consider using multimodal context learning (also called "contextual learning") combined with ChatGPT's powerful text generation capabilities to generate fine-grained language descriptions for the tail classes, further enhancing the characteristics of the tail classes.

[0112] In this example, some manually designed contextual prompts are used to prompt ChatGPT to generate several language descriptions for the tail category. The specific contextual prompt template can be seen in Figure 2 In order to reduce the randomness and instability of the large model output, different instructions are made for each class, and these instructions are used to randomly generate K times for each tail class. The final performance is reported as the average of K times.

[0113] In this embodiment, in the meta-learning stage, the traditional cross entropy loss is used as the loss function of the task, see Formula 12:

[0114]

[0115] In the fine-tuning stage, following the Equalization Loss method, a gradient-based weighted loss function is introduced. For a given class c, other classes are considered to be negative classes of c. During training, if the sample belongs to class c, the weight accumulates a positive gradient. On the contrary, if the sample belongs to class j (j≠c), the weight corresponding to class c accumulates a negative gradient. If the number of samples of Classc is particularly large, the accumulated positive gradient will exceed the negative gradient. Conversely, if there are few classes c, the weight corresponding to class c will accumulate many more negative gradients than positive gradients, resulting in prediction errors. Since the labeled samples of the tail relationship class are scarce, the rare tail category weights mainly accumulate negative gradients during training, while the positive gradients are masked. In order to offset this, in this embodiment, Equalization Loss is introduced to artificially and dynamically limit the accumulation of negative gradients of rare categories, and only allow positive gradients to accumulate. The calculation formula is as follows:

[0116]

[0117] Where β is a random weight variable used to randomly control the gradient accumulation of negative samples.

[0118] In summary, the purpose of this embodiment is to fundamentally solve the long tail problem of multimodal relationship extraction, and a multimodal relationship extraction method and device based on meta-learning are designed to extract relationship triplets from images and corresponding texts. This method uses a meta-learning framework to learn from the data-rich distribution head class, obtain a good parameter initialization, and then fine-tune the relationship data at the tail. The prior knowledge learned from the head class is used to improve the performance of the data-poor tail class, and it is independent of the model. In addition, the powerful text generation ability of ChatGPT, which is now very popular, is combined with the advantages of multimodal context learning to provide additional fine-grained attribute descriptions for the tail class to bridge the feature gap between the head class and the tail class. This method adopts a progressive learning process from simple (head class) to complex (tail class), which is closely integrated with the human cognitive process, and mainly solves the problem of sparsity of tail relationship data to meet the needs of class imbalance in actual scenarios.

[0119] In this embodiment, the training process of the prediction model used includes the following specific steps:

[0120] Input to the algorithm:

[0121] 1. Image-text pair data and relationship labels: including training data (train), validation set (valid), and test set (test). Training images are used to train the model's ability to extract features, and validation and test sets are used to verify model performance.

[0122] 2. Model algorithm hyperparameters: including batch size, number of iterations and learning rate in training, meta-learning rate, weight coefficient β of the reweighted loss function, etc.

[0123] Output of the algorithm:

[0124] The parameter weights of the model algorithm trained to achieve the performance evaluation criteria.

[0125] Execution steps:

[0126] 1. Data preprocessing stage

[0127] Step 1-1: Load the image data and text data of the multimodal dataset into the GPU memory;

[0128] Step 1-2: Generate description text for image objects, using the popular image caption model ClipCap to obtain attribute information about specific objects in the image. Generate a corresponding text description Caption for each object in each image. Use the tag <o>Wrap the obtained object Caption and mark it with <s>Mark corresponding entities in the sentence, and then stitch them together using [SEP];

[0129] Step 1-3: Get the image depth information and use the depth estimation model S2RDepthNet to get the depth map of the entire image. Then we get a separate depth image from the area corresponding to each object.

[0130] 2: Re-ranking by class frequency

[0131] Step 2-1: According to the relationship category, group the data belonging to the same relationship category together, sort the categories according to the frequency of occurrence, regard each relationship Ri as a task, and train the model in a multi-task manner.

[0132] Step 2-2: Organize the data into batches according to the set batch size for each class, which will be used as input for the later model algorithm. Meta-training starts with the class with the highest frequency (head class).

[0133] 3. Feature extraction stage

[0134] Step 3-1: For the input image and text data, use different encoders to extract their modality-specific features. Use the first layer of ViT as the visual encoder and the first layer of BERT as the text encoder to encode the image and text data respectively.

[0135] Step 3-2: Based on the image features, the position features of each object in the image are extracted.

[0136] 4. Multimodal feature fusion stage

[0137] Step 4-1: The obtained visual features and text features are passed through a modal fusion encoder M-Encoder for hierarchical interaction of multimodal information;

[0138] Step 4-2: Concatenate the visual features and text features and enter the self-attention module to obtain preliminary fusion features;

[0139] Step 4-3: By calculating the similarity between text tokens and image features, perform cross-modal interaction at the token-wise layer to obtain the final fusion feature;

[0140] V. Meta-learning training phase

[0141] Step 5-1: Input the fused features into the linear layer of the backbone network for feature learning;

[0142] Step 5-2: Calculate the cross entropy loss;

[0143] Step 5-4: Add the loss of each sample to get the total loss value.

[0144] 6. Model Iteration Optimization Phase

[0145] Step 6-1: The code implementation is based on the PyTorch deep learning framework, which can perform backpropagation based on the final calculated loss value and automatically calculate the gradient value of the parameters in the model;

[0146] Step 6-2: Using the gradient calculated in the previous step, use the optimizer (Pytorch's Adam optimizer) to update the learnable parameter values ​​of the model algorithm;

[0147] Step 6-3: Repeat all the above execution steps until the model reaches the number of rounds set by the hyperparameter. Stop the model training process after reaching the number of training rounds, and finally obtain a good model training parameter θ0 on the data of the Top 10 head relationship classes.

[0148] 7. Fine-tuning learning phase

[0149] Step 7-1: Load the model weight θ0 trained in the meta-learning phase, and use the remaining tail class data for supervised fine-tuning training;

[0150] Step 7-2: Repeat steps 3 to 6 above, replacing the cross entropy loss with a gradient-guided reweighted loss Equalization Loss;

[0151] Step 7-3: Finally, we obtain the model weight parameter θ that performs well in all classes.

[0152] 8. Test and Evaluation Phase

[0153] Step 8-1: Read the images and texts of the test set, load them into GPU memory, and perform the same data processing as in the training phase;

[0154] Step 8-2: Use the commonly used evaluation indicators Precision, Recall, Micro F1 and Macro F1 for multimodal relationship extraction to evaluate the quality of the model by evaluating the calculated indicator values.

[0155] Example 3

[0156] This embodiment 3 provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the multimodal relationship extraction method based on meta-learning as described above is implemented. The method includes:

[0157] Get the text image pair to be predicted;

[0158] The acquired text-image pairs are processed by using a pre-trained relationship prediction model to obtain a multimodal relationship between entities; wherein, the training of the relationship prediction model includes: extracting modality-specific features of the input image and text data by using different encoders; extracting the position features of each object in the image based on the image features; performing a hierarchical interaction of multimodal information on the obtained visual features and text features, splicing the visual features and text features, and obtaining preliminary fusion features; performing a token-wise layer cross-modal interaction by calculating the similarity between the text token and the image feature, and obtaining the final fusion feature ; Input the fused features into the linear layer of the backbone network for feature learning; Calculate the cross entropy loss; Add up the losses of each sample to get the total loss value; Perform backpropagation from the final calculated loss value to automatically calculate the gradient values ​​of the parameters in the model; Use the gradient values ​​of the parameters to use the optimizer to update the learnable parameter values ​​of the model algorithm; Stop the model training process after the number of training rounds to obtain the model training parameters; Load the model training parameters, use the remaining tail class data for supervised fine-tuning training, replace the cross entropy loss with the gradient-guided reweighted loss, and finally obtain the model weight parameters that perform well in all classes.

[0159] Example 4

[0160] This embodiment 4 provides a computer device, including a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, the processor calls the program instructions to execute the above-mentioned multimodal relationship extraction method based on meta-learning, the method comprising:

[0161] Get the text image pair to be predicted;

[0162] The acquired text-image pairs are processed by using a pre-trained relationship prediction model to obtain a multimodal relationship between entities; wherein, the training of the relationship prediction model includes: extracting modality-specific features of the input image and text data by using different encoders; extracting the position features of each object in the image based on the image features; performing a hierarchical interaction of multimodal information on the obtained visual features and text features, splicing the visual features and text features, and obtaining preliminary fusion features; performing a token-wise layer cross-modal interaction by calculating the similarity between the text token and the image feature, and obtaining the final fusion feature ; Input the fused features into the linear layer of the backbone network for feature learning; Calculate the cross entropy loss; Add up the losses of each sample to get the total loss value; Perform backpropagation from the final calculated loss value to automatically calculate the gradient values ​​of the parameters in the model; Use the gradient values ​​of the parameters to use the optimizer to update the learnable parameter values ​​of the model algorithm; Stop the model training process after the number of training rounds to obtain the model training parameters; Load the model training parameters, use the remaining tail class data for supervised fine-tuning training, replace the cross entropy loss with the gradient-guided reweighted loss, and finally obtain the model weight parameters that perform well in all classes.

[0163] Example 5

[0164] This embodiment 5 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the multimodal relationship extraction method based on meta-learning as described above, the method including:

[0165] Get the text image pair to be predicted;

[0166] The acquired text-image pairs are processed by using a pre-trained relationship prediction model to obtain a multimodal relationship between entities; wherein, the training of the relationship prediction model includes: extracting modality-specific features of the input image and text data by using different encoders; extracting the position features of each object in the image based on the image features; performing a hierarchical interaction of multimodal information on the obtained visual features and text features, splicing the visual features and text features, and obtaining preliminary fusion features; performing a token-wise layer cross-modal interaction by calculating the similarity between the text token and the image feature, and obtaining the final fusion feature ; Input the fused features into the linear layer of the backbone network for feature learning; Calculate the cross entropy loss; Add up the losses of each sample to get the total loss value; Perform backpropagation from the final calculated loss value to automatically calculate the gradient values ​​of the parameters in the model; Use the gradient values ​​of the parameters to use the optimizer to update the learnable parameter values ​​of the model algorithm; Stop the model training process after the number of training rounds to obtain the model training parameters; Load the model training parameters, use the remaining tail class data for supervised fine-tuning training, replace the cross entropy loss with the gradient-guided reweighted loss, and finally obtain the model weight parameters that perform well in all classes.

[0167] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0168] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0171] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative work on the basis of the technical solution disclosed in the present invention should be included in the scope of protection of the present invention.< / s> < / o> < / s> < / o> < / s> < / o> < / s> < / o>

Claims

1. A multimodal relationship extraction method based on meta-learning, characterized in that: include: Get the text image pair to be predicted; The acquired text-image pairs are processed by using a pre-trained relationship prediction model to obtain a multimodal relationship between entities; wherein, the training of the relationship prediction model includes: extracting modality-specific features of the input image and text data by using different encoders; extracting the position features of each object in the image based on the image features; performing a hierarchical interaction of multimodal information on the obtained visual features and text features, splicing the visual features and text features, and obtaining preliminary fusion features; performing a token-wise layer cross-modal interaction by calculating the similarity between the text token and the image feature, and obtaining the final fusion feature ; Input the fused features into the linear layer of the backbone network for feature learning; Calculate the cross entropy loss; Add up the losses of each sample to get the total loss value; Perform backpropagation from the final calculated loss value to automatically calculate the gradient values ​​of the parameters in the model; Use the gradient values ​​of the parameters to use the optimizer to update the learnable parameter values ​​of the model algorithm; Stop the model training process after the number of training rounds to obtain the model training parameters; Load the model training parameters, use the remaining tail class data for supervised fine-tuning training, replace the cross entropy loss with the gradient-guided reweighted loss, and finally obtain the model weight parameters that perform well in all classes.

2. The multimodal relationship extraction method based on meta-learning according to claim 1, characterized in that: Use the first layer of ViT as a visual encoder and the first layer of BERT as a text encoder to encode images and text data respectively.

3. The multimodal relationship extraction method based on meta-learning according to claim 1, characterized in that: Treat each relation Ri as a task, train multi-class classifiers for all relations in a multi-task manner, arrange the categories according to their frequency of occurrence, and learn all categories from easy to difficult; for all relations R from distribution p(R) i , optimize the following objects: in is the loss on relation Ri, f θ is a relation classification model, and finally learns the optimal model parameter θ with good prediction performance for all relations ※ .

4. The multimodal relationship extraction method based on meta-learning according to claim 1, characterized in that: Use the first layer of ViT as a visual encoder to extract image features H v , and the first layer of BERT is used as a text encoder; using the token <o>Wrap the obtained object title Scap and mark it with <s>Mark the corresponding entities in the sentence and then stitch them together with [SEP]; the above sentence will be used as the input of the text encoder BERT to extract the text feature H T ;< / s> < / o> <s> IN cap =ClipCap(o k ) Among them k represents the object to be predicted, x i represents the i-th entity in the sentence, MHA, FFN and LN represent multi-head attention, feedforward network and Layernorm operator respectively.

5. The multimodal relationship extraction method based on meta-learning according to claim 1, characterized in that: We use the depth estimation model S2RDepthNet to obtain the depth map of the entire image. Then we obtain a separate depth image from the area corresponding to each object, denoted as V D , the RGB features and depth features of all objects are connected to generate the visual feature H V :

6. The multimodal relationship extraction method based on meta-learning according to claim 1, characterized in that: A modality fusion encoder M-Encoder is used for hierarchical interaction of multimodal information. The encoding strategy of the lth layer is as follows: in and Representing text and visual feature representations respectively; The PGI module redefines the text and visual and Self-attention calculation:

7. A multimodal relation extraction system based on meta-learning, characterized in that: include: An acquisition module, used to acquire a text image pair to be predicted; The processing module is used to process the acquired text image pairs using a pre-trained relationship prediction model to obtain the multimodal relationship between entities; wherein, the training of the relationship prediction model includes: extracting modality-specific features of the input image and text data using different encoders; extracting the position features of each object in the image based on the image features; performing a hierarchical interaction of multimodal information on the obtained visual features and text features, splicing the visual features and text features to obtain preliminary fusion features; and performing a token-wise cross-modal interaction by calculating the similarity between the text token and the image feature to obtain the final Fuse features; input the fused features into the linear layer of the backbone network for feature learning; calculate the cross entropy loss; add up the losses of each sample to get the total loss value; perform backpropagation from the final calculated loss value to automatically calculate the gradient values ​​of the parameters in the model; use the gradient values ​​of the parameters to use the optimizer to update the learnable parameter values ​​of the model algorithm; stop the model training process after the number of training rounds to obtain the model training parameters; load the model training parameters, use the remaining tail class data for supervised fine-tuning training, replace the cross entropy loss with the gradient-guided reweighted loss, and finally obtain the model weight parameters that perform well in all classes.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, the multimodal relationship extraction method based on meta-learning as described in any one of claims 1-6 is implemented.

9. A computer device, characterized in that: It includes a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the multimodal relationship extraction method based on meta-learning as described in any one of claims 1-6.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the multimodal relationship extraction method based on meta-learning as described in any one of claims 1-6. < / s>

Images