Multi-modal named entity recognition and positioning method in low-resource scene

Through the combination of instance learning and LLaMA model, the problem of difficult training of multimodal named entity recognition and positioning technology in low-resource scenarios is solved, and better results are achieved under low-resource conditions. Through semantic similarity screening and instance learning are The performance of multimodal named entity recognition and positioning under low resource conditions has been improved.

CN119005190AActive Publication Date: 2024-11-22CHINA ACADEMY OF ELECTRONICS AND INFORMATION TECHNOLOGY OF CHINA ELECTRONICS TECHNOLOGY GROUP CORPORATION +2
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411037062.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-22
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

In low-resource scenarios, existing multimodal named entity recognition and positioning technologies are difficult to make full use of data and cannot effectively train models, resulting in the failure to achieve better multimodal named entity recognition and positioning effects under low-resource conditions.

Method used

Using the instance learning method, filter the training set pairs with the most semantic similarity by calculating the semantic similarity of text-image pairs, build demonstration instances, and use the LLaMA model for instance learning, combine the attention mechanism to weight the text and visual features to generate joint features Indicates that naming entity recognition and image entity positioning are performed.

Benefits of technology

The data set is more efficiently utilized in low-resource scenarios, which improves the effect of multimodal named entity recognition and positioning, and obtains better recognition and positioning performance with fewer samples through pre-training knowledge.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005190A_ABST
    Figure CN119005190A_ABST
Patent Text Reader

Abstract

The invention provides a multi-modal named entity recognition and positioning method in a low-resource scene, and the method comprises the steps: screening image-text pairs with similar semantics through calculating the similarity, and using an LLaMA as a core structure in the low-resource scene, and more fully utilizing a model to pre-train knowledge through constructing a multi-modal instance. In the training stage, image-text pairs with similar semantics are screened by calculating similarity, instances are constructed to assist in training, and a named entity recognition loss function and an entity positioning loss function are calculated at the same time in the training process to assist in training; in a non-training stage, instance auxiliary reasoning is constructed through semantic similarity calculation, and the effect of multi-modal named entity recognition and positioning in a low-resource scene is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a multimodal named entity recognition and positioning method in a low-resource scenario. Background Art

[0002] In recent years, with the rapid development of artificial intelligence, the field of natural language processing has gradually emerged. Information extraction is a key task in natural language processing. The goal is to identify and extract specific types of information from unstructured or semi-structured data, that is, to convert scattered information into structured data. Information extraction mainly includes three subtasks: named entity recognition, relationship extraction, and event extraction.

[0003] Named entity recognition is an important direction in information extraction. Traditional named entity recognition tasks mainly deal with pure text data; while multimodal named entity recognition tasks are further extended to multimodal information including images and texts, which is a relatively new research direction.

[0004] With the development of technology, multimodal named entity recognition and localization tasks have gradually emerged. This task requires identifying named entities in text and locating areas related to entities in images. Its application scenarios are: On social media, the text and picture content posted by users often contains a lot of information. Through multimodal named entity recognition and localization technology, this information can be automatically extracted for user portraits and interest analysis; in intelligent question-answering systems, multimodal named entity recognition and localization technology can help the system better understand user queries. In addition to text information, the system can further understand user needs based on the information included in the pictures provided by users, thereby providing more accurate and detailed answers; in the medical field, by combining medical images and case descriptions, multimodal named entity recognition and localization technology can assist doctors in diagnosis and treatment, and improve the quality and efficiency of medical services.

[0005] In the study of multimodal named entity recognition and localization, data from both text and image modalities are combined to provide richer information. However, the multimodality of this data also brings many difficulties: since text and images themselves are quite different, it is very difficult to fuse the information of the two modalities. Therefore, it is necessary to design an effective mechanism to capture the semantic relationship between images and texts. In addition, it is also necessary to develop a model that can learn cross-modal associations and establish effective connections between images and texts.

[0006] Some recent technical methods have been able to solve the above difficulties better. However, through the study of existing multimodal named entity recognition and localization technologies, it was found that a large amount of text data and corresponding images, and a large amount of annotated data are required in the training process to complete the task. As a result, the current technology cannot train the model well in low-resource scenarios and cannot fully utilize pre-trained knowledge in low-resource scenarios. Summary of the invention

[0007] The technical problem to be solved by the present invention is how to construct a multimodal named entity recognition and positioning model that can make more full use of data in a low-resource scenario; in view of this, the present invention provides a multimodal named entity recognition and positioning method in a low-resource scenario.

[0008] The technical solution adopted by the present invention is a multimodal named entity recognition and positioning method in a low-resource scenario, comprising:

[0009] Step 1: Each text-image pair (T i , I i ) in the text T i Word segmentation, using the pre-trained language model encoding, each text is represented as a vector form v T ; Encode the image using the visual feature extractor, represented as a vector v I ;

[0010] Step 2: Calculate the text semantic similarity and image semantic similarity between each text-image pair in the test set and each text-image pair in the training set, combine them into a comprehensive semantic similarity, and form a similarity matrix;

[0011] Step 3: According to the semantic similarity matrix, select the text-image pairs (T i , I i ) the most semantically similar training set text-image pair (T j , I j );

[0012] Step 4: (T i , I i ) and (T j , I j ) are combined as a set of data, where (T i , I i ) as the content to be inferred, (T j , I j ) is used to construct a demonstration example;

[0013] Step 5: (T j , I j ), for the text T j, according to the predefined entity types, a natural language template filling method is used to fill in the text T j Create an example and get the text Use the text encoder to convert T j and Encoding as text vector and

[0014] For image I j , use the visual feature extractor to extract visual features from each specific area containing the entity, and obtain the visual feature vector of the entity And extract the features of the entire image to obtain the visual feature vector of the entire image

[0015] Finally, we get the image-text pair represented by the vector and Together they constitute a demonstrative example;

[0016] Step 6: (T i , I i ), change the text T i Tokenize and encode into text vectors using a text encoder Using visual feature extractor from I i Extract visual features from the image and obtain the visual feature vector To get the image-text pair represented by the vector

[0017] Step 7: Convert all image feature vectors to the same dimension as the text vector through a linear projection layer;

[0018] Step 8: Concatenate the converted image feature vector and text vector as the input vector;

[0019] Step nine: input the input vector into the LLaMA model for instance learning, and perform weighted fusion of the extracted text features and visual features through attention mechanism learning to generate a joint feature representation;

[0020] Step 10: Decode the joint feature representation using the LLaMA decoder to obtain the image-text pair (T i , I i )’s prediction output;

[0021] The method further includes a process of training the model, including:

[0022] Step 11: According to the predicted output and the actual information of the test set, determine the named entity recognition loss function and the image entity localization loss function based on the cross entropy loss and IoU respectively;

[0023] Step 12: construct an overall loss function based on the named entity recognition loss function and the image entity positioning loss function;

[0024] Step 13: Use the overall loss function and the optimizer to update the parameters.

[0025] In one embodiment, in step 2, the semantic similarity is calculated as follows:

[0026]

[0027]

[0028]

[0029] in, and Represents the vector form of text, and Represents the vector form of the image; Indicates the semantic similarity of texts. Represents the semantic similarity of images, s ij represents the comprehensive semantic similarity; α and β are parameters for controlling weights; the similarity matrix composed of semantic similarity is denoted as S.

[0030] In one embodiment, in step five, the predefined entity types include Person, Organization, Location, and Other.

[0031] In one embodiment, in step seven, the linear projection is calculated as follows:

[0032]

[0033]

[0034]

[0035] Among them, Linear(x) is the linear projection layer; and is the image feature of the original dimension; and It is the projected image feature vector, which has the same dimension as the text vector.

[0036] In one embodiment, in step eight, the sequence of the input vectors is constructed as follows:

[0037]

[0038] in, Indicates the instance content. Indicates the training content.

[0039] In one embodiment, in step 11, the named entity recognition loss function and the image entity localization loss function are calculated as follows:

[0040] Named entity recognition loss function:

[0041]

[0042] Among them, p i,c represents the probability that word i belongs to category c; if word i belongs to category c, then y i,c is 1, otherwise y i,c is 0; N represents the total number of words, and C represents the total number of categories.

[0043] Image entity localization loss function:

[0044]

[0045] Where M represents the number of entities; IoU represents the intersection over union ratio, that is, the ratio of the intersection area and the union area of ​​the predicted area and the true area; ε is a positive constant.

[0046] In one embodiment, in step twelve, the overall loss function is constructed as follows:

[0047] L=L NER +λ·L BOX

[0048] Among them, λ is the parameter controlling the weight; L NER is the named entity recognition loss function, L BOX is the image entity positioning loss function, and L is the final loss function.

[0049] Another aspect of the present invention also provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the multimodal named entity recognition and positioning method in a low-resource scenario as described in any one of the above items.

[0050] Another aspect of the present invention further provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multimodal named entity recognition and positioning method in a low-resource scenario as described in any of the above items are implemented.

[0051] Compared with the prior art, the present invention has at least the following advantages:

[0052] The embodiment of the present application provides a method for multimodal named entity recognition and localization in a low-resource scenario based on instance learning, introduces instance learning, constructs new training methods and reasoning methods, and utilizes data sets more efficiently in low-resource scenarios; by using the LLaMA language model, pre-trained knowledge is better utilized for instance learning, the utilization rate of data is improved when there are fewer samples, and better multimodal named entity recognition and localization effects are obtained under low-resource conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A logical schematic diagram of a multimodal named entity recognition and positioning method in a low-resource scenario according to an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of a flow chart of a multimodal named entity recognition and positioning method in a low-resource scenario according to an embodiment of the present invention;

[0055] Figure 3 FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose, the present invention is described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0057] Unless otherwise defined, all terms (including technical terms and scientific terms) used in this article have the same meaning as commonly understood by ordinary technicians in the field to which this application belongs. It should also be understood that terms (such as terms defined in commonly used dictionaries) should be interpreted as having the same meaning as their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined in this article.

[0058] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0059] For ease of understanding, the abbreviations and key terms appearing in this embodiment are defined as follows:

[0060] SBERT: Sentence-BERT is a pre-trained model suitable for calculating text similarity;

[0061] ResNet: Residual Neural Network, a convolutional neural network;

[0062] LLaMA: Large Language Model MetaAI, an autoregressive pre-training model;

[0063] Linear: linear layer;

[0064] Adam: Adaptive Moment Estimation is an optimizer:

[0065] IoU: Intersection over Union, the area of ​​overlapping regions divided by the area of ​​the union of the two regions;

[0066] BP: Back Propagation, a common method for training.

[0067] An embodiment of the present invention provides a multimodal named entity recognition and positioning method in a low-resource scenario, comprising:

[0068] Step 1: Each text-image pair (T i , I i ) in the text T i Word segmentation, using the pre-trained language model encoding, each text is represented as a vector form v T ; Encode the image using the visual feature extractor, represented as a vector v I ;

[0069] Step 2: Calculate the text semantic similarity and image semantic similarity between each text-image pair in the test set and each text-image pair in the training set, combine them into a comprehensive semantic similarity, and form a similarity matrix;

[0070] Step 3: According to the semantic similarity matrix, select the text-image pairs (T i , I i ) the most semantically similar training set text-image pair (T j , I j );

[0071] Step 4: (T i , I i ) and (T j , I j ) are combined as a set of data, where (T i , I i ) as the content to be inferred, (T j , I j ) is used to construct a demonstration example;

[0072] Step 5: (T j , I j ), for the text T j , according to the predefined entity types, a natural language template filling method is used to fill in the text T jCreate an example and get the text Use the text encoder to convert T j and Encoding as text vector and

[0073] For image I j , use the visual feature extractor to extract visual features from each specific area containing the entity, and obtain the visual feature vector of the entity And extract the features of the entire image to obtain the visual feature vector of the entire image

[0074] Finally, we get the image-text pair represented by the vector and Together they constitute a demonstrative example;

[0075] Step 6: (T i , I i ), change the text T i Tokenize and encode into text vectors using a text encoder Using visual feature extractor from I i Extract visual features from the image and obtain the visual feature vector To get the image-text pair represented by the vector

[0076] Step 7: Convert all image feature vectors to the same dimension as the text vector through a linear projection layer;

[0077] Step 8: Concatenate the converted image feature vector and text vector as the input vector;

[0078] Step nine: input the input vector into the LLaMA model for instance learning, and perform weighted fusion of the extracted text features and visual features through attention mechanism learning to generate a joint feature representation;

[0079] Step 10: Decode the joint feature representation using the LLaMA decoder to obtain the image-text pair (T i , I i )’s prediction output;

[0080] The method further includes a process of training the model, including:

[0081] Step 11: According to the predicted output and the actual information of the test set, determine the named entity recognition loss function and the image entity localization loss function based on the cross entropy loss and IoU respectively;

[0082] Step 12: construct an overall loss function based on the named entity recognition loss function and the image entity positioning loss function;

[0083] Step 13: Use the overall loss function and the optimizer to update the parameters.

[0084] refer to Figure 1 as well as Figure 2 , the method provided in this embodiment will be described in detail step by step below.

[0085] Specifically, when using the model for reasoning, the method includes:

[0086] Step 1: Each text-image pair (T i , I i ) in the text T i Word segmentation, using the pre-trained language model encoding, each text is represented as a vector form v T ; Encode the image using the visual feature extractor, represented as a vector v I ;

[0087] In this embodiment, the pre-trained language model uses SBERT and the visual feature extractor uses ResNet;

[0088] Step 2: Calculate the text semantic similarity and image semantic similarity between each text-image pair in the test set and each text-image pair in the training set, combine them into a comprehensive semantic similarity, and form a similarity matrix;

[0089] Furthermore, in step 2, the semantic similarity is calculated as follows:

[0090]

[0091]

[0092]

[0093] in, and Represents the vector form of text, and Represents the vector form of the image; Indicates the semantic similarity of texts. Indicates the semantic similarity of images; s ij represents the comprehensive semantic similarity, α and β are parameters that control the weight; the similarity matrix composed of semantic similarity is denoted as S.

[0094] In this embodiment, α is set to 0.7 and β is set to 0.3.

[0095] Step 3: According to the semantic similarity matrix, select the text-image pairs (T i , I i ) the most semantically similar training set text-image pair (T j , I j );

[0096] Furthermore, in step 3, the calculation of selecting a text-image pair is as follows:

[0097]

[0098] in,

[0099] Step 4: (T i , I i ) and (T j , I j ) are combined as a set of data, where (T i , I i ) as the content to be inferred, (T j , I j ) is used to construct a demonstration example;

[0100] Step 5: (T j , I j ), for the text T j , according to the predefined entity types, a natural language template filling method is used to fill in the text T j Create an example and get the text Use the text encoder to convert T j and Encoding as text vector and

[0101] For image I j , use the visual feature extractor to extract visual features from each specific area containing the entity, and obtain the visual feature vector of the entity And extract the features of the entire image to obtain the visual feature vector of the entire image

[0102] Finally, we get the image-text pair represented by the vector and Together they constitute a demonstrative example;

[0103] Furthermore, in step five, the predefined entity types include {Person, Organization, Location, Other}.

[0104] Step 6: For training content (T i , I i), the text T i Word segmentation and encoding into text vectors And use visual feature extractor from I i Extract image features and obtain image feature vector Finally, we get the image-text pair represented by the vector

[0105] In this embodiment, the visual feature extractor uses ResNet.

[0106] Step 7: Convert the image feature vector to the same dimension as the text vector through a linear projection layer;

[0107] Furthermore, in step 7, the linear projection is calculated as follows:

[0108]

[0109]

[0110]

[0111] Among them, Linear(x) is the linear projection layer; and is the image feature of the original dimension; and It is the projected image feature vector, which has the same dimension as the text vector.

[0112] Step 8: Concatenate the converted image feature vector and text vector as the input vector of the model;

[0113] Furthermore, in step eight, the input sequence is constructed as follows:

[0114]

[0115] in, Represents the instance content; Indicates the training content.

[0116] Step 9: Input the constructed input vector into the LLaMA model for instance learning. The extracted text features and visual features are weightedly fused through attention mechanism learning to generate a joint feature representation.

[0117] Step 10: Decode the joint feature representation using the LLaMA decoder to obtain the image-text pair (T i , I i )’s prediction output.

[0118] Furthermore, in step 10, the output is as follows:

[0119] Output = [E, T, L]

[0120] Where E is the entity, T is the entity type, and L is the bounding box coordinates of the entity in the image.

[0121] The method provided in this application further includes a training process for the model, including:

[0122] Step 11: Calculate the named entity recognition loss function and the image entity localization loss function respectively according to the output obtained by decoding and the actual information of the training set;

[0123] Furthermore, in step 11, the two loss functions are calculated as follows:

[0124] (1) Named Entity Recognition Loss Function:

[0125]

[0126] Among them, p i,c represents the probability that word i belongs to category c; if word i belongs to category c, then y i,c is 1, otherwise y i,c is 0; N represents the total number of words; C represents the total number of categories.

[0127] (2) Image entity localization loss function:

[0128]

[0129] Where M represents the number of entities; IoU represents the intersection over union ratio, that is, the ratio of the intersection area and the union area of ​​the predicted area and the true area; ε is a positive constant.

[0130] Step 12: Combine the two loss functions to construct the overall loss function;

[0131] L=L NER +λ·L BoX

[0132] Among them, λ is the parameter that controls the weight, L NER is the named entity recognition loss function, L BOX is the image entity positioning loss function, and L is the final loss function.

[0133] Step 13: During the BP process, use the optimizer to update the parameters.

[0134] In this embodiment, the optimizer selects the Adam optimizer, the learning rate is set to 3e-5, and the training rounds are set to 30.

[0135] In the second embodiment of the present invention, when performing multimodal named entity recognition and positioning in a non-training situation, step 11, step 12, and step 13 are removed.

[0136] Can refer to again Figure 1 First, the image-text pairs in the training set are encoded, where the text is encoded using SBERT and the image is encoded using ResNet. The text semantic similarity and image semantic similarity between each pair of image-text pairs are calculated respectively, and the comprehensive semantic similarity is obtained by weighted summation, and a semantic similarity matrix is ​​constructed; according to the semantic similarity matrix, another image-text pair with the highest similarity is found in the training set for the image-text pair to be trained or inferred, which is used to construct the instance; then, the text and image instances are constructed by natural language template filling and specific entity region encoding, and the multimodal instances are combined and added to the training or inference process; the integrated vector is input into the LLaMA model for instance learning. During the training process, a loss function that comprehensively considers named entity recognition and entity localization is constructed, and the Adam optimizer is used to update the weights during the back-propagation process; during the non-training process, instances are selected in the training set according to the comprehensive semantic similarity to assist in reasoning, and the final multimodal named entity recognition and localization results are obtained according to the output of the LLaMA decoder.

[0137] Embodiment 1 of the present invention uses a multimodal named entity recognition and positioning method in low-resource scenarios based on instance learning. This method selects semantically similar image-text pairs by calculating similarity. In low-resource scenarios, LLaMA is used as the core structure, and multimodal instances are constructed to make more effective use of model pre-training knowledge. In the training phase, similarity is calculated to select semantically similar image-text pairs, and instance-assisted training is constructed. During the training process, the named entity recognition loss function and the entity positioning loss function are calculated to assist training. In the non-training phase, instance-assisted reasoning is constructed through semantic similarity calculation to improve the effect of multimodal named entity recognition and positioning in low-resource scenarios.

[0138] A second embodiment of the present invention is an electronic device, such as Figure 3 As shown, it can be understood as a physical device, including a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the following operations are performed:

[0139] Step 1: Each text-image pair (T i , I i ) in the text T i Word segmentation, using the pre-trained language model encoding, each text is represented as a vector form v T ; Encode the image using the visual feature extractor, represented as a vector v I ;

[0140] Step 2: Calculate the text semantic similarity and image semantic similarity between each text-image pair in the test set and each text-image pair in the training set, combine them into a comprehensive semantic similarity, and form a similarity matrix;

[0141] Step 3: According to the semantic similarity matrix, select the text-image pairs (T i , I i ) the most semantically similar training set text-image pair (T j , I j );

[0142] Step 4: (T i , I i ) and (T j , I j ) are combined as a set of data, where (T i , I i ) as the content to be inferred, (T j , I j ) is used to construct a demonstration example;

[0143] Step 5: (T j , I j ), for the text T j , according to the predefined entity types, a natural language template filling method is used to fill in the text T j Create an example and get the text Use the text encoder to convert T j and Encoding as text vector Know

[0144] For image I j , use the visual feature extractor to extract visual features from each specific area containing the entity, and obtain the visual feature vector of the entity And extract the features of the entire image to obtain the visual feature vector of the entire image

[0145] Finally, we get the image-text pair represented by the vector and Together they constitute a demonstrative example;

[0146] Step 6: (T i , I i ), change the text T i Tokenize and encode into text vectors using a text encoder Using visual feature extractor from I i Extract visual features from the image and obtain the visual feature vector To get the image-text pair represented by the vector

[0147] Step 7: Convert all image feature vectors to the same dimension as the text vector through a linear projection layer;

[0148] Step 8: Concatenate the converted image feature vector and text vector as the input vector;

[0149] Step nine: input the input vector into the LLaMA model for instance learning, and perform weighted fusion of the extracted text features and visual features through attention mechanism learning to generate a joint feature representation;

[0150] Step 10: Decode the joint feature representation using the LLaMA decoder to obtain the image-text pair (T i , I i )’s prediction output;

[0151] The method further includes a process of training the model, including:

[0152] Step 11: According to the predicted output and the actual information of the test set, determine the named entity recognition loss function and the image entity localization loss function based on the cross entropy loss and IoU respectively;

[0153] Step 12: construct an overall loss function based on the named entity recognition loss function and the image entity positioning loss function;

[0154] Step 13: Use the overall loss function and the optimizer to update the parameters.

[0155] The third embodiment of the present invention, the process of the multimodal named entity recognition and positioning method in a low-resource scenario of this embodiment is the same as that of the first and second embodiments, the difference is that, in engineering implementation, this embodiment can be implemented by means of software plus a necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the method of the present invention can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including a number of instructions for a device to execute the method described in the embodiment of the present invention.

[0156] Through the description of the specific implementation methods, a deeper and more specific understanding of the technical means and effects adopted by the present invention to achieve the predetermined purpose should be obtained. However, the accompanying drawings are only for reference and illustration purposes and are not intended to limit the present invention.

Claims

1. A multimodal named entity recognition and localization method in a low-resource scenario, characterized in that: When using the model for inference, this includes: Step 1: Each text-image pair (T i ,I i ) in the text T i Word segmentation, using the pre-trained language model encoding, each text is represented as a vector form v T ; Encode the image using the visual feature extractor, represented as a vector v I ; Step 2: Calculate the text semantic similarity and image semantic similarity between each text-image pair in the test set and each text-image pair in the training set, combine them into a comprehensive semantic similarity, and form a similarity matrix; Step 3: According to the semantic similarity matrix, select the text-image pairs (T i ,I i ) the most semantically similar training set text-image pair (T j ,I j ); Step 4: (T i ,I i ) and (T j ,I j ) are combined as a set of data, where (T i ,I i ) as the content to be inferred, (T j ,I j ) is used to construct a demonstration example; Step 5: (T j ,I j ), for the text T j , according to the predefined entity types, a natural language template filling method is used to fill in the text T j Create an example and get the text Use the text encoder to convert T j and Encoding as text vector and For image I j , use the visual feature extractor to extract visual features from each specific area containing the entity, and obtain the visual feature vector of the entity And extract the features of the entire image to obtain the visual feature vector of the entire image Finally, we get the image-text pair represented by the vector and Together they constitute a demonstrative example; Step 6: (T i ,I i ), change the text T i Tokenize and encode into text vectors using a text encoder Using visual feature extractor from I i Extract visual features from the image and obtain the visual feature vector To get the image-text pair represented by the vector Step 7: Convert all image feature vectors to the same dimension as the text vector through a linear projection layer; Step 8: Concatenate the converted image feature vector and text vector as the input vector; Step nine: input the input vector into the LLaMA model for instance learning, and perform weighted fusion of the extracted text features and visual features through attention mechanism learning to generate a joint feature representation; Step 10: Decode the joint feature representation using the LLaMA decoder to obtain the image-text pair (T i ,I i )’s prediction output; The method further includes a process of training the model, including: Step 11: According to the predicted output and the actual information of the test set, determine the named entity recognition loss function and the image entity localization loss function based on the cross entropy loss and IoU respectively; Step 12: construct an overall loss function based on the named entity recognition loss function and the image entity positioning loss function; Step 13: Use the overall loss function and the optimizer to update the parameters.

2. The multimodal named entity recognition and positioning method in a low-resource scenario as claimed in claim 1, characterized in that: In the step 2, the semantic similarity is calculated as follows: in, and Represents the vector form of text, and Represents the vector form of the image; Indicates the semantic similarity of texts. Represents the semantic similarity of images, s ij represents the comprehensive semantic similarity; α and β are parameters for controlling weights; the similarity matrix composed of semantic similarity is denoted as S.

3. The multimodal named entity recognition and positioning method in a low-resource scenario as claimed in claim 1, characterized in that: In the step 5, the predefined entity types include Person, Organization, Location, and Other.

4. The multimodal named entity recognition and positioning method in a low-resource scenario as claimed in claim 1, characterized in that: In step 7, the linear projection is calculated as follows: Among them, Linear(x) is the linear projection layer; and is the image feature of the original dimension; and It is the projected image feature vector, which has the same dimension as the text vector.

5. The multimodal named entity recognition and positioning method in a low-resource scenario as claimed in claim 1, characterized in that: In step eight, the sequence of the input vector is constructed as follows: in, Indicates the instance content. Indicates the training content.

6. The multimodal named entity recognition and positioning method in a low-resource scenario as claimed in claim 1, characterized in that: In the step 11, the named entity recognition loss function and the image entity localization loss function are calculated as follows: Named entity recognition loss function: Among them, p i,c represents the probability that word i belongs to category c; if word i belongs to category c, then y i,c is 1, otherwise y i,c is 0; N represents the total number of words, and C represents the total number of categories; Image entity localization loss function: Where M represents the number of entities; IoU represents the intersection over union ratio, that is, the ratio of the intersection area and the union area of ​​the predicted area and the true area; ε is a positive constant.

7. The multimodal named entity recognition and positioning method in a low-resource scenario as claimed in claim 1, characterized in that: In step 12, the overall loss function is constructed as follows: L=L NER +λ·L BOX Among them, λ is the parameter controlling the weight; L NER is the named entity recognition loss function, L BOX is the image entity positioning loss function, and L is the final loss function.

8. An electronic device, characterized in that: The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the multimodal named entity recognition and positioning method in a low-resource scenario as described in any one of claims 1 to 7 are implemented.

9. A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the multimodal named entity recognition and positioning method in a low-resource scenario as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Template-based multi-modal named entity recognition method and related equipment

    CN114548067A

  • Multi-modal scene graph knowledge enhanced adversarial multi-modal pre-training method

    CN115331075A

  • Ancient language named entity recognition method based on knowledge embedding

    CN117272998A

  • Knowledge graph construction method based on man-machine collaborative information extraction labeling tool

    CN117521792A

  • Relationship extraction method based on multi-modal large language model

    CN117972121A