A multi-modal named entity recognition method based on multi-granularity feature enhancement

Abstract

The application discloses a kind of based on multi-granularity feature enhancement multi-modal named entity recognition method, first to input data carries out single modal feature extraction, then based on the result of single modal feature extraction carries out the multi-modal multi-granularity feature interaction of input data, finally based on the result of multi-granularity feature interaction carries out the label prediction of input data, completes entity recognition.The scheme of the present application defines a kind of multi-modal named entity recognition framework that can be used in multimedia post, wherein single modal feature extraction rate reduces the influence of the result of text and image irrelevant interference first, has the performance of processing informal and short text content, effectively solve the problem of sparse noisy data, multi-modal multi-granularity semantic interaction fully considers the correlation of text and picture, the necessity of picture feature local and whole, obtain the more comprehensive visual guidance required by text, to improve the accuracy of final label prediction, the present experiment has good experimental result on twitter data set, has certain popularization practical value.

Description

Technical Field

[0001] This invention belongs to the fields of machine learning and natural language processing, and specifically relates to a multimodal named entity recognition method based on multi-granularity feature enhancement. Background Technology

[0002] Social media has become a primary channel for people to share their daily lives and express their opinions, leading to a surge in postings. This combination of text and images makes user expression more accurate, engaging, and diverse, better reflecting user needs and emotions, while also providing a wealth of raw corpus data for Natural Language Processing (NLP). Named Entity Recognition (NER), as a prelude to many information extraction tasks, aims to discover multiple categories of named entities, such as Person (PER), Location (LOC), and Organization (ORG), from raw text data. Currently, text-based NER methods perform well on well-formatted texts, such as news reports, using CNNs, LSTMs, and Transformers to learn context and softmax and CRFs for decoding. However, social media texts often contain informal, colloquial expressions, even typos and spelling errors, and short texts contain less contextual information, resulting in data sparsity and affecting accuracy. Multimodal Named Entity Recognition (MNER) addresses this by focusing on visual features in relevant images, using visual information to enhance language features and achieve better results than NER. This is mainly achieved by encoding each word in the entire image or by aligning text vectors with visual object features.

[0003] Nevertheless, multimodal named entity recognition still heavily relies on textual content. Informal expressions and lack of context remain challenges that need to be overcome. Furthermore, the entire image may contain a large amount of irrelevant information, which can interfere with entity extraction. Therefore, we cannot focus solely on the visual features of the entire image while ignoring the semantic correspondence between objects in the image and words in the text. Thus, there is an urgent need for a multimodal named entity recognition method to address these issues and further improve the accuracy of multimodal named entity recognition in social media posts. Summary of the Invention

[0004] To address the aforementioned problems, the present invention aims to provide a multimodal named entity recognition method based on multi-granularity feature enhancement.

[0005] The specific technical solution for achieving the objective of this invention is as follows:

[0006] A multimodal named entity recognition method based on multi-granularity feature enhancement includes the following steps:

[0007] Step 1: Extract single-modal features from the input data;

[0008] Step 2: Perform multimodal, multi-granularity feature interaction on the input data based on the results of single-modal feature extraction;

[0009] Step 3: Based on the results of multi-granularity feature interaction, predict the label of the input data to complete entity recognition.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0011] (1) The solution of the present invention uses a well-trained text feature enhancement technology to obtain the contextual representation of each word input, effectively weakening the interference in the informal expression of the text and enhancing the weight of effective words in the text; the text features interact with multi-granular image features, and consider the correspondence between text and global features, text and local features, and text and image matching multiple times, find images that support the text features as a basis, reduce the influence of image-irrelevant noise, and further improve the accuracy of named entity recognition;

[0012] (2) In the multimodal interaction stage, the present invention constructs a multimodal transformer structure, and multi-level semantic interaction fully learns the most valuable word-aware multimodal representation of the image, providing the required visual cues for the text more accurately and comprehensively;

[0013] (3) This invention defines a framework for multimodal named entity recognition based on multi-granularity feature enhancement, which can be used for multimedia image and text posts. Among them, single-modal feature extraction first reduces the influence of irrelevant interference between text and images, and has the performance to handle informal and short text content, effectively solving the problem of sparse and noisy data. The multimodal multi-granularity semantic interaction of this scheme fully considers the correlation between text and images, the necessity of local and overall image features, and obtains more comprehensive visual guidance required by the text, thereby improving the accuracy of the final label prediction. This experiment has good experimental results on the Twitter dataset.

[0014] The present invention will be further described below with reference to specific embodiments. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall framework of the multimodal named entity recognition method based on multi-granularity feature enhancement of the present invention.

[0016] Figure 2 This is a schematic diagram of CLIP text sequence processing according to the present invention.

[0017] Figure 3 This is a schematic diagram of the multimodal transformer structure of the present invention. Detailed Implementation

[0018] Example

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0021] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0022] Combination Figure 1 A multimodal named entity recognition method based on multi-granularity feature enhancement includes the following steps:

[0023] Step 1: Extract single-modal features from the input data:

[0024] Step 1-1: Extract features from the text information in the input data to obtain the text feature representation T:

[0025] The feature extraction methods for the text information include BERT model, semantic enhancement, and CLIP model;

[0026] When using the BERT model for text feature extraction, the text data X = {x0,...,x} is... M The input is fed into the Encoder part of the BERT model for encoding, and the resulting feature vector is used as input for feature extraction to obtain the features of the input text data.

[0027] T i B =BERT(x i ;θ bert )

[0028]

[0029] When using semantic enhancement for text feature extraction, a pre-trained Word2Vec model is used to transform each input word x i Use the K most similar words {w} in the Embedding space i1 ,w i2 ,...,w iK} indicates that these similar words w ij Then, through the Word2Vec model, we obtain υ ij And assign a weight α to it. ij To weigh the effects of different words and obtain the features of the input text data:

[0030]

[0031] Semantic enhancement enables the effective extraction of contextual representations for each word from short texts and context-sparse content.

[0032] When using the CLIP model for text feature extraction, although the entire text can be obtained using a trained CLIP model, in order to extract features from individual word inputs in a consistent manner with the first two methods and obtain the same seq_length feature representation, it is necessary to process the inputs required by the three methods simultaneously during word segmentation. Therefore, the input text information needs to be segmented into several tokens first. The sequence of original word combination strings corresponding to each token with the same length is used as the text input of CLIP. Since the text information corresponding to this sequence is the same as that of the token sequence, the extracted features have the same dimension and can be directly linearly combined.

[0033] After encoding by the CLIP model, the feature extraction of a single word input is performed using encode_text, resulting in the CLIP representation of the text features, T. C ;

[0034] This method has excellent performance in feature extraction and matching of text and images.

[0035] Steps 1-2: Extract features from the image information in the input data to obtain image feature representations:

[0036] The image feature extraction methods include ResNet network, MASK RCNNL model and CLIP model;

[0037] ResNet: When using the ResNet network for image feature extraction, the image is first divided into visual blocks. The encapsulated ResNet network is used to obtain visual feature maps, and the feature vectors of the visual blocks at a certain scale are converted into the same dimension as the feature representation extracted from the text information to obtain the global visual representation G. This step can extract deeper and more meaningful features from the input image and use it to extract the global features of the entire image.

[0038] MASK R-CNNL: This method is widely used in tasks such as object classification and segmentation. When using the MASK R-CNNL network for image feature extraction, the trained MASK R-CNN is used to detect and classify objects in the image, obtain the predefined categories of objects, and then the feature representation of the detected objects is obtained by looking up the category embedding table, thus obtaining the local object representation L;

[0039] CLIP: This step is the same as ResNet, using the entire image as input. However, CLIP does not require dividing the image into visual feature blocks. When using the CLIP model for image feature extraction, the image is input into the CLIP model, and after image encoding, `encode_image` is called to extract image features corresponding to the model's dimension, for example, 1*1024, resulting in the CLIP representation of the image features, C. Figure 2 As shown.

[0040] Step 2: Perform multimodal, multi-granularity feature interaction on the input data based on the results of single-modal feature extraction.

[0041] Step 2-1: Multi-granularity feature interaction mainly involves cross-attention between text features and image features of different granularities. Text features are not considered for classification, so the text features represented in three different ways need to be naively connected to obtain the text feature representation T, which is then used to complete the subsequent interaction task.

[0042] To better learn word-aware multimodal representations from image features, a cross-modal transformer is used to facilitate interaction. First, a self-attention mechanism is used to obtain textual hidden representations `t`, global visual hidden representations `g`, local object hidden representations `l`, and CLIP image feature hidden representations `c`, which serve as inputs for cross-modal interaction. The cross-modal transformer layer integrates a multi-head attention mechanism and a prefix network, and constructs a structure as follows: Figure 3 The multi-modal transformer structure uses the result of the previous layer as Q to input the next layer, thereby completing multi-level semantic interaction.

[0043] Step 2-2: Perform multi-level semantic interaction based on cross-modal transformer to obtain word-aware multimodal representation S:

[0044] Step 2-2-1: Perform interaction between global visual representation G and text feature representation T: Based on the results obtained from the self-attention mechanism, using Q=t, K=V=g as input, complete simple semantic interaction processing between text and image to obtain word-aware image representation S. g1 ,Right now:

[0045] Q, K, V inputs transform as queries and key-value pairs to perform multi-head attention (MA):

[0046] q i =QW i q ,k i =KW i k ,v i =VW i v

[0047]

[0048] Among them, W i q W i k W i v ∈R d×(d / m) Here, d and m are the trainable parameters of the i-th attention head, respectively, and d and m are the number of attention heads and the representation dimension. The word-aware image representation S is obtained based on the inputs Q = t, K = V = g. g1 =z; then, using Q=g, K=V=t as input, perform simple semantic interaction processing of text and image again to obtain the image-aware word representation S. g2′ And serve as the input for the next layer, i.e., Q = t, K = V = S g2′ The complex word perception image representation S is obtained through an attention mechanism. g2 Finally, based on the coefficient u, the final semantic interaction representation between the global visual representation G and the text feature representation T is obtained: S g =u×S g1 +(1-u)S g2 ,

[0049] Step 2-2-2: Perform interaction between local object representation L and text feature representation T: Based on the results obtained from the self-attention mechanism, using Q=t, K=V=l as input, complete simple semantic interaction processing of text and image to obtain word-aware image representation S. l1Then, using Q=t, K=V=l as input, simple semantic interaction processing of text and images is performed again to obtain the image-aware word representation S. l2′ And used as the input to the next layer, i.e., Q = l, K = V = S l2′ The complex word perception image representation S is obtained. l2 Finally, the semantic interaction representation of the local object representation L and the text feature T is obtained through a gating function as S. l =u×S l1 +(1-u)S l2 ,in

[0050] Step 2-2-3: Perform interaction between image feature CLIP representation C and text feature representation T: Unlike before, CLIP features lack one dimension, so we first need to expand it with a dimension to correspond to the seq_length dimension. Therefore, we first expand the dimension of the image feature CLIP representation C. Based on the results obtained by the self-attention mechanism, using Q=t, K=V=c as input, we complete the simple semantic interaction processing between text and image to obtain the word-aware image representation S. c1 Then, using Q=c and K=V=t as input, simple semantic interaction processing of text and images is performed again to obtain the image-aware word representation S. c2′ And used as the input to the next layer, i.e., Q = l, K = V = S c2′ The complex word perception image representation S is obtained. c2 Finally, the final semantic interaction representation of image feature CLIP C and text feature T is obtained through a gating function, which is S. c =u×S c1 +(1-u)S c2 ,in

[0051] Step 2-2-4: Based on the results of steps 2-2-1 to 2-2-3, use a linear function to concatenate and obtain the word-aware multimodal representation of a single word input:

[0052] S = linear(S g ,S l ,S c )

[0053] Step 3: Based on the results of multi-granularity feature interaction, predict the label of the input data to complete entity recognition.

[0054] Step 3-1: Although multimodal interaction provides effective basis for word input from images and reduces interference from irrelevant information in the images, text features are still an important part. Therefore, in order to fully utilize these two parts, the text feature representation T and the word-aware multimodal representation S are connected using torch.cat to obtain the final multimodal representation P = {P}. i |P i ∈R 2d ,i=0,...,M;

[0055] Step 3-2: Construct a label prediction model based on the CRF layer and train the model using maximum likelihood estimation;

[0056] Step 3-3: Represent the final multimodal mode P = {P} i |P i ∈R 2d The probability of the CRF layer label prediction model (i = 0, ..., M) generating a predicted note sequence y after inputting the input:

[0057]

[0058] in, is a potential function, where Y is all possible label sequences;

[0059] Steps 3-4: Maximize the score according to the formula and decode to output the final label prediction result:

[0060]

[0061] The predicted label is the type of entity being identified, such as task, location, etc.; the work of named entity recognition is sequence labeling, which finds entity labels in a sequence.

[0062] This invention defines a framework for multimodal named entity recognition (MAR) of multimedia text and image posts based on a multi-granularity feature-enhanced method. The single-modal feature extraction method first reduces the impact of irrelevant interference from text and images, possessing performance in handling informal and short text content and effectively solving the problem of sparse and noisy data. The multimodal, multi-granularity semantic interaction of this scheme fully considers the relevance of text and images, and the necessity of local and overall image features, obtaining more comprehensive visual guidance required by the text, thereby improving the accuracy of the final label prediction. This experiment shows excellent results on the Twitter dataset.

[0063] This invention also provides a multimodal named entity recognition system based on multi-granularity feature enhancement, comprising the following modules:

[0064] Single-modal feature extraction module: used to extract single-modal features from input data;

[0065] Multimodal and multigranular feature interaction module: used for multimodal and multigranular feature interaction of input data based on the results of single-modal feature extraction;

[0066] Label prediction module: Used to predict the label of input data based on the results of multi-granularity feature interaction, thus completing entity recognition.

[0067] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the following steps:

[0068] Step 1: Extract single-modal features from the input data;

[0069] Step 2: Perform multimodal, multi-granularity feature interaction on the input data based on the results of single-modal feature extraction;

[0070] Step 3: Based on the results of multi-granularity feature interaction, predict the label of the input data to complete entity recognition.

[0071] A computer-storable medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0072] Step 1: Extract single-modal features from the input data;

[0073] Step 2: Perform multimodal, multi-granularity feature interaction on the input data based on the results of single-modal feature extraction;

[0074] Step 3: Based on the results of multi-granularity feature interaction, predict the label of the input data to complete entity recognition.

[0075] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multi-modal named entity recognition method based on multi-granularity feature enhancement, characterized in that, The method comprises the following steps: Step 1, single-modal feature extraction on input data; Step 2, multi-modal and multi-granularity feature interaction based on the result of single-modal feature extraction on input data: Step 2-1, obtaining text hidden representation t, global visual hidden representation g, local object hidden representation l and CLIP image feature hidden representation c through self-attention mechanism; Step 2-2, multi-level semantic interaction based on cross-modal transformer to obtain word-aware multi-modal representation S: Step 2-2-1, interaction between global visual representation G and text feature representation T: according to the result obtained by the self-attention mechanism, taking Q=t, K=V=g as input, completing simple semantic interaction processing between text and picture, obtaining word perception image representation S g1 That is: Q, K, V input transform as query and key-value pair to perform multi-head attention MA: q i = QW i q k i = KW i k v i = VW i v where W i q ,W i k ,W i v ∈R d×(d / m) is the trainable parameter of the ith head, d, m are the number of attention heads and representation dimensions, respectively, and the word-aware image representation S g1 is obtained according to the input Q = t, K = V = g; again, taking Q = g, K = V = t as input, the simple semantic interaction processing between text and picture is completed again, and the image-aware word representation S g2′ is obtained, which is input to the next layer, i.e. Q = t, K = V = S g2′ , and the complex word-aware image representation S g2 is obtained through the attention mechanism, and finally, based on the coefficient u, the final semantic interaction representation between the global visual representation G and the text feature representation T is obtained: S g = u x S g1 + (1 - u) S g2 , Step 2-2-2, interaction of local object representation L and text feature representation T: according to the result obtained by the self-attention mechanism, taking Q=t, K=V=l as input, simple semantic interaction processing of text and picture is completed to obtain word-aware image representation S l1 , and taking Q=t, K=V=l as input again, simple semantic interaction processing of text and picture is completed again to obtain image-aware word representation S l2′ , and as the input of the next layer, that is, Q=l, K=V=S l2′ , to obtain complex word-aware image representation S l2 , and finally through the gating function, the final semantic interaction representation of local object representation L and text feature T is S l = u x S l1 +(1-u)S l2 , wherein Step 2-2-3, the interaction of the picture feature CLIP representation C and the text feature representation T is performed: first, the picture feature CLIP representation C is dimensionally expanded, and a simple semantic interaction of the text and the picture is completed according to a result obtained according to a self-attention mechanism, to obtain a word-perception image representation S, with Q=t, K=V=c as input c1 , and then the simple semantic interaction of the text and the picture is again completed, to obtain an image-perception word representation S, with Q=c, K=V=t as input c2′ , and as input of the next layer, that is, Q=l, K=V=S c2′ , to obtain a complex word-perception image representation S c2 , and finally, a final semantic interaction representation of the picture feature CLIP representation C and the text feature T is obtained through a gating function as S c = u x S c1 + (1-u) S c2 , wherein Step 2-2-4, obtaining word-aware multi-modal representation of a single word based on the results of steps 2-2-1 to 2-2-3 by using linear function linkage: S = linear(S g ,S l ,S c ) Step 3, label prediction based on the result of multi-granularity feature interaction to complete entity recognition. 2.The multi-granularity feature enhancement based multi-modal named entity recognition method of claim 1, wherein, The single-modal feature extraction in the step 1 is specifically: Step 1-1, feature extraction on text information in input data to obtain text feature representation T; Step 1-2, feature extraction on image information in input data to obtain picture feature representation.

3. The multi-modal named entity recognition method based on multi-granularity feature enhancement according to claim 2, wherein the feature extraction mode of the text information in the step 1-1 comprises a BERT model, semantic enhancement and a CLIP model; wherein When the BERT model is used for text feature extraction, the text data X = {x0,..., x M} is input into the Encoder part of the BERT model for encoding, and a feature vector is obtained as input for feature extraction, and the features of the input text data are obtained: T i B = BERT(x i ; θ bert ) When text feature extraction is performed using semantic enhancement, each input word x i is represented by the K most similar words {w i1 , w i2 , ..., w iK} in the Embedding space, these similar words w ij are then passed through the Word2Vec model to obtain υ ij , and are assigned a weight α ij , to obtain the features of the input text data: When text feature extraction is performed using the CLIP model, first, the input text information is segmented into tokens, and a sequence of original word combination strings with the same length as each token is used as the text input of the CLIP. After encoding by the CLIP model, the encode_text is used for single word input feature extraction, and the text feature CLIP representation form T is obtained. C . 4.The multi-granularity feature enhancement based multi-modal named entity recognition method of claim 2, wherein, The picture feature extraction mode in the step 1-2 comprises a ResNet network, a MASK RCNNL model and a CLIP model; When the ResNet network is used for picture feature extraction, the picture is first divided into visual blocks, the encapsulated ResNet network is used to obtain a visual feature map, and the feature vector of a visual block with a certain scale is converted into the same dimension as the feature representation form of the text information extraction to obtain a global visual representation G; When the MASK RCNNL network is used for picture feature extraction, a trained MASK RCNN is used to detect objects in the picture and classify them to obtain object pre-defined categories, and then a category embedding table is used to obtain feature representation of the detected objects to obtain a local object representation L; When the CLIP model is used for picture feature extraction, the picture is input into the CLIP model, the image features of the corresponding model dimension are extracted through picture encoding and calling of encode_image to obtain picture feature CLIP representation C. 5.The multi-granularity feature enhancement based multi-modal named entity recognition method of claim 1, wherein, The label prediction in the step 3 is specifically: Step 3-1, connecting the text feature representation T and the word-aware multi-modal representation S to obtain a final multi-modal representation P; Step 3-2, constructing a label prediction model based on a CRF layer and training the model by using maximum likelihood estimation; Step 3-3, input the final multi-modal representation P = {P i |P i ∈R 2d , i = 0,..., M} into the trained CRF layer label prediction model to generate the probability of the predicted label sequence y: wherein, is a Potential function, Y is all possible sequences of labels; Step 3-4, decoding the final label prediction result according to the formula maximum score:

6. A multi-modal named entity recognition system based on multi-granularity feature enhancement, characterized in that, The method comprises the following modules: A single-modal feature extraction module for single-modal feature extraction on input data; A multi-modal and multi-granularity feature interaction module for multi-modal and multi-granularity feature interaction on input data based on the result of single-modal feature extraction, comprising: The text hidden representation t, the global visual hidden representation g, the local object hidden representation l, and the CLIP image feature hidden representation c are obtained through a self-attention mechanism; Based on the cross-modal transformer, multi-level semantic interaction is performed to obtain a word perception multi-modal representation S, which specifically includes: Interaction between global visual representation G and text feature representation T is performed: according to the result obtained according to the self-attention mechanism, taking Q=t, K=V=g as input, simple semantic interaction processing of text and picture is completed, and word perception image representation S is obtained g1 That is: The Q, K, and V inputs of the transformer are executed as a query and a key-value pair to perform multi-head attention MA: q i = QW i q ,k i = KW i k ,v i = VW i v where W i q ,W i k ,W i v ∈R d×(d / m) is the trainable parameter of the ith head, d, m are the number of attention heads and representation dimensions respectively, and the word-aware image representation S g1 is obtained according to the input Q = t, K = V = g; again taking Q = g, K = V = t as input, the simple semantic interaction processing between text and picture is completed again, and the image-aware word representation S g2′ is obtained, which is input to the next layer, i.e. Q = t, K = V = S g2′ , and the complex word-aware image representation S g2 is obtained through the attention mechanism, and finally the final semantic interaction representation between the global visual representation G and the text feature representation T is obtained based on the coefficient u: S g = u x S g1 + (1 - u) S g2 , Interaction between local object representation L and text feature representation T is performed: according to the result obtained from the self-attention mechanism, taking Q=t, K=V=l as input, simple semantic interaction processing of text and picture is completed, and word-aware image representation S is obtained l1 , and taking Q=t, K=V=l as input again, the simple semantic interaction processing of text and picture is completed again, and image-aware word representation S is obtained l2′ , and as the input of the next layer, that is, Q=l, K=V=S l2′ , to obtain complex word-aware image representation S l2 , and finally, the final semantic interaction representation of local object representation L and text feature T is obtained through a gating function as S l =u×S l1 +(1-u)S l2 , wherein The interaction of the picture feature CLIP representation C and the text feature representation T is performed: first, the picture feature CLIP representation C is dimensionally expanded, and a simple semantic interaction process of text and picture is completed according to a result obtained according to a self-attention mechanism, with Q=t, K=V=c as input, to obtain a word-perceived image representation S c1 Then, Q=c, K=V=t are taken as input, and the simple semantic interaction process of text and picture is completed again to obtain an image-perceived word representation S c2′ And taken as input of the next layer, that is, Q=l, K=V=S c2′ To obtain a complex word-perceived image representation S c2 Finally, a final semantic interaction representation of the picture feature CLIP representation C and the text feature T is obtained through a gating function as S c = u x S c1 +(1-u)S c2 Wherein Based on the above results, a linear function is used to link to obtain a word perception multi-modal representation of a single word input: S = linear(S g ,S l ,S c ) A label prediction module is configured to predict the label of the input data based on the result of the multi-granularity feature interaction, and complete entity recognition.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1-5.

8. A computer storable medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method in any one of claims 1-5.

Images