Social media field-oriented multi-modal named entity recognition system and method

Abstract

The invention discloses a social media field-oriented multi-modal named entity recognition system and method. The system comprises an acquisition module, an extraction module, a calculation module, a construction module and a recognition module, the acquisition module is used for acquiring a multi-modal named entity recognition data set and dividing the data set into a training set, a verification set and a test set according to preset, and the data set comprises text data and image data; the extraction module is used for performing feature extraction on the data set; the calculation module is used for calculating the semantic similarity of the text data and the image data and a position index associated with the text data and the image data according to the semantic similarity based on the extracted features; the construction module is used for constructing a multi-modal named entity recognition model based on the semantic similarity and the position index; and the identification module is used for completing the prediction of the named entity type by using the multi-modal named entity identification model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a multimodal named entity recognition system and method for the social media field. Background Art

[0002] Social media platforms have become the main channels for people to express emotions, exchange opinions and share insights in their daily lives. These platforms gather a large amount of user-generated unstructured content, including multimodal information such as text, pictures, and audio, forming a huge and rich social information network. The information contained in these social media platforms is diverse and complex, which makes it crucial to extract valuable information from them. In this context, we are faced with a new challenge: how to accurately and efficiently extract named entities from these massive and multimodal unstructured content to meet the new information requirements of tasks such as knowledge graph construction, personalized recommendations, and sentiment analysis.

[0003] Traditional named entity recognition is to identify and classify specific types of named entities from text, such as names of people, places, organizations, and other types. However, because traditional named entity recognition relies heavily on text information and ignores the rich contextual information contained in other modal data (such as images, voice, video, etc.), this limits the full utilization of rich information in multimodal data. Moreover, when processing longer texts, traditional named entity recognition methods may have limited understanding of the context and difficulty in capturing long-distance semantic relationships, resulting in decreased performance in complex contexts. In summary, traditional named entity recognition methods face a series of limitations and challenges when facing complex scenarios such as unstructured, multilingual, and multimodal.

[0004] Therefore, multimodal named entity recognition came into being on the basis of named entity recognition. In addition to using text information, this method also combines data from other modalities, such as images, voice, and video, to achieve more comprehensive and accurate named entity recognition. However, images often contain a lot of irrelevant information. Therefore, if only the global part of the image is considered, the model may focus on useless information, resulting in incorrect prediction results and failure to achieve good results. If the global features of the image are discarded and only the local areas in the image are considered, some information about the overall atmosphere may be lost. Therefore, the global features of the image and the features of each target detection area can be extracted separately, and the similarity between the image and the text and the associated position information can be combined to participate in entity recognition.

[0005] At present, deep learning methods are the mainstream methods for solving multimodal named entity recognition, and their performance exceeds that of other types of machine learning methods. In the methods of using deep learning for entity recognition, existing research methods usually use a more direct text and image feature splicing to solve the differences between cross-modal entity information, but this is prone to unclear information fusion between modalities, and the model cannot fully capture the semantic associations and complementary information between different modalities. Therefore, in the field of multimodal named entity recognition, how to make full use of the association information between text and images is an unresolved problem. Summary of the invention

[0006] In view of the problems existing in the prior art, the present invention aims to evaluate the similarity between text and image from a more fine-grained aspect by making full use of the association between image and text, and to filter out the effective data in the image and text for fusion, so as to improve the model performance.

[0007] To achieve the above-mentioned object, the present invention provides a multimodal named entity recognition system for the social media field, comprising: a collection module, an extraction module, a calculation module, a construction module and a recognition module;

[0008] The acquisition module is used to acquire a multimodal named entity recognition data set and divide the data set into a training set, a verification set and a test set according to a preset, wherein the data set includes text data and image data;

[0009] The extraction module is used to extract features from the data set;

[0010] The calculation module is used to calculate the semantic similarity between the text data and the image data and the position index associated with the text data and the image data according to the semantic similarity based on the extracted features;

[0011] The building module is used to build a multimodal named entity recognition model based on the semantic similarity and the position index;

[0012] The recognition module is used to complete the prediction of named entity types using the multimodal named entity recognition model.

[0013] Preferably, the extraction module includes: an image extraction unit and a text extraction unit;

[0014] The image extraction unit is used to obtain image features based on the image data;

[0015] The text extraction unit is used to obtain text features based on the text data.

[0016] Preferably, the image extraction unit comprises: a preprocessing device, a target detection device and an image feature extraction device;

[0017] The preprocessing device is used to preprocess the image data to obtain a processed image;

[0018] The target detection device is used to perform target detection on the processed image to obtain the target area;

[0019] The image extraction unit is used to obtain the image feature based on the processed image and the target area.

[0020] Preferably, in the image extraction unit,

[0021] The preprocessing device uses OpenCV to read the image data, then performs scaling operation according to the size of 224*224 pixels, and then normalizes the scaled image data according to its mean and variance to obtain a scaled and normalized image;

[0022] The normalization method is implemented by the following formula:

[0023]

[0024] Where x is the image pixel value; μ is the image pixel mean; σ is the image pixel variance;

[0025] The target detection device performs target detection on the processed data using the VinVL model, obtains all possible visual objects and detection frames in the processed image, and takes a number of target areas with high confidence;

[0026] The image extraction unit uses the image feature extractor Clip model to extract features from the processed image and the target area, and obtains the global features I = [I1, I2, ..., I n ] and the individual features i=[i1,i2,…,i n ].

[0027] Preferably, the text data is input into the text extraction unit to obtain the text features: T = [T1, T2, ..., T n ]; After the text data is split at word level, the features of each word are obtained: t = [t1, t2, ..., t n ].

[0028] Preferably, the calculation module calculates the cosine similarity of the text data and the image data to obtain the semantic similarity of the text data and the image data and the position index associated with the text data and the image data according to the semantic similarity;

[0029] Cosine similarity is calculated using the following formula:

[0030]

[0031] Among them, I represents image features; T represents text features; I·T represents the product of image vector and text vector; ||I|| and ||T|| are the moduli of image and text vectors respectively.

[0032] Preferably, the workflow of the building block includes:

[0033] S401. Use the Transformer-based sequence-to-sequence model BART to generate entity indexes and entity type indexes;

[0034] S402. Encode the text feature and the image feature representation through the encoder of the BART model:

[0035] H=[H T ;H I ;H i ]=Encoder([E T ; E I ; E i )

[0036] Among them, E T The three-layer embedding layer input represents the text part; E I The three-layer embedding layer input represents the original image; E i Represents the three-layer embedding input of the image corresponding to the target area; H T , H I and H i are the encoded vector representations of the above three respectively;

[0037] S403. At the tth time step, the model decoder represents the hidden layer H output by the encoder and the output index X before the tth time step ＜t Get the last hidden layer output h t :

[0038] h t =Decoder(H; X ＜t );

[0039] The probability distribution of the output index of the current time step is predicted based on the data obtained in S401-S403. The specific process includes:

[0040] L = TokenEmbed(Label)

[0041]

[0042] Among them, L represents the Token representation of the Label of the entity type; λ is a hyperparameter; H T represents the vector representation of the encoded text part; E T Token representation of the embedding layer of the text part; Represents the dot product operation;

[0043] S404. After completing the prediction of all time steps, calculate the cross entropy loss function to measure the difference between the true category probability distribution and the probability distribution predicted by the model;

[0044] S405. Back propagation updates the model parameters.

[0045] The present invention also provides a multimodal named entity recognition method for the social media field, which is applied to the above system and comprises the following steps:

[0046] Collect multimodal named entity recognition dataset;

[0047] Performing feature extraction on the data set;

[0048] Based on the extracted features, calculating the semantic similarity between the text data and the image data and the position index associated with the text data and the image data according to the semantic similarity;

[0049] Based on the semantic similarity and the position index, construct a multimodal named entity recognition model;

[0050] The multimodal named entity recognition model is used to complete the prediction of named entity types.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The present invention aims at the problem of insufficient semantic fusion caused by the differences between different modal features, and provides a multimodal named entity recognition method for the social media field. The present invention includes a unified Transformer model architecture with three layers of embedding, which can fuse images, texts and the association between images and texts. In addition to using image and text feature representation, the architecture also uses the position embedding and association degree of sentence-level text features with the original image global features and word-level entity features with the corresponding target area features as an embedding layer, so that the model can fully capture the interactive relationship between cross-modal features from a more fine-grained aspect. Finally, the above-constructed embedding layer is input into the pre-trained model BART for training, and four named entity types are predicted. The present invention improves the accuracy of entity recognition by fully integrating the association between images and texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 A schematic diagram of the system structure of an embodiment of the present invention;

[0055] Figure 2 is a flowchart of the working process of an embodiment of the present invention;

[0056] Figure 3 Schematic diagram of an encoder according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Embodiment 1

[0060] like Figure 1 As shown, it is a schematic diagram of the system structure of this embodiment, including: a collection module, an extraction module, a calculation module, a construction module and a recognition module;

[0061] The acquisition module is used to collect a multimodal named entity recognition dataset and divide the dataset into a training set, a validation set and a test set according to a preset, and the dataset includes text data and image data; the extraction module is used to extract features from the dataset; the calculation module is used to calculate the semantic similarity of the text data and the image data and the position index associated with the text data and the image data according to the semantic similarity based on the extracted features; the construction module is used to construct a multimodal named entity recognition model based on the semantic similarity and the position index; the recognition module is used to use the multimodal named entity recognition model to complete the prediction of the named entity type.

[0062] The following will combine this embodiment and Figure 2 , and explain in detail how the present invention solves technical problems in actual work.

[0063] First, the acquisition module is used to obtain a public multimodal named entity recognition dataset and divide the dataset into training set, validation set and test set in a ratio of 7:2:1. The dataset consists of social media posts, including text data and image data, where each post consists of a text content and an image.

[0064] Afterwards, the image features are obtained based on the image data using the image extraction unit. Before performing image feature extraction on the images in each data set and performing target detection on the original image, the image data in the data set needs to be preprocessed. The specific process includes:

[0065] The preprocessing device uses OpenCV to read the image data, then performs scaling operation according to the size of 224*224 pixels, and then normalizes the scaled image data according to its mean and variance to obtain a scaled and normalized image;

[0066] The normalization method is implemented by the following formula:

[0067]

[0068] Among them, x is the image pixel value; μ is the image pixel mean; σ is the image pixel variance.

[0069] Afterwards, the target detection device uses the VinVL model to perform target detection on the processed data, obtains all possible visual objects and detection frames in the processed image, and takes k target regions with high confidence according to the actual situation.

[0070] The image extraction unit uses the image feature extractor Clip model to extract features from the processed image and the target area, and obtains the global features I = [I1, I2, ..., I n ] and the individual features i=[i1,i2,…,i n ].

[0071] Input text data into the text extraction unit to obtain text features: T = [T1, T2, ..., T n ]; After splitting the text data at the word level, obtain the features of each word: t = [t1, t2, ..., t n ]. After the feature extraction is completed, the calculation module calculates the cosine similarity of the text data and the image data to obtain the semantic similarity of the text data and the image data and the position index associated with the two according to the semantic similarity.

[0072] Cosine similarity is calculated using the following formula:

[0073]

[0074] Among them, I represents image features; T represents text features; I·T represents the product of image vector and text vector; ||I|| and ||T|| are the moduli of image and text vectors respectively.

[0075] The multimodal named entity recognition model is constructed using the building blocks, and the multimodal named entity recognition model for the social media field is trained using the training set. For the constructed multimodal named entity recognition model, the extracted text features and image features are embedded into the input layer of the model in three layers: feature embedding, position embedding, and similarity embedding. The specific composition of the input layer is as follows:

[0076] E=E feature ×E similarity +E position

[0077] Among them, E feature The feature vectors representing the text sentence, global image and local image; E similarity Indicates the degree of association between the image and the text. For example, for the original image as a whole, it indicates the similarity with the entire text; for each target area, it indicates the similarity with the entity in the text; for the text part, it indicates the similarity with itself, which is usually 1; E position Press E similarity The corresponding positions with the highest correlation are embedded.

[0078] The model building process includes:

[0079] S401. Use the Transformer-based sequence-to-sequence model BART to generate entity index and entity type index;

[0080] S402. Encode the text features and the image features through the encoder of the BART model:

[0081] H=[H T ;H I ;H i ]=Encoder([E T ; E I ; E i ])

[0082] Among them, E T The three-layer embedding layer input represents the text part; E I The three-layer embedding layer input represents the original image; E i Represents the three-layer embedding input of the image corresponding to the target area; H T , H I and H i They are E T 、EI and E i The encoded vector representation; the encoder is as follows Figure 3 shown.

[0083] S403. At the tth time step, the model decoder represents the hidden layer H output by the encoder and the output index X before the tth time step ＜t Get the last hidden layer output h t :

[0084] h t =Decoder(H; X ＜t );

[0085] The probability distribution of the output index of the current time step is predicted based on the data obtained in S401-S403. The specific process includes:

[0086] L = TokenEmbed(Label)

[0087]

[0088] Among them, L represents the Token representation of the Label of the entity type; λ is a hyperparameter; H T represents the vector representation of the encoded text part; E T Token representation of the embedding layer of the text part; Represents the dot product operation;

[0089] S404. After completing the prediction of all time steps, calculate the cross entropy loss function to measure the difference between the true category probability distribution and the probability distribution predicted by the model;

[0090] S405. Back propagation updates the model parameters.

[0091] After the model is built, the trained multimodal named entity recognition model is verified and evaluated using the verification set, and then the training is continued by returning to S401-S405.

[0092] After multiple rounds of iterative training, save the model weight parameters with the highest accuracy in the validation set during training. Load the trained model weight parameters, perform entity recognition on the data in the test set, and predict the four entity types.

[0093] In this embodiment, the cross entropy loss function formula used is:

[0094]

[0095] Among them, y j represents the label of sample j, the positive sample label is 1, and the negative sample label is 0; p jRepresents the probability that sample j is predicted to be a positive sample.

[0096] Finally, the recognition module uses the constructed model to complete the prediction of the named entity type.

[0097] Embodiment 2

[0098] The present invention also provides a multimodal named entity recognition method for the social media field, comprising the following steps:

[0099] S1. Collect multimodal named entity recognition dataset.

[0100] Obtain a public multimodal named entity recognition dataset and divide it into training set, validation set, and test set in a ratio of 7:2:1. The dataset consists of social media posts, including text data and image data, where each post consists of a text content and an image.

[0101] S2. Perform feature extraction on the dataset.

[0102] Before performing image feature extraction on the images in each data set and performing target detection on the original images, the image data in the data set needs to be preprocessed. The specific steps include:

[0103] The image data is read using OpenCV, and then scaled according to a size of 224*224 pixels, and then the scaled image data is normalized according to its mean and variance to obtain a scaled and normalized image;

[0104] The normalization method is implemented by the following formula:

[0105]

[0106] Among them, x is the image pixel value; μ is the image pixel mean; σ is the image pixel variance.

[0107] The VinVL model is used to perform target detection on the processed data, obtain all possible visual objects and detection boxes in the processed image, and select k target regions with high confidence based on the actual situation.

[0108] The image feature extractor Clip model is used to extract features from the processed image and the target area, and the global features I = [I1, I2, ..., I n ] and the individual features i=[i1,i2,…,i n ].

[0109] The text data is input into the text feature extractor for feature extraction to obtain text features: T = [T1, T2, ..., Tn ]; After splitting the text data at the word level, obtain the features of each word: t = [t1, t2, ..., t n ].

[0110] S3. Based on the extracted features, the semantic similarity between the text data and the image data and the position index associated with the text data and the image data according to the semantic similarity are calculated.

[0111] After feature extraction is completed, the semantic similarity of the text data and the image data and the position index associated with the two according to the semantic similarity are obtained by calculating the cosine similarity of the text data and the image data.

[0112] Cosine similarity is calculated using the following formula:

[0113]

[0114] Among them, I represents image features; T represents text features; I·T represents the product of image vector and text vector; ||I|| and ||T|| are the moduli of image and text vectors respectively.

[0115] S4. Based on the semantic similarity and the position index, a multimodal named entity recognition model is constructed.

[0116] Construct a multimodal named entity recognition model, and use the training set to train the constructed multimodal named entity recognition model for the social media field. For the constructed multimodal named entity recognition model, the extracted text features and image features are embedded in three layers of feature embedding, position embedding, and similarity embedding to form the input layer of the model. The specific composition of the input layer is as follows:

[0117] E=E feature ×E similarity +E position

[0118] Among them, E feature The feature vectors representing the text sentence, global image and local image; E similarity Indicates the degree of association between the image and the text. For example, for the original image as a whole, it indicates the similarity with the entire text; for each target area, it indicates the similarity with the entity in the text; for the text part, it indicates the similarity with itself, which is usually 1; E position Press E similarity The corresponding positions with the highest correlation are embedded.

[0119] The model building process includes:

[0120] S401. Use the Transformer-based sequence-to-sequence model BART to generate entity index and entity type index;

[0121] S402. Encode the text feature and the image feature representation through the encoder of the BART model:

[0122] H=[H T ;H I ;H i ]=Encoder([E T ; E I ; E i ])

[0123] Among them, E T The three-layer embedding layer input represents the text part; E I The three-layer embedding layer input represents the original image; E i Represents the three-layer embedding input of the image corresponding to the target area; H T , H I and H i They are E T 、E I and E i The encoded vector representation;

[0124] S403. At the tth time step, the model decoder represents the hidden layer H output by the encoder and the output index X before the tth time step ＜t Get the last hidden layer output h t :

[0125] h t =Decoder(H; X ＜t );

[0126] The probability distribution of the output index of the current time step is predicted based on the data obtained in S401-S403. The specific process includes:

[0127] L = TokenEmbed(Label)

[0128]

[0129] Among them, L represents the Token representation of the Label of the entity type; λ is a hyperparameter; H T represents the vector representation of the encoded text part; E T Token representation of the embedding layer of the text part; Represents the dot product operation;

[0130] S404. After completing the prediction of all time steps, calculate the cross entropy loss function to measure the difference between the true category probability distribution and the probability distribution predicted by the model;

[0131] S405. Back propagation updates the model parameters.

[0132] After the model is built, the trained multimodal named entity recognition model is verified and evaluated using the verification set, and then the training is continued by returning to S401-S405.

[0133] After multiple rounds of iterative training, save the model weight parameters with the highest accuracy in the validation set during training. Load the trained model weight parameters, perform entity recognition on the data in the test set, and predict the four entity types.

[0134] In this embodiment, the cross entropy loss function formula used is:

[0135]

[0136] Among them, y j represents the label of sample j, the positive sample label is 1, and the negative sample label is 0; p j Represents the probability that sample j is predicted to be a positive sample.

[0137] S5. Use the multimodal named entity recognition model to complete the prediction of named entity types.

[0138] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multimodal named entity recognition system for the social media field, characterized in that: include: Acquisition module, extraction module, calculation module, construction module and recognition module; The acquisition module is used to acquire a multimodal named entity recognition data set and divide the data set into a training set, a verification set and a test set according to a preset, wherein the data set includes text data and image data; The extraction module is used to extract features from the data set; The calculation module is used to calculate the semantic similarity between the text data and the image data and the position index associated with the text data and the image data according to the semantic similarity based on the extracted features; The building module is used to build a multimodal named entity recognition model based on the semantic similarity and the position index; The recognition module is used to complete the prediction of named entity types using the multimodal named entity recognition model.

2. The multimodal named entity recognition system for social media according to claim 1, characterized in that: The extraction module includes: an image extraction unit and a text extraction unit; The image extraction unit is used to obtain image features based on the image data; The text extraction unit is used to obtain text features based on the text data.

3. The multimodal named entity recognition system for the social media field according to claim 2, characterized in that: The image extraction unit includes: a preprocessing device, a target detection device and an image feature extraction device; The preprocessing device is used to preprocess the image data to obtain a processed image; The target detection device is used to perform target detection on the processed image to obtain the target area; The image extraction unit is used to obtain the image feature based on the processed image and the target area.

4. The multimodal named entity recognition system for the social media field according to claim 3, characterized in that: In the image extraction unit, The preprocessing device uses OpenCV to read the image data, then performs scaling operation according to the size of 224*224 pixels, and then normalizes the scaled image data according to its mean and variance to obtain a scaled and normalized image; The normalization method is implemented by the following formula: Where x is the image pixel value; μ is the image pixel mean; σ is the image pixel variance; The target detection device performs target detection on the processed data using the VinVL model, obtains all possible visual objects and detection frames in the processed image, and takes a number of target areas with high confidence; The image extraction unit uses the image feature extractor Clip model to extract features from the processed image and the target area, and obtains the global features I = [I1, I2, ..., I n ] and the individual features i=[i1,i2,…,i n ].

5. The multimodal named entity recognition system for social media field according to claim 2, characterized in that: The text data is input into the text extraction unit to obtain the text features: T = [T1, T2, ..., T n ]; After the text data is split at word level, the features of each word are obtained: t = [t1, t2, ..., t n ].

6. The multimodal named entity recognition system for social media field according to claim 1, characterized in that: The calculation module calculates the cosine similarity of the text data and the image data to obtain the semantic similarity of the text data and the image data and the position index associated with the text data and the image data according to the semantic similarity; Cosine similarity is calculated using the following formula: Among them, I represents image features; T represents text features; I·T represents the product of image vector and text vector; ||I|| and ||T|| are the moduli of image and text vectors respectively.

7. The multimodal named entity recognition system for social media field according to claim 1, characterized in that: The workflow of the building block includes: S401. Use the Transformer-based sequence-to-sequence model BART to generate entity index and entity type index; S402. Encode the text feature and the image feature representation through the encoder of the BART model: H=[H T ;H I ;H i ]=Encoder([E T ;E I ;E i ]) Among them, E T The three-layer embedding layer input represents the text part; E I Represents the three-layer embedding layer input of the original image; E i Represents the three-layer embedding input of the image corresponding to the target area; H T , H I and H i They are E T 、E I and E i The encoded vector representation; S403. At the tth time step, the model decoder represents the hidden layer H output by the encoder and the output index X before the tth time step ＜t Get the last hidden layer output h t : h t =Decoder(H;X ＜t ); The probability distribution of the output index of the current time step is predicted based on the data obtained in S401-S403. The specific process includes: L = TokenEmbed (Lable) Among them, L represents the Token representation of the Label of the entity type; λ is a hyperparameter; H T represents the vector representation of the encoded text part; E T Token representation of the embedding layer of the text part; Represents the dot product operation; S404. After completing the prediction of all time steps, calculate the cross entropy loss function to measure the difference between the true category probability distribution and the probability distribution predicted by the model; S405. Back propagation updates the model parameters.

8. A multimodal named entity recognition method for the social media field, the method being applied to the system according to any one of claims 1 to 7, characterized in that the steps include: Collect multimodal named entity recognition dataset; Performing feature extraction on the data set; Based on the extracted features, calculating the semantic similarity between the text data and the image data and the position index associated with the text data and the image data according to the semantic similarity; Based on the semantic similarity and the position index, construct a multimodal named entity recognition model; The multimodal named entity recognition model is used to predict the named entity type.

Images