Named entity recognition system and method based on multi-modal fusion and deep learning

Abstract

The invention provides a named entity recognition system and method based on multi-modal fusion and deep learning, and relates to the technical field of natural language processing and financial named entity recognition. The system comprises a named entity corpus construction module, a multi-modal feature extraction module, a named entity corpus construction module, a multi-modal feature extraction module and a data processing module, wherein the named entity corpus construction module is used for generating text paragraphs by performing data cleaning and segmentation on obtained original financial text data and constructing a financial corpus; the multi-modal feature extraction module is used for performing multi-modal feature extraction on the text paragraphs in the financial corpus; the multi-modal feature fusion module is used for fusing the multi-modal features, the feature coding module is used for coding the multi-modal fusion features, the entity sequence labeling module is used for carrying out entity sequence labeling on the multi-modal fusion feature sequence of each text paragraph, and finally the entity sequence of each text paragraph is generated. According to the method, the accuracy and precision of named entity recognition on the financial text can be effectively improved by fusing the character, phonetic symbol and font multi-modal features.

Description

Technical Field

[0001] The present invention relates to the technical fields of natural language processing and financial named entity recognition, and in particular, to a named entity recognition system and method based on multimodal fusion and deep learning. Background Art

[0002] The financial text named entity recognition technology refers to processing a large amount of unstructured financial text through the powerful computing ability of a neural network, identifying financial entities with specific meanings from the text, and classifying the entities into pre-set categories. The entities in common financial fields mainly include: time, location, person name, organization name, company name, product name, and position. Early named entity recognition technologies mainly relied on manually written rules and pattern matching. These rules were usually based on domain-specific knowledge and text formats and could quickly identify some common entity types. However, this method has obvious limitations in the financial field because the complexity and diversity of financial texts make it difficult for the rules to cover all situations. For named entity recognition in the financial field, professional financial corpus is usually required as support for training. However, due to the lack of high-quality annotated corpus in the financial field, there are many professional nouns, resulting in difficult word segmentation, and the annotation work is complex and costly. The rise of deep learning technology has brought new breakthroughs to named entity recognition. Recurrent neural networks and their variants can effectively capture long-distance dependencies in text. In addition, the Transformer architecture and pre-trained language models perform well in the financial text named entity recognition task, significantly improving the accuracy and efficiency of recognition. However, pre-trained language models require a large amount of domain-specific data for adjustment and optimization. At the same time, although pre-trained language models can learn rich context representations, some features in financial texts, such as glyphs and phonetic notations, are still not fully utilized, resulting in the dilemma that financial entities are difficult to accurately identify.

[0003] In summary, in the field of financial named entity recognition, existing research usually lacks high-quality annotated financial corpus resources, resulting in a lack of data support for text processing. At the same time, due to the existence of a large number of proper nouns in the financial field, accurate word segmentation of the corpus cannot be performed, making it impossible to fully utilize the text information. Summary of the Invention

[0004] Aiming at the deficiencies of the above-mentioned existing technologies, the present invention proposes a financial text named entity recognition system and method based on multimodal feature fusion by integrating character, phonetic notation, and glyph multimodal features and a specific neural network structure, aiming to improve the accuracy and precision of financial entity recognition and provide an effective method for financial entity recognition.

[0005] A named entity recognition system based on multimodal fusion and deep learning is proposed in the first aspect of the present invention. The system includes: a named entity corpus construction module, a multimodal feature extraction module, a multimodal feature fusion module, a feature encoding module, and an entity sequence annotation module;

[0006] The named entity corpus construction module is used to obtain the original financial text data, generate text paragraphs and construct a financial corpus by cleaning and segmenting the obtained original financial text data, and then transmit the financial corpus to the multimodal feature extraction module;

[0007] The multimodal feature extraction module is used to perform multimodal feature extraction on each text paragraph in the financial corpus, generate character feature vectors, international phonetic alphabet feature vectors, and Wubi feature vectors for each text paragraph, and transmit them to the multimodal feature fusion module;

[0008] The multimodal feature fusion module is used to fuse the character feature vectors, international phonetic alphabet feature vectors, and Wubi feature vectors of each text paragraph by using a multimodal feature attention fusion mechanism, generate multimodal fusion features for each text paragraph, and transmit them to the feature encoding module;

[0009] The feature encoding module is used to perform feature encoding on the multimodal fusion features of each text paragraph, generate multimodal fusion feature sequences for each text paragraph, and transmit them to the entity sequence annotation module;

[0010] The entity sequence annotation module is used to perform entity sequence annotation on the multimodal fusion feature sequences of each text paragraph, generate entity sequences for each text paragraph in the financial corpus.

[0011] A named entity recognition method based on multimodal fusion and deep learning is proposed in the second aspect of the present invention. It is implemented by using a named entity recognition system based on multimodal fusion and deep learning. The method includes the following processes:

[0012] Obtain the original financial text data and perform data cleaning, segmentation, and named entity recognition to generate annotated text paragraphs and construct a financial corpus;

[0013] Use a multimodal feature extraction network model to perform multimodal feature extraction on the annotated text paragraphs in the financial corpus to generate character feature vectors, international phonetic alphabet feature vectors, and Wubi feature vectors;

[0014] Use a multimodal feature attention fusion model to fuse the character feature vectors, international phonetic alphabet feature vectors, and Wubi feature vectors to generate multimodal fusion features;

[0015] Input the multimodal fusion features into a feature encoding network model for feature encoding to generate multimodal fusion feature sequences;

[0016] Use an entity recognition annotation decoder to perform entity sequence annotation on the multi-modal fusion feature sequence, generate the entity sequence of each annotated text paragraph in the financial corpus, and calculate the total loss function value of each annotated text paragraph;

[0017] Based on the obtained total loss function value, train the multi-modal feature extraction network model, multi-modal feature attention fusion model, feature encoding network model, and entity recognition annotation decoder to obtain the trained multi-modal feature extraction network model, multi-modal feature attention fusion model, feature encoding network model, and entity recognition annotation decoder;

[0018] Obtain any financial text data, perform data cleaning and segmentation to generate text paragraphs of the financial text data, and then sequentially input the generated text paragraphs into the trained multi-modal feature extraction network model, multi-modal feature attention fusion model, feature encoding network model, and entity recognition annotation decoder for data processing, so as to obtain the entity sequence of the financial text data.

[0019] Further, the specific content of obtaining the original financial text data, performing data cleaning, segmentation, and named entity recognition is: obtaining the existing original financial text data, performing data cleaning on the original financial text data, and then dividing the cleaned original financial text data into text paragraphs by identifying the sentence boundaries in the cleaned original financial text data;

[0020] Set financial entity categories and annotation criteria, and according to the financial entity categories and annotation criteria, use the BIO annotation format to perform entity annotation on each divided text paragraph to obtain the annotated text paragraph and construct a financial corpus;

[0021] The financial entity categories include: person names, place names, time, institution names, company names, and product names;

[0022] The process of constructing the financial corpus is as follows: for any divided text paragraph, perform word segmentation on the text paragraph, and perform annotation on each split word or character according to the financial entity categories and annotation criteria to obtain the true label of the text paragraph; for the same text paragraph independently annotated by multiple annotators, use kappa consistency to test the true label of the text paragraph. If the test result is not less than the set threshold, generate the annotated text paragraph according to the true label, and use all the annotated text paragraphs to construct the financial corpus; if the test result is less than the set threshold, re-perform data cleaning, segmentation, and entity annotation on the original financial text data, and then perform kappa consistency test on the newly obtained true label until the test result reaches the set threshold;

[0023] Divide the labeled text paragraphs in the financial corpus into a training set, a validation set, and a test set according to a set ratio.

[0024] Furthermore, the specific content of using the multi-modal feature extraction network model to extract multi-modal features from the labeled text paragraphs in the financial corpus is as follows:

[0025] Use a pre-trained language model to extract semantic information from all labeled text paragraphs and generate character feature representation vectors for each labeled text paragraph;

[0026] Based on the financial corpus, perform character extraction on all labeled text paragraphs and establish a dictionary mapping according to the extracted characters;

[0027] For any labeled text paragraph, based on the dictionary mapping, use the International Phonetic Alphabet table to convert the character feature representation vector of this text paragraph into an International Phonetic Alphabet feature representation vector; use a Wubi code conversion tool to convert the character feature representation vector of this text paragraph into a Wubi feature representation vector;

[0028] Input the character feature representation vector, International Phonetic Alphabet feature representation vector, and Wubi feature representation vector of each labeled text paragraph into the multi-modal feature extraction network model respectively to obtain the character feature vector, International Phonetic Alphabet feature vector, and Wubi feature vector of each labeled text paragraph.

[0029] Furthermore, the multi-modal feature extraction network model includes: one-hot encoding, an embedding layer, a first convolutional neural network, a first linear neural network, a normalization layer, and a normalization layer;

[0030] The one-hot encoding is used to perform one-hot encoding conversion on the input International Phonetic Alphabet feature representation vector and Wubi feature representation vector respectively, obtain the converted International Phonetic Alphabet feature representation vector and Wubi feature representation vector, and input them into the embedding layer;

[0031] The embedding layer is used to map the converted International Phonetic Alphabet feature representation vector and Wubi feature representation vector into a continuous vector space respectively, obtain the International Phonetic Alphabet embedding vector and Wubi embedding vector, and transmit them to the first convolutional neural network;

[0032] The first convolutional neural network is used to perform local feature extraction on the input character feature representation vector, International Phonetic Alphabet embedding vector, and Wubi embedding vector respectively through convolution operations, obtain the character local feature representation, International Phonetic Alphabet local feature representation, and Wubi local feature representation, and transmit them to the first linear neural network;

[0033] The first linear neural network is used to map the character local feature representation, the IPA local feature representation, and the Wubi local feature representation to the same dimension through linear transformation, obtain the adjusted character local feature representation, IPA local feature representation, and Wubi local feature representation, and transmit them to the normalization layer;

[0034] The normalization layer is used to perform normalization processing on the adjusted character local feature representation, IPA local feature representation, and Wubi local feature representation, generate the normalized character local feature representation, IPA local feature representation, and Wubi local feature representation, and transmit them to the normalization layer;

[0035] The normalization layer is used to scale the normalized character local feature representation, IPA local feature representation, and Wubi local feature representation to the same fixed range, obtaining the character feature vector, IPA feature vector, and Wubi feature vector.

[0036] Further, the specific content of using the multi-modal feature attention fusion model to fuse the character feature vector, IPA feature vector, and Wubi feature vector is as follows:

[0037] For any labeled text paragraph, two convolutional neural networks are respectively used to extract features from the IPA feature vector and the Wubi feature vector of the text paragraph, and then the feature vectors extracted by the two convolutional neural networks are linearly concatenated to generate a glyph-IPA feature representation;

[0038] A multi-modal feature attention fusion mechanism is used to perform feature fusion on the character feature vector and the glyph-IPA feature representation to generate a multi-modal fusion feature.

[0039] Further, the method of using the multi-modal feature attention fusion mechanism to perform feature fusion on the character feature vector and the glyph-IPA feature representation is as follows:

[0040] The character feature vector and the glyph-IPA feature representation are adjusted to vector representations of the same dimension through linear transformation, obtaining the aligned character feature vector and the aligned glyph-IPA feature representation;

[0041] An attention module is respectively defined for the aligned character feature vector and the aligned glyph-IPA feature representation, the aligned character feature vector and the aligned glyph-IPA feature representation are input into their respective attention modules, the attention scores of the character feature vector and the attention scores of the glyph-IPA feature representation are respectively calculated, and then the attention scores of the character feature vector and the attention scores of the glyph-IPA feature representation are normalized to obtain the attention weights of the character feature vector and the attention weights of the glyph-IPA feature representation;

[0042] The character feature vector and the grapheme - phonetic feature representation are weighted and fused according to the attention weights of the character feature vector and the attention weights of the grapheme - phonetic feature representation to obtain a multi - modal fusion feature.

[0043] Further, the feature encoding network model includes, connected in sequence: a second convolutional neural network, a first Dropout layer, a bidirectional long short - term memory model BiLSTM, and a second Dropout layer;

[0044] The second convolutional neural network is used to extract local features of the input multi - modal fusion feature to obtain multi - modal fusion local features;

[0045] The first Dropout layer is used to prevent overfitting by randomly discarding a preset proportion of neurons to obtain multi - modal fusion local features after random discarding processing;

[0046] The bidirectional long short - term memory model BiLSTM is used to extract temporal features of the multi - modal fusion local features after random discarding processing to obtain multi - modal fusion global temporal features;

[0047] The second Dropout layer is used to prevent overfitting by randomly discarding a preset proportion of neurons to obtain multi - modal fusion global temporal features after random discarding processing and use it as a multi - modal fusion feature sequence.

[0048] Further, the specific content of generating the entity sequence of each annotated text paragraph in the financial corpus and calculating the total loss function value of each annotated text paragraph is as follows:

[0049] For any annotated text paragraph in the financial corpus, the multi - modal fusion feature sequence corresponding to the annotated text paragraph is input into the entity recognition annotation decoder; the entity recognition annotation decoder includes, connected in sequence: a second linear neural network, a third Dropout layer, a conditional random field, and a boundary loss function;

[0050] The second linear neural network is used to map the input multi - modal fusion feature sequence to the entity annotation space, and the third Dropout layer is used to regularize the multi - modal fusion feature sequence after space mapping, and then the generated label score vector corresponding to the annotated text paragraph is input into the conditional random field for decoding to obtain the entity sequence of the annotated text paragraph; at the same time, according to the entity sequence of the annotated text paragraph and the true label of the annotated text paragraph, the conditional random field loss function value of the annotated text paragraph is calculated;

[0051] Then, by comparing the entity sequence of the annotated text paragraph with the true label of the annotated text paragraph, the boundary loss function value of the annotated text paragraph is calculated using the boundary loss function;

[0052] Using the conditional random field loss function value of the annotated text paragraph and the boundary loss function value of the annotated text paragraph, a weighted method is adopted to establish the total loss function of the annotated text paragraph, and the total loss function value of the annotated text paragraph is calculated.

[0053] Furthermore, the method for training the multi-modal feature extraction network model, the multi-modal feature attention fusion model, the feature encoding network model, and the entity recognition annotation decoder is as follows:

[0054] The annotated text paragraphs in the training set are sequentially input into the multi-modal feature extraction network model, the multi-modal feature attention fusion model, the feature encoding network model, and the entity recognition annotation decoder for data processing, and the total loss function values of all the annotated text paragraphs in the training set are obtained;

[0055] According to the obtained total loss function value, the network parameters in the multi-modal feature extraction network model, the multi-modal feature attention fusion model, the feature encoding network model, and the entity recognition annotation decoder are updated using the gradient descent method to complete one round of iterative training, and the financial entity recognition evaluation index in the current iteration round is calculated;

[0056] Judge whether the financial entity recognition evaluation index in the current iteration round meets the preset expected value. If not, continue the iterative training until the financial entity recognition evaluation index in the current iteration round meets the preset expected value; if so, use the validation set and the test set to verify and test the updated multi-modal feature extraction network model, multi-modal feature attention fusion model, feature encoding network model, and entity recognition annotation decoder in the current iteration round to obtain the trained multi-modal feature extraction network model, multi-modal feature attention fusion model, feature encoding network model, and entity recognition annotation decoder.

[0057] The beneficial effects of adopting the above technical solutions are as follows:

[0058] The method of the present invention first constructs a named entity corpus, that is, a financial corpus, to provide high-quality training data support. Based on the constructed financial corpus, the method of the present invention uses a deep learning model to extract rich semantic information from the original data, especially the glyph and phonetic information that is often ignored in financial texts. By mining and using the multi-modal feature fusion mechanism, the ignored and finer-grained text information is fused, including: the glyph and phonetic features of Chinese characters, and the character features extracted under the training of the financial large language model FinBERT.

[0059] Considering that the recognition of heteromorphic characters, polyphonic characters, characters with similar shapes and pronunciations can provide additional information sources, the method of the present invention also utilizes the recognition of heteromorphic characters, polyphonic characters, characters with similar shapes and pronunciations to supplement additional information sources, so as to improve the performance of named entity recognition, and thus better complete the named entity recognition task in the financial field. Through the joint embedding of various different information features, the method of the present invention can accurately distinguish the boundaries of financial entities, thereby extracting financial entities that meet the standards, achieving an improvement in the accuracy of financial entity recognition, being able to better complete the named entity recognition task in the financial field, and having a relatively fast operation speed and strong applicability at the same time.

[0060] Based on the fusion of multiple groups of features, the method of the present invention uses a neural network model to extract features from financial data in a financial corpus, and uses a sequence labeling model to obtain financial entity labels, so as to achieve the purpose of efficiently and accurately identifying financial entities. Description of the Drawings

[0061] Figure 1 It is a structural diagram of a named entity recognition system based on multimodal fusion and deep learning in this embodiment;

[0062] Figure 2 It is a flowchart of a named entity recognition method based on multimodal fusion and deep learning in this embodiment. Detailed Embodiment

[0063] To facilitate the understanding of this application, the following further describes the specific embodiments of the present invention in detail in conjunction with the drawings and embodiments. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention. On the contrary, the purpose of providing these embodiments is to make the disclosure content of this application more thoroughly and comprehensively understood.

[0064] A named entity recognition system based on multimodal fusion and deep learning in this embodiment, as Figure 1 shown, the system includes: a named entity corpus construction module, a multimodal feature extraction module, a multimodal feature fusion module, a feature encoding module, and an entity sequence labeling module.

[0065] The named entity corpus construction module is used to obtain the original financial text data, generate text paragraphs and construct a financial corpus by cleaning and segmenting the obtained original financial text data, and then transmit the financial corpus to the multimodal feature extraction module.

[0066] The multimodal feature extraction module is used to perform multimodal feature extraction on each text paragraph in the financial corpus, generate character feature vectors, international phonetic alphabet feature vectors, and Wubi feature vectors for each text paragraph, and transmit them to the multimodal feature fusion module.

[0067] In this embodiment, the multimodal feature extraction module extracts character features, international phonetic alphabet features, and glyph features from the constructed financial corpus using a network model, learns the multimodal features of the corpus, and converts the features into vector representations.

[0068] The multimodal feature fusion module is used to fuse the character feature vectors, international phonetic alphabet feature vectors, and Wubi feature vectors of each text paragraph using a multimodal feature attention fusion mechanism, generate multimodal fusion features for each text paragraph, and transmit them to the feature encoding module.

[0069] In this embodiment, the character feature vectors, international phonetic alphabet feature vectors, and Wubi feature vectors obtained by multimodal feature extraction are fused into one using a multimodal feature fusion mechanism, so that each character in the text paragraph corresponds to these three types of features, so that more information can be used in subsequent training and prediction, so as to obtain vectors that can be used for deep learning model training.

[0070] The feature encoding module is used to perform feature encoding on the multimodal fusion features of each text paragraph, generate multimodal fusion feature sequences for each text paragraph, and transmit them to the entity sequence annotation module.

[0071] In this embodiment, combined with text information, the text entity feature vectors obtained after feature fusion, that is, multimodal fusion features, are further feature extracted, transformed, and represented through a feature encoding network model, so that they are suitable for model training and prediction.

[0072] The entity sequence annotation module is used to perform entity sequence annotation on the multimodal fusion feature sequences of each text paragraph, and generate entity sequences for each text paragraph in the financial corpus.

[0073] In this embodiment, the results of the feature encoding module are input into the entity recognition annotation decoder for model training and tuning, converted back to the original data or a human - understandable form, that is, the preset financial entity categories, and the accuracy of the finally obtained predicted entities is evaluated.

[0074] A named entity recognition method based on multimodal fusion and deep learning in this embodiment is implemented using a named entity recognition system based on multimodal fusion and deep learning, as Figure 2 shown. This method includes the following processes:

[0075] Obtain the original financial text data, perform data cleaning, segmentation, and named entity recognition, generate the annotated text paragraphs, and construct a financial corpus.

[0076] In this embodiment, a web crawler is used to crawl financial corpora from major financial data websites, where the major financial data websites include websites such as Caixin, Business Society, People's Daily Online, and 21st Century Business Herald; the financial corpora include time, title, news content, web links, etc., and are mixed with existing publicly available financial data sets, such as the publicly available entity recognition data set including BosonNLP_NER_6C, to obtain the original financial text data.

[0077] Further, the specific content of obtaining the original financial text data and performing data cleaning, segmentation, and named entity recognition is: obtaining the existing original financial text data, performing data cleaning on the original financial text data, and then dividing the cleaned original financial text data into text paragraphs by identifying the sentence boundaries in the cleaned original financial text data.

[0078] In this embodiment, special symbols, web page tags such as URL tags, short texts, irrelevant content, large spaces, and line breaks are removed from the titles, contents, times, and web links of the news in the crawled website data, and the content is segmented on this basis.

[0079] Set the financial entity categories and annotation criteria, and according to the financial entity categories and annotation criteria, use the BIO annotation format to perform entity annotation on each divided text paragraph respectively, to obtain the annotated text paragraphs and construct a financial corpus.

[0080] Further, the financial entity categories include: personal names, place names, times, organization names, company names, and product names.

[0081] In this embodiment, commonly used entity categories in the financial field are selected to set the financial entity categories, and annotation criteria are formulated for fuzzy and conflicting contexts, and subsequent annotation will be carried out according to this criterion. Based on the set financial entity categories of personal names, place names, times, organization names, company names, and product names and the annotation criteria, the BIO annotation format is used for entity annotation, where B is for begin, used to represent the beginning of an entity; I is for inside, used to represent the middle of an entity; O is for outside, used to represent non-entity.

[0082] Further, the process of constructing the financial corpus is as follows: for any arbitrarily divided text paragraph, perform word segmentation on the text paragraph, and label each split word or character according to the financial entity category and annotation criteria to obtain the true label of the text paragraph; for the same text paragraph independently annotated by multiple annotators, use Kappa consistency to test the true label of the text paragraph. If the test result is not less than the set threshold, generate the annotated text paragraph according to the true label, and use all the annotated text paragraphs to construct the financial corpus; if the test result is less than the set threshold, re - perform data cleaning, segmentation, and entity annotation on the original financial text data, and then perform Kappa consistency test on the newly obtained true label until the test result reaches the set threshold.

[0083] In this embodiment, the text paragraph is split into independent words or characters, and each word or character is annotated with BIO according to the defined financial entity category and annotation criteria. When the same paragraph is independently annotated by multiple annotators, it is necessary to perform Kappa statistic test on the entity annotations of the same text paragraph by multiple annotators and evaluate the consistency of the true labels to ensure the accuracy of the annotation, so as to complete the construction of a high - quality financial corpus. The so - called high - quality is an evaluation obtained by judging the consistency of entity annotations of the same text. In this embodiment, the threshold in Kappa consistency is set to 0.8. When the test result reaches 0.8 or above, it means that there is very good consistency among the multiple annotators.

[0084] The annotated text paragraphs in the financial corpus are divided into a training set, a validation set, and a test set according to a set ratio.

[0085] In this embodiment, by collecting existing financial text data, and successively performing data cleaning, setting financial entity categories and formulating annotation criteria, and using annotation methods for annotation, a high - quality annotated financial corpus is constructed. The data in the financial corpus is divided into a training set, a validation set, and a test set according to a weight ratio of 8:1:1, and can then be directly fed into the corresponding network model for training.

[0086] Use a multi - modal feature extraction network model to perform multi - modal feature extraction on the annotated text paragraphs in the financial corpus, generating character feature vectors, international phonetic alphabet feature vectors, and Wubi feature vectors.

[0087] Further, the specific content of using the multi - modal feature extraction network model to perform multi - modal feature extraction on the annotated text paragraphs in the financial corpus is: denote the financial corpus as S, and S = {c1, c2,..., c n}, where c1 represents the first annotated text paragraph in the financial corpus; c2 represents the second annotated text paragraph in the financial corpus; c n represents the nth annotated text paragraph in the financial corpus.

[0088] Use the pre-trained language model to extract semantic information from all the annotated text paragraphs and generate the character feature representation vectors of each annotated text paragraph, denoted as Z = {z1, z2,..., z n}; where Z represents the set of character feature representation vectors; z1 represents the character feature representation vector of the text paragraph c1; z2 represents the character feature representation vector of the text paragraph c2; z n represents the character feature representation vector of the text paragraph c n .

[0089] In this embodiment, training the financial corpus using the pre-trained language model can extract the character features of each text paragraph in the financial corpus, that is, the features of the text data itself. In this embodiment, the open-source Chinese FinBERT 1.0 pre-trained model is used as the pre-trained language model, and this model embeds rich semantic information. Therefore, the set of character feature representation vectors containing rich semantic information is represented as Z = {z1, z2,..., z n} = FinBERT(S); where FinBERT(·) represents training using the open-source Chinese FinBERT 1.0 pre-trained model.

[0090] Based on the financial corpus, perform character extraction on all the annotated text paragraphs and establish a dictionary mapping according to the extracted characters.

[0091] In this embodiment, based on the financial corpus, screen out the characters that appear and establish a dictionary mapping. Specifically: by extracting all the characters or words that appear and establishing a mapping table from the characters or words to a unique index according to the complete set of characters or words. This mapping table is usually a dictionary.

[0092] For any annotated text paragraph, based on the dictionary mapping, use the International Phonetic Alphabet table to convert the character feature representation vector of this text paragraph into an International Phonetic Alphabet feature representation vector, denoted as: IPA = IPA_dict{i1, i2,…, i n}; where IPA represents the set of International Phonetic Alphabet feature representation vectors; IPA_dict{·} represents vector conversion using the International Phonetic Alphabet table; i1 represents the International Phonetic Alphabet feature representation vector of the text paragraph c1; i2 represents the International Phonetic Alphabet feature representation vector of the text paragraph c2; i n represents the International Phonetic Alphabet feature representation vector of the text paragraph c n .

[0093] In this embodiment, the pronunciation features in the language are captured using the International Phonetic Alphabet (IPA), which is more detailed than the pinyin features. The character feature representation vector of the text paragraph is converted into IPA features, that is, IPA features. Subsequently, through the embedding layer and one-hot encoding in the multi-modal feature extraction network model, it is represented as a static embedding vector. During subsequent training, the static embedding vector does not undergo gradient update.

[0094] The character feature representation vector of the text paragraph is converted into a Wubi feature representation vector using a Wubi code conversion tool, denoted as: Wubi = wubi{w1, w2, …, w n}; where Wubi represents the set of Wubi feature representation vectors; wubi{·} represents the vector conversion using the Wubi code conversion tool; w1 represents the Wubi feature representation vector of the text paragraph c1; w2 represents the Wubi feature representation vector of the text paragraph c2; w n represents the Wubi feature representation vector of the text paragraph c n .

[0095] In this embodiment, wubi is used as the Wubi code conversion tool. By extracting the Wubi features, that is, the glyph features, of the corpus characters, the Chinese characters are decomposed into several strokes or components and converted into vector representations to obtain the Wubi feature representation vectors. Subsequently, through the embedding layer and one-hot encoding in the multi-modal feature extraction network model, it is represented as a static embedding vector. During subsequent training, the static embedding vector does not undergo gradient update.

[0096] The character feature representation vectors, IPA feature representation vectors, and Wubi feature representation vectors of each annotated text paragraph are respectively input into the multi-modal feature extraction network model to obtain the character feature vectors, IPA feature vectors, and Wubi feature vectors of each annotated text paragraph.

[0097] Furthermore, the multi-modal feature extraction network model includes: one-hot encoding, an embedding layer, a first one-dimensional convolutional neural network (Conv1D), a first linear neural network (Linear), a normalization layer, and a normalization layer.

[0098] Among them, the one-hot encoding is used to respectively perform one-hot encoding conversion on the input IPA feature representation vector and Wubi feature representation vector to obtain the converted IPA feature representation vector and Wubi feature representation vector and input them into the embedding layer.

[0099] In this embodiment, one-hot encoding is a coding method that converts discrete features into binary vectors.

[0100] The embedding layer is used to map the converted international phonetic alphabet feature representation vector and the five-stroke feature representation vector into a continuous vector space respectively, obtain the international phonetic alphabet embedding vector and the five-stroke embedding vector, and transmit them to the first convolutional neural network.

[0101] The first convolutional neural network is used to perform local feature extraction on the input character feature representation vector, international phonetic alphabet embedding vector and five-stroke embedding vector respectively through convolution operations, obtain the character local feature representation, international phonetic alphabet local feature representation and five-stroke local feature representation, and transmit them to the first linear neural network.

[0102] In this embodiment, the character feature representation vector is used as a dynamic embedding vector, while the international phonetic alphabet feature representation vector and the five-stroke feature representation vector are respectively converted into static embedding vectors through one-hot encoding and the embedding layer.

[0103] The first linear neural network is used to map the character local feature representation, international phonetic alphabet local feature representation and five-stroke local feature representation to the same dimension through linear transformation, obtain the adjusted character local feature representation, international phonetic alphabet local feature representation and five-stroke local feature representation, and transmit them to the normalization layer.

[0104] The normalization layer is used to perform normalization processing on the adjusted character local feature representation, international phonetic alphabet local feature representation and five-stroke local feature representation, generate the normalized character local feature representation, international phonetic alphabet local feature representation and five-stroke local feature representation, and transmit them to the normalization layer.

[0105] The normalization layer is used to scale the normalized character local feature representation, international phonetic alphabet local feature representation and five-stroke local feature representation to the same fixed range, obtain the character feature vector, international phonetic alphabet feature vector and five-stroke feature vector.

[0106] In this embodiment, by scaling the normalized character local feature representation, international phonetic alphabet local feature representation and five-stroke local feature representation to the same fixed range, the dimension difference is eliminated.

[0107] The multi-modal feature attention fusion model is used to fuse the character feature vector, international phonetic alphabet feature vector and five-stroke feature vector to generate a multi-modal fusion feature.

[0108] Further, the specific content of using the multi-modal feature attention fusion model to fuse the character feature vector, international phonetic alphabet feature vector and five-stroke feature vector is as follows:

[0109] For any labeled text paragraph, two convolutional neural networks are respectively used to extract the phonetic symbol feature vectors and Wubi feature vectors of the text paragraph, and then the feature vectors extracted by the two convolutional neural networks are linearly concatenated to generate a glyph-phonetic symbol feature representation.

[0110] The glyph-phonetic symbol feature representation is:

[0111]

[0112] where Wubi_IPA j is the glyph-phonetic symbol feature representation of the labeled text paragraph j; Conv1d(·) represents feature extraction using a convolutional neural network; Wubi j is the Wubi feature vector of the labeled text paragraph j; represents linear concatenation; IPA j is the phonetic symbol feature vector of the labeled text paragraph j.

[0113] A multi-modal feature attention fusion mechanism is adopted to fuse the character feature vectors and the glyph-phonetic symbol feature representation to generate a multi-modal fusion feature.

[0114] In this embodiment, multi-modal refers to text data that combines text, graphics, and audio features, containing deeper semantic information. All the extracted features are fused through a multi-modal feature fusion model to obtain a unified character-glyph-phonetic symbol fusion feature, that is, the multi-modal fusion feature. This embodiment adopts a multi-modal feature attention fusion mechanism, namely the LAFF mechanism (ECCV2022: LAFF for Text_to_Video Retrieval), and the LAFF mechanism is a mechanism based on the attention mechanism to fuse different modalities to construct a comprehensive vector representation.

[0115] Furthermore, the method of adopting the multi-modal feature attention fusion mechanism to fuse the character feature vectors and the glyph-phonetic symbol feature representation is:

[0116] The character feature vectors and the glyph-phonetic symbol feature representation are adjusted to vector representations of the same dimension through linear transformation to obtain the aligned character feature vectors and the aligned glyph-phonetic symbol feature representation.

[0117] In this embodiment, since the character feature vectors and the glyph-phonetic symbol feature representation may have different dimensions, feature alignment needs to be performed through linear transformation, that is, they are adjusted to the same dimension for subsequent fusion.

[0118] Define an attention module for the aligned character feature vector and the aligned grapheme-phoneme feature representation respectively. Input the aligned character feature vector and the aligned grapheme-phoneme feature representation into their respective attention modules, calculate the attention scores of the character feature vector and the attention scores of the grapheme-phoneme feature representation respectively, and then normalize the attention scores of the character feature vector and the attention scores of the grapheme-phoneme feature representation to obtain the attention weights of the character feature vector and the attention weights of the grapheme-phoneme feature representation.

[0119] Perform weighted fusion on the character feature vector and the grapheme-phoneme feature representation according to the attention weights of the character feature vector and the attention weights of the grapheme-phoneme feature representation to obtain multi-modal fusion features.

[0120] Input the multi-modal fusion features into the feature encoding network model for feature encoding to generate a multi-modal fusion feature sequence.

[0121] Further, the feature encoding network model includes: a second convolutional neural network, a first Dropout layer, a bidirectional long short-term memory model BiLSTM, and a second Dropout layer connected in sequence.

[0122] The second convolutional neural network is used to extract local features from the input multi-modal fusion features to obtain multi-modal fusion local features.

[0123] The first Dropout layer is used to prevent overfitting by randomly discarding a preset proportion of neurons to obtain the multi-modal fusion local features after random discarding processing.

[0124] The bidirectional long short-term memory model BiLSTM is used to extract temporal features from the multi-modal fusion local features after random discarding processing to obtain multi-modal fusion global temporal features.

[0125] The second Dropout layer is used to prevent overfitting by randomly discarding a preset proportion of neurons to obtain the multi-modal fusion global temporal features after random discarding processing and use them as the multi-modal fusion feature sequence.

[0126] In this embodiment, the feature encoding network model is a multi-layer deep learning neural network including a second one-dimensional convolutional neural network (Conv1D), a bidirectional long short-term memory model (BiLSTM), and regularization Dropout, which is used to perform in-depth feature learning and modeling on the input multi-modal fusion features. The second one-dimensional convolutional neural network (Conv1D) and the first Dropout are used to extract local features; the BiLSTM and the second Dropout are used to learn features with long-distance and global information. In the named entity recognition task, the BiLSTM can effectively identify the boundaries of entities by using the context information before and after the entities.

[0127] The entity recognition annotation decoder is used to perform entity sequence annotation on the multi-modal fusion feature sequence, generate the entity sequence of each annotated text paragraph in the financial corpus, and calculate the total loss function value of each annotated text paragraph.

[0128] In this embodiment, inputting the multi-modal fusion feature sequence into the entity recognition annotation decoder for sequence annotation and entity recognition involves techniques such as conditional random field (CRF) and boundary loss functions such as Dice Loss. Since each character in an entity has complex dependencies, that is, the label of the previous character will affect the label of the subsequent character, and the label of the subsequent character will verify the reliability of the label of the previous character, the role of the entity recognition annotation decoder is to use these complex relationships to predict the label of each character in the multi-modal fusion feature sequence, that is, to decode the financial entity category of each character in the BIO format.

[0129] Furthermore, the specific content of generating the entity sequence of each annotated text paragraph in the financial corpus and calculating the total loss function value of each annotated text paragraph is as follows:

[0130] For any annotated text paragraph in the financial corpus, the multi-modal fusion feature sequence corresponding to the annotated text paragraph is input into the entity recognition annotation decoder; the entity recognition annotation decoder includes, connected in sequence: a second linear neural network (Linear), a third Dropout layer, a conditional random field, and a boundary loss function.

[0131] The input multi-modal fusion feature sequence is mapped to the entity annotation space using a second linear neural network, and the third Dropout layer is used to regularize the multi-modal fusion feature sequence after spatial mapping. Then, the generated label score vector corresponding to the annotated text paragraph is input into a conditional random field for decoding to obtain the entity sequence of the annotated text paragraph. At the same time, according to the entity sequence of the annotated text paragraph and the true labels of the annotated text paragraph, the conditional random field loss function value CRF of the annotated text paragraph is calculated.

[0132] Then, by comparing the entity sequence of the annotated text paragraph with the true labels of the annotated text paragraph, the boundary loss function value Dice Loss of the annotated text paragraph is calculated using the boundary loss function.

[0133] Using the conditional random field loss function value CRF and the boundary loss function value Dice Loss of the annotated text paragraph, a weighted method is adopted to establish the total loss function of the annotated text paragraph and calculate the total loss function value of the annotated text paragraph.

[0134] The total loss function is expressed as:

[0135] OUtput=u*CRF+v*Dice Loss

[0136] Where Output represents the total loss function value of the annotated text paragraph; u represents the weight of the conditional random field loss function value CRF; v represents the weight of the boundary loss function value Dice Loss; and u + v = 1.

[0137] In this embodiment, due to the problem of data imbalance brought by the self-made dataset, the multi-modal fusion feature sequence is sent to the boundary loss function (ACL2020: Dice Loss for Data_imbalanced NLP Tasks) to calculate the loss Dice Loss. Due to the dependency relationship between entities, the multi-modal fusion feature sequence is processed by the second linear neural network Linear and the third Dropout layer and then sent to the conditional random field CRF for decoding. The CRF layer can capture the constraint relationship between labels, thereby improving the rationality of the label sequence, and the dynamic programming algorithm is used to find the optimal label sequence. In addition, the conditional random field CRF usually uses the conditional log-likelihood loss (Negative Log-Likelihood, NLL) as its standard loss function. The total loss function is established by assigning corresponding weights to the conditional random field loss function value CRF and the boundary loss function value Dice Loss, thereby updating the gradient.

[0138] Based on the obtained total loss function value, train the multi-modal feature extraction network model, the multi-modal feature attention fusion model, the feature encoding network model, and the entity recognition annotation decoder to obtain the trained multi-modal feature extraction network model, the multi-modal feature attention fusion model, the feature encoding network model, and the entity recognition annotation decoder;

[0139] Further, the method for training the multi-modal feature extraction network model, the multi-modal feature attention fusion model, the feature encoding network model, and the entity recognition annotation decoder is as follows:

[0140] Input the labeled text paragraphs in the training set into the multi-modal feature extraction network model, the multi-modal feature attention fusion model, the feature encoding network model, and the entity recognition annotation decoder in sequence for data processing, and obtain the total loss function value of all the labeled text paragraphs in the training set.

[0141] According to the obtained total loss function value, use the gradient descent method to update the network parameters in the multi-modal feature extraction network model, the multi-modal feature attention fusion model, the feature encoding network model, and the entity recognition annotation decoder, complete one round of iterative training, and calculate the financial entity recognition evaluation metrics at the current iteration round.

[0142] In this embodiment, for any labeled text paragraph, compare the entity sequence of the text paragraph with the true label, and calculate the financial entity recognition evaluation metrics of the text paragraph respectively, that is, the precision p, the recall rate r, and the F1 value, to evaluate the accuracy of entity recognition.

[0143] The calculation method of the precision p is:

[0144]

[0145] Where True Positives (TP) represents the number of samples correctly predicted as the positive class; False Positives (FP) represents the number of samples wrongly predicted as the positive class.

[0146] The calculation method of the recall rate r is:

[0147]

[0148] Where False Negatives (FN) represents the number of samples wrongly predicted as the negative class.

[0149] The calculation method of the F1 value is:

[0150]

[0151] Where F1 score represents the F1 value.

[0152] Determine whether the financial entity recognition evaluation index in the current iteration round meets the preset expected value. If it does not meet, continue the iterative training until the financial entity recognition evaluation index in the current iteration round meets the preset expected value; if it meets, use the validation set and the test set to verify and test the updated multi-modal feature extraction network model, multi-modal feature attention fusion model, feature encoding network model, and entity recognition annotation decoder in the current iteration round, respectively, to obtain the trained multi-modal feature extraction network model, multi-modal feature attention fusion model, feature encoding network model, and entity recognition annotation decoder.

[0153] In this embodiment, compare the entity labels predicted by the model with the true labels, including complete matching and partial matching, and calculate the digital evaluation index, that is, the financial entity recognition evaluation index, including three major indexes of precision, recall, and F1-score based on the confusion matrix. If the financial entity recognition evaluation index does not meet the requirements, continuously adjust some relevant parameters of the neural network to continuously improve the model accuracy, so as to realize the optimization of the model. Until the financial entity recognition evaluation index meets the expected requirements, obtain the trained model and use it in the actual application of financial text named entity recognition.

[0154] Obtain any financial text data, perform data cleaning and segmentation to generate text paragraphs of the financial text data, and then input the generated text paragraphs into the trained multi-modal feature extraction network model, multi-modal feature attention fusion model, feature encoding network model, and entity recognition annotation decoder in sequence for data processing, so as to obtain the entity sequence of the financial text data.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope defined by the claims of the present invention.

Claims

1. A named entity recognition system based on multimodal fusion and deep learning, characterized in that The system includes: a named entity corpus construction module, a multi-modal feature extraction module, a multi-modal feature fusion module, a feature encoding module, and an entity sequence annotation module; The named entity corpus construction module is used to obtain the original financial text data, generate text paragraphs and construct a financial corpus by performing data cleaning and segmentation on the obtained original financial text data, and then transmit the financial corpus to the multi-modal feature extraction module; The multi-modal feature extraction module is used to perform multi-modal feature extraction on each text paragraph in the financial corpus, generate character feature vectors, international phonetic alphabet feature vectors, and five-stroke feature vectors for each text paragraph, and transmit them to the multi-modal feature fusion module; The multi-modal feature fusion module is used to fuse the character feature vectors, international phonetic alphabet feature vectors, and five-stroke feature vectors of each text paragraph by adopting a multi-modal feature attention fusion mechanism, generate multi-modal fusion features for each text paragraph, and transmit them to the feature encoding module; The feature encoding module is used to perform feature encoding on the multi-modal fusion features of each text paragraph, generate multi-modal fusion feature sequences for each text paragraph, and transmit them to the entity sequence annotation module; The entity sequence annotation module is used to perform entity sequence annotation on the multi-modal fusion feature sequences of each text paragraph, and generate entity sequences for each text paragraph in the financial corpus.

2. A named entity recognition method based on multimodal fusion and deep learning, implemented using the named entity recognition system based on multimodal fusion and deep learning, characterized in that, The method includes the following processes: Obtain the original financial text data and perform data cleaning, segmentation, and named entity recognition to generate labeled text paragraphs and construct a financial corpus; Adopt a multi-modal feature extraction network model to perform multi-modal feature extraction on the labeled text paragraphs in the financial corpus to generate character feature vectors, international phonetic alphabet feature vectors, and five-stroke feature vectors; Use a multi-modal feature attention fusion model to fuse the character feature vectors, international phonetic alphabet feature vectors, and five-stroke feature vectors to generate multi-modal fusion features; Input the multi-modal fusion features into a feature encoding network model for feature encoding to generate multi-modal fusion feature sequences; Use an entity recognition annotation decoder to perform entity sequence annotation on the multi-modal fusion feature sequences, generate entity sequences for each labeled text paragraph in the financial corpus, and calculate the total loss function value for each labeled text paragraph; Based on the obtained total loss function value, train the multi-modal feature extraction network model, the multi-modal feature attention fusion model, the feature encoding network model, and the entity recognition annotation decoder to obtain trained multi-modal feature extraction network model, multi-modal feature attention fusion model, feature encoding network model, and entity recognition annotation decoder; Obtain any financial text data and perform data cleaning and segmentation to generate text paragraphs of the financial text data, and then sequentially input the generated text paragraphs into the trained multi-modal feature extraction network model, multi-modal feature attention fusion model, feature encoding network model, and entity recognition annotation decoder for data processing, so as to obtain the entity sequence of the financial text data.

3. The named entity recognition method based on multi-modal fusion and deep learning according to claim 2, wherein The specific content of obtaining the original financial text data and performing data cleaning, segmentation, and named entity recognition is as follows: Obtain the existing original financial text data, perform data cleaning on the original financial text data, and then divide the cleaned original financial text data into text paragraphs by identifying the sentence boundaries in the cleaned original financial text data; Set the financial entity categories and annotation criteria, and according to the financial entity categories and annotation criteria, use the BIO annotation format to perform entity annotation on each divided text paragraph respectively, obtain the annotated text paragraphs, and construct a financial corpus; The financial entity categories include: person names, place names, time, institution names, company names, and product names; The process of constructing the financial corpus is as follows: For any divided text paragraph, perform word segmentation on the text paragraph, and perform annotation on each split word or character according to the financial entity categories and annotation criteria to obtain the true label of the text paragraph; For the same text paragraph independently annotated by multiple annotators, use kappa consistency to test the true label of the text paragraph. If the test result is not less than the set threshold, generate the annotated text paragraph according to the true label, and use all the annotated text paragraphs to construct a financial corpus; If the test result is less than the set threshold, re-perform data cleaning, segmentation, and entity annotation on the original financial text data, and then perform kappa consistency test on the newly obtained true label until the test result reaches the set threshold; Divide the annotated text paragraphs in the financial corpus into a training set, a validation set, and a test set according to a set ratio.

4. The named entity recognition method based on multi-modal fusion and deep learning according to claim 3, characterized in that The specific content of using the multi-modal feature extraction network model to extract multi-modal features from the annotated text paragraphs in the financial corpus is as follows: Use a pre-trained language model to extract semantic information from all the annotated text paragraphs and generate character feature representation vectors for each annotated text paragraph; Based on the financial corpus, perform character extraction on all the annotated text paragraphs and establish a dictionary mapping according to the extracted characters; For any annotated text paragraph, based on the dictionary mapping, use the International Phonetic Alphabet table to convert the character feature representation vector of the text paragraph into an International Phonetic Alphabet feature representation vector; Use a Wubi code conversion tool to convert the character feature representation vector of the text paragraph into a Wubi feature representation vector; Input the character feature representation vector, International Phonetic Alphabet feature representation vector, and Wubi feature representation vector of each annotated text paragraph into the multi-modal feature extraction network model respectively to obtain the character feature vector, International Phonetic Alphabet feature vector, and Wubi feature vector of each annotated text paragraph.

5. The named entity recognition method based on multi-modal fusion and deep learning according to claim 4, wherein The multi-modal feature extraction network model includes: one-hot encoding, an embedding layer, a first convolutional neural network, a first linear neural network, a normalization layer, and a normalization layer; The one-hot encoding is used to perform one-hot encoding conversion on the input International Phonetic Alphabet feature representation vector and Wubi feature representation vector respectively, obtain the converted International Phonetic Alphabet feature representation vector and Wubi feature representation vector, and input them into the embedding layer; The embedding layer is used to map the converted international phonetic alphabet feature representation vectors and five-stroke feature representation vectors into a continuous vector space respectively, obtain the international phonetic alphabet embedding vectors and five-stroke embedding vectors, and transmit them to the first convolutional neural network; The first convolutional neural network is used to perform local feature extraction on the input character feature representation vectors, international phonetic alphabet embedding vectors and five-stroke embedding vectors respectively through convolutional operations, obtain the character local feature representation, international phonetic alphabet local feature representation and five-stroke local feature representation, and transmit them to the first linear neural network; The first linear neural network is used to map the character local feature representation, international phonetic alphabet local feature representation and five-stroke local feature representation to the same dimension through linear transformation, obtain the adjusted character local feature representation, international phonetic alphabet local feature representation and five-stroke local feature representation, and transmit them to the normalization layer; The normalization layer is used to perform normalization processing on the adjusted character local feature representation, international phonetic alphabet local feature representation and five-stroke local feature representation, generate the normalized character local feature representation, international phonetic alphabet local feature representation and five-stroke local feature representation, and transmit them to the normalization layer; The normalization layer is used to scale the normalized character local feature representation, international phonetic alphabet local feature representation and five-stroke local feature representation to the same fixed range, obtain the character feature vectors, international phonetic alphabet feature vectors and five-stroke feature vectors.

6. The named entity recognition method based on multi-modal fusion and deep learning according to claim 5, characterized in that, The specific content of using the multi-modal feature attention fusion model to fuse the character feature vectors, international phonetic alphabet feature vectors and five-stroke feature vectors is as follows: For any labeled text paragraph, two convolutional neural networks are used to extract features from the international phonetic alphabet feature vectors and five-stroke feature vectors of the text paragraph respectively, and then the feature vectors extracted by the two convolutional neural networks are linearly concatenated to generate a glyph-phonetic alphabet feature representation; A multi-modal feature attention fusion mechanism is used to perform feature fusion on the character feature vectors and the glyph-phonetic alphabet feature representation to generate a multi-modal fusion feature.

7. The named entity recognition method based on multi-modal fusion and deep learning according to claim 6, characterized in that, The method of using the multi-modal feature attention fusion mechanism to perform feature fusion on the character feature vectors and the glyph-phonetic alphabet feature representation is as follows: The character feature vectors and the glyph-phonetic alphabet feature representation are adjusted to vector representations of the same dimension through linear transformation to obtain the aligned character feature vectors and the aligned glyph-phonetic alphabet feature representation; An attention module is defined for the aligned character feature vectors and the aligned glyph-phonetic alphabet feature representation respectively. The aligned character feature vectors and the aligned glyph-phonetic alphabet feature representation are input into their respective attention modules, and the attention scores of the character feature vectors and the attention scores of the glyph-phonetic alphabet feature representation are calculated respectively. Then, by normalizing the attention scores of the character feature vectors and the attention scores of the glyph-phonetic alphabet feature representation, the attention weights of the character feature vectors and the attention weights of the glyph-phonetic alphabet feature representation are obtained; The character feature vectors and the glyph-phonetic alphabet feature representation are weighted and fused according to the attention weights of the character feature vectors and the attention weights of the glyph-phonetic alphabet feature representation to obtain a multi-modal fusion feature.

8. The named entity recognition method based on multimodal fusion and deep learning according to claim 7, characterized in that, The feature encoding network model includes, connected in sequence: a second convolutional neural network, a first Dropout layer, a bidirectional long short-term memory model BiLSTM, and a second Dropout layer; The second convolutional neural network is used to perform local feature extraction on the input multi-modal fusion features to obtain multi-modal fusion local features; The first Dropout layer is used to prevent overfitting by randomly discarding a preset proportion of neurons, and obtain the multi-modal fusion local features after random discarding processing; The bidirectional long short-term memory model BiLSTM is used to perform temporal feature extraction on the multi-modal fusion local features after random discarding processing to obtain multi-modal fusion global temporal features; The second Dropout layer is used to prevent overfitting by randomly discarding a preset proportion of neurons, and obtain the multi-modal fusion global temporal features after random discarding processing and use them as the multi-modal fusion feature sequence.

9. The named entity recognition method based on multimodal fusion and deep learning according to claim 8, characterized in that, The specific content of generating the entity sequence of each annotated text paragraph in the generated financial corpus and calculating the total loss function value of each annotated text paragraph is as follows: For any annotated text paragraph in the financial corpus, input the multi-modal fusion feature sequence corresponding to the annotated text paragraph into the entity recognition annotation decoder; the entity recognition annotation decoder includes, connected in sequence: a second linear neural network, a third Dropout layer, a conditional random field, and a boundary loss function; Use the second linear neural network to map the input multi-modal fusion feature sequence to the entity annotation space, use the third Dropout layer to perform regularization processing on the multi-modal fusion feature sequence after space mapping, and then input the generated label score vector corresponding to the annotated text paragraph into the conditional random field for decoding to obtain the entity sequence of the annotated text paragraph; at the same time, calculate the conditional random field loss function value of the annotated text paragraph according to the entity sequence of the annotated text paragraph and the true label of the annotated text paragraph; Then, by comparing the entity sequence of the annotated text paragraph with the true label of the annotated text paragraph, use the boundary loss function to calculate the boundary loss function value of the annotated text paragraph; Use the conditional random field loss function value and the boundary loss function value of the annotated text paragraph, and use a weighted method to establish the total loss function of the annotated text paragraph and calculate the total loss function value of the annotated text paragraph.

10. The named entity recognition method based on multimodal fusion and deep learning according to claim 9, characterized in that The method for training the multi-modal feature extraction network model, the multi-modal feature attention fusion model, the feature encoding network model, and the entity recognition annotation decoder is as follows: Input the annotated text paragraphs in the training set into the multi-modal feature extraction network model, the multi-modal feature attention fusion model, the feature encoding network model, and the entity recognition annotation decoder in sequence for data processing, and obtain the total loss function values of all annotated text paragraphs in the training set; According to the obtained total loss function value, use the gradient descent method to update the network parameters in the multi-modal feature extraction network model, multi-modal feature attention fusion model, feature encoding network model, and entity recognition annotation decoder, complete one round of iterative training, and calculate the financial entity recognition evaluation metrics at the current iteration round; Judge whether the financial entity recognition evaluation metrics at the current iteration round meet the preset expected values. If they do not meet, continue with iterative training until the financial entity recognition evaluation metrics at the current iteration round meet the preset expected values; if they meet, use the validation set and test set to verify and test the updated multi-modal feature extraction network model, multi-modal feature attention fusion model, feature encoding network model, and entity recognition annotation decoder at the current iteration round respectively, to obtain the trained multi-modal feature extraction network model, multi-modal feature attention fusion model, feature encoding network model, and entity recognition annotation decoder.

Images