Fine-grained multi-modal named entity recognition method for flight control test knowledge extraction

Abstract

The application discloses a fine-grained multi-modal named entity recognition method for flight control test knowledge extraction, and belongs to the technical field of aviation test data processing and artificial intelligence fusion. The application comprises the following steps: obtaining flight control test document text and image data; extracting text feature sequences and visual feature sequences; performing semantic enhancement based on a pre-constructed label prototype memory bank, wherein the memory bank comprises multi-modal prototype representations of entities of various categories extracted through a clustering algorithm; calculating cross-modal attention distribution and generating intra-modal correlation matrices through mutual construction operation; performing feature fusion according to the intra-modal correlation matrices; performing sequence decoding on the multi-modal joint features, and outputting entity category labels. The application solves the problem that the existing method lacks reliable guide signals when the text information is insufficient, breaks through the limitation of the traditional one-to-one alignment assumption, can explicitly model the complex many-to-many semantic mapping relationship between the flight control test document text and the visual objects, and realizes deeper semantic fusion.

Description

Technical Field

[0001] This invention belongs to the field of aviation test data processing and artificial intelligence fusion technology, specifically relating to a fine-grained multimodal named entity recognition method for extracting knowledge from flight control tests. Background Technology

[0002] Flight control system testing is a crucial part of aircraft development. It refers to the comprehensive verification of the functionality, performance, stability, and reliability of aircraft components such as flight control laws, control surfaces, sensors, and actuators in laboratory environments (e.g., iron bird test benches, flight simulators) or during actual flight. The flight control testing process involves a vast amount of multimodal documentation, including test outlines, test plans, test process records, test logs, and test conclusions. These documents not only contain free text data but also statistical charts, system architecture diagrams, and flowcharts, collectively forming the key knowledge carriers for flight control system testing.

[0003] The purpose of constructing a knowledge graph for flight control system experiments is to structurally and semantically associate these scattered, multimodal, and isolated data units through entities and relationships, forming a unified, reasonable knowledge network. Entities are a crucial component of the knowledge graph, representing basic units of various objects or concepts in the real world. For example, inertial measurement units, gyroscopes, elevators, and yaw angles can all be abstracted as knowledge entities. Entity knowledge extraction is a key step in constructing a high-quality knowledge graph for flight control system experiments. However, due to the multimodal characteristics of the aforementioned flight control system experiment documents, traditional entity extraction and recognition methods for single-modal documents are inadequate for this task.

[0004] Multimodal named entity recognition (MAR), a fundamental task in natural language processing, aims to identify corresponding organizations, concepts, terms, and predefined types of entities from unstructured text. Its core lies in overcoming the limitations of pure text methods by introducing and fusing text-related image information to supplement semantic cues and improve the accuracy of entity recognition. However, the semantic relationship between images and text in flight control system test documents is not a simple one-to-one correspondence between words and image visual regions, but rather a typical many-to-one, one-to-many, or hierarchical correspondence. Existing traditional MAR methods cannot explicitly model such fine-grained, structured correspondences in engineering scenarios, resulting in insufficient multimodal information fusion and significantly limited entity recognition capabilities in complex aircraft test documents. Furthermore, existing traditional methods often focus on the fusion of original image features and text word vectors, lacking the extraction and guidance of essential semantic features for specific entity categories, making it difficult to focus on truly discriminative multimodal features.

[0005] Therefore, there is a lack of existing technologies for multimodal named entity recognition methods that can achieve fine-grained cross-modal alignment based on the characteristics of flight control test documents. Summary of the Invention

[0006] This invention addresses the aforementioned problems and overcomes the shortcomings of existing technologies by providing a fine-grained multimodal named entity recognition method for extracting knowledge from flight control experiments. It can explicitly model fine-grained structured semantic correspondences between text and images, and utilizes a tag prototype memory to guide feature enhancement, thereby significantly improving the accuracy and robustness of entity recognition in complex engineering documents.

[0007] To achieve the above objectives, the present invention adopts the following technical solution.

[0008] This invention provides a fine-grained multimodal named entity recognition method for knowledge extraction in flight control experiments, comprising the following steps; S1: Obtain multimodal data from the flight control test document, wherein the multimodal data includes text data and image data; S2: Extract the text feature sequence of the text data and the visual feature sequence of the image data respectively; S3: Based on the pre-built tag prototype memory, semantic enhancement is performed on the text feature sequence and the visual feature sequence respectively to obtain enhanced text features and enhanced visual features. The tag prototype memory stores multimodal prototype representations of entities of each category extracted from the training set by a clustering algorithm. S4: Based on the enhanced text features and the enhanced visual features, calculate the cross-modal attention distribution, and generate the intra-modal correlation matrix through the mutual construction operation of the cross-modal attention distribution; S5: Based on the intramodal correlation matrix, the enhanced text features and the enhanced visual features are fused to obtain multimodal joint features; S6: Perform sequence decoding on the multimodal joint features and output the entity category label of each word in the text.

[0009] As a preferred embodiment of the present invention, in step S3, the tag prototype memory is constructed through the following steps: S31: Obtain multimodal entity instances from the training set. Each multimodal entity instance includes text words, entity category labels, and associated images. S32: Map each text word to the semantic space through the encoder to obtain a set of semantic vectors; S33: Divide the semantic vector set according to entity category, and perform clustering operation on the semantic vector set of each entity category to obtain K cluster centers; S34: Select a preset number of neighboring samples for each cluster center to construct an auxiliary semantic library for that category. The auxiliary semantic libraries for all categories together constitute the label prototype memory library.

[0010] As a preferred embodiment of the present invention, in step S33, the clustering operation adopts the K-Means++ algorithm, and the number of clusters K is automatically selected using the silhouette coefficient maximization criterion.

[0011] In a preferred embodiment of the present invention, step S3, the semantic enhancement step includes: The dynamic fusion weights between prototype features and original features derived from the tag prototype memory are calculated using a gating mechanism. The prototype features and the original features are weighted and fused according to the dynamic fusion weights to obtain enhanced text features and enhanced visual features.

[0012] As a preferred embodiment of the present invention, in step S4, the cross-modal attention distribution includes a first cross-attention distribution from text to vision and a second cross-attention distribution from vision to text; The mutual construction operation includes: Multiply the first cross-attention distribution with the transpose of the second cross-attention distribution by matrix multiplication to obtain the intramodal correlation matrix on the text side; Multiply the second cross-attention distribution with the transpose of the first cross-attention distribution to obtain the intramodal correlation matrix on the visual side.

[0013] As a preferred embodiment of the present invention, the multimodal named entity recognition method for flight control test knowledge extraction further includes: Calculate the first KL divergence loss between the intramodal correlation matrix of the text and the text self-attention distribution; Calculate the second KL divergence loss between the intramodal correlation matrix of the visual side and the visual self-attention distribution; The first KL divergence loss and the second KL divergence loss are used as auxiliary losses to jointly optimize the multimodal named entity recognition model with the main loss.

[0014] As a preferred embodiment of the present invention, in step S2, the text feature sequence is extracted by a pre-trained BERT model, and the visual feature sequence is extracted by a pre-trained ResNet model. The visual feature sequence includes local object-level features and global image features.

[0015] As a preferred embodiment of the present invention, the gating mechanism includes: taking the prototype feature as input, calculating gating parameters through linear transformation and sigmoid activation function, and performing element-wise weighted summation of the prototype feature and the original feature according to the gating parameters to obtain enhanced text features or enhanced visual features.

[0016] As a preferred embodiment of the present invention, in step S1, the flight control test document includes one or more of the following: test outline, test plan, test process record, test log, and test conclusion; the image data includes one or more of the following: statistical charts, system structure diagrams, and flowcharts. In step S6, the entity categories include person names, place names, organization names, and other entity categories; the sequence decoding uses a conditional random field to perform globally optimal label sequence decoding.

[0017] In addition, the present invention also provides a fine-grained multimodal named entity recognition system for extracting knowledge from flight control experiments, which is used to implement the fine-grained multimodal named entity recognition method for extracting knowledge from flight control experiments, including a data acquisition module, a feature extraction module, a tag prototype memory module, a semantic enhancement module, a cross-modal alignment module, a feature fusion module, and a decoding output module; The data acquisition module is used to acquire multimodal data from flight control test documents, including text data and image data. The feature extraction module is used to extract the text feature sequence of the text data and the visual feature sequence of the image data; The tag prototype memory module is used to store multimodal prototype representations of entities of each category extracted from the training set by a clustering algorithm. The semantic enhancement module is used to perform semantic enhancement on the text feature sequence and the visual feature sequence based on the tag prototype memory module, respectively, to obtain enhanced text features and enhanced visual features. The cross-modal alignment module is used to calculate the cross-modal attention distribution based on the enhanced text features and the enhanced visual features, and to generate an intra-modal correlation matrix through the mutual construction operation of the cross-modal attention distribution; The feature fusion module is used to fuse the enhanced text features and the enhanced visual features according to the intramodal correlation matrix to obtain multimodal joint features; The decoding output module is used to perform sequence decoding on the multimodal joint features and output the entity category label of each word in the text.

[0018] Beneficial effects of this invention: 1. This invention provides a fine-grained multimodal named entity recognition method for knowledge extraction in flight control experiments. By constructing a tag prototype memory, representative multimodal prototype representations are extracted for each entity category. During the inference stage, prototype-based semantic information is provided to supplement and enhance the input text, solving the problem of existing methods lacking reliable guiding signals when text information is insufficient. When processing new samples, the tag prototype memory can provide strong semantic guidance, effectively supplementing the semantic information of short texts or ambiguous descriptions and suppressing interference from irrelevant noise in images.

[0019] 2. Simultaneously, this invention proposes a novel alignment paradigm based on the mutual construction of attention matrices. Through the high-order synthesis operation of dual cross-attention matrices, text-side association matrices and visual-side association matrices are derived respectively, realizing a structured representation of cross-modal semantic association. This breaks through the limitations of the traditional "one-to-one" alignment assumption and can explicitly model the complex "many-to-many" semantic mapping relationship between text and visual objects in flight control test documents, achieving a deeper level of semantic fusion.

[0020] 3. This invention provides high-quality technical support for the construction of knowledge graphs for flight control experiments by accurately aligning semantics and achieving high recognition accuracy. This invention can explicitly model fine-grained structured semantic correspondences between text and graphics, and use a tag prototype memory to guide feature enhancement, thereby significantly improving the accuracy and robustness of entity recognition in complex engineering documents. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the overall process of a fine-grained multimodal named entity recognition method for extracting knowledge from flight control experiments, provided in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram illustrating the construction process of a tag prototype memory for a fine-grained multimodal named entity recognition method for extracting knowledge from flight control experiments, as provided in an embodiment of the present invention.

[0024] Figure 3 This is a schematic block diagram of a fine-grained multimodal named entity recognition system for extracting knowledge from flight control experiments, provided as an embodiment of the present invention.

[0025] In the diagram, the following modules are marked: 100 for data acquisition, 101 for feature extraction, 102 for tag prototype memory, 103 for semantic enhancement, 104 for cross-modal alignment, 105 for feature fusion, and 106 for decoding output. Detailed Implementation

[0026] To make the technical problems solved, the technical solutions, and the beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0027] This invention proposes a fine-grained multimodal named entity recognition method for knowledge extraction in flight control experiments. Through steps S1 to S6, the multimodal named entity recognition model of this invention is constructed, referred to as the CM-MNER model (CM-MNER: Cluster-guided and Multi-grained Cross-Modal Alignment for Multimodal Named Entity Recognition). The implementation methods and corresponding system implementations of this method are described in detail below.

[0028] like Figure 1 and Figure 2 As shown in the figure, the fine-grained multimodal named entity recognition method for extracting knowledge from flight control experiments provided by this embodiment of the invention specifically includes the following steps; Step S1: Acquire multimodal data; Acquire multimodal data from flight control test documents, including text data and image data. Flight control test documents include one or more of the following: test outline, test plan, test process record, test log, and test conclusions; image data includes one or more of the following: statistical charts, system structure diagrams, and flowcharts.

[0029] Step S2: Extract features; The text feature sequences of the text data and the visual feature sequences of the image data are extracted respectively. Specifically, a pre-trained BERT model is used as the text encoder for the input text sequence X = (x1, x2, …, x…). n After inputting text into BERT, the last hidden state is used as the context-aware text feature representation. The input word embedding sequence interacts deeply with the stacked feedforward network through a multi-head self-attention mechanism, ultimately outputting a context-sensitive dynamic feature sequence: S t =BERT(X)=(t1, t2, …, tn ), where t i ∈R d (i=1,...,n) represents the contextual representation of each word in the text, and d is the word embedding dimension. To obtain the overall feature representation of the text, the output labeled with [CLS] is aggregated: T=h CLS , T∈R d It is the final representation of text features, containing the semantic information of the entire text.

[0030] For image data, the ResNet architecture is used as the core method for image feature extraction. Faster R-CNN is used to detect and extract the top K most salient visual object regions from the image, resulting in an object-level feature set V. obj =(v1,v2, …, v k ), where each v i ∈R dv (i=1,...,k) encodes the visual semantics of the corresponding object. Simultaneously, global image features V are extracted. glo ∈R dv The image feature map F = ResNet(I) is obtained through the output of the last convolutional layer, where F ∈ R. 7 ×7×2048 Perform global average pooling on the feature map F: ;in H 'and W ' represents the height and width of the feature map F, respectively. The visual feature sequence includes local object-level features and global image features.

[0031] Step S3: Perform semantic enhancement based on the tag prototype memory; Based on a pre-built tag prototype memory, semantic enhancement is performed on the text feature sequence and the visual feature sequence respectively to obtain enhanced text features and enhanced visual features. The tag prototype memory stores multimodal prototype representations of entities of each category extracted from the training set through a clustering algorithm.

[0032] The tag prototype memory is constructed through the following steps: S31: Obtain multimodal entity instances from the training set. Each multimodal entity instance includes text words, entity category labels, and associated images. S32: Map each text word to the semantic space through the encoder to obtain a set of semantic vectors; S33: Divide the semantic vector set according to entity category, and perform clustering operation on the semantic vector set of each entity category to obtain K cluster centers; S34: Select a preset number of neighboring samples for each cluster center to construct an auxiliary semantic library for that category. The auxiliary semantic libraries for all categories together constitute the label prototype memory library.

[0033] Specifically, first, a complete structured semantic scan is performed on the original training set; for each text-image pair in the dataset, all entity spans in its text sequence are identified, and each text word, its category label, and its paired original image are encapsulated together as a "multimodal entity instance"; this process is formally represented as: F: D raw →U i {(e i , c i , I)}; where, e i For text word units, c i Label it to its category, where I represents the associated image.

[0034] Then, according to the four predefined entity types (PER, LOC, ORG, MISC), each multimodal entity instance is classified into the corresponding category pool, resulting in four structured multimodal subsets: D PER D LOC D ORG D MISC .

[0035] For each entity category c∈{PER,LOC,ORG,MISC} and its corresponding dataset D c Each text word in different tags is vectorized into a word vector; word vectors from all categories are processed by the same encoder. θ Mapped to the same semantic space R d This yields a set of semantic word vectors: N c This represents the total number of training samples for entity category c.

[0036] For each category's set of word vectors H c The K-Means++ algorithm is used for clustering. In the K-Means++ algorithm, the first cluster center μ1 is randomly selected. In subsequent iterations, when selecting the remaining cluster centers, the squared distance from each sample point to the nearest selected cluster center is calculated. μ represents a center point in the selected cluster center set.

[0037] Finally, using probability The next cluster center is selected, and the number of clusters K is automatically selected using the silhouette coefficient maximization criterion; the N of each category is... c The sample points are divided into K cFor each cluster, the top n samples are selected as the most representative auxiliary semantic library for that category: A=U c∈{PER,LOC,ORG,MISC} A c This auxiliary semantic library is the tag prototype memory library, A c This represents the prototype feature vector of entity category c.

[0038] The semantic enhancement steps include: calculating dynamic fusion weights between prototype features and original features derived from the tag prototype memory through a gating mechanism; performing weighted fusion of the prototype features and the original features according to the dynamic fusion weights to obtain enhanced text features and enhanced visual features; the gating mechanism includes: taking the prototype features as input, calculating gating parameters through linear transformation and sigmoid activation function, and performing element-wise weighted summation of the prototype features and the original features according to the gating parameters to obtain enhanced text features or enhanced visual features.

[0039] Specifically, an input-adaptive gating mechanism is introduced to dynamically weight and integrate enhanced features from different paths; the gating parameter g is defined as: g = σ(W g ·A c + b g ), where W g It is a learnable gating weight, b g It is a learnable bias term, and σ is the sigmoid activation function; the resulting text fusion feature Z is: Z = g⊙A c +(1-g) ⊙T,A c This represents the prototype feature vector of entity category c (extracted from the label prototype memory), and ⊙ represents element-wise multiplication to obtain the enhanced text feature representation. The enhancement of visual features employs the same gating mechanism, resulting in enhanced visual features represented as follows: .

[0040] Step S4: Calculate the cross-modal attention distribution and perform the mutual construction operation; Based on the enhanced text features and the enhanced visual features, a cross-modal attention distribution is calculated, and an intra-modal correlation matrix is ​​generated through the mutual construction operation of the cross-modal attention distribution; the cross-modal attention distribution includes a first cross-attention distribution from text to vision and a second cross-attention distribution from vision to text.

[0041] Specifically, the first cross-attention distribution is represented as: The second cross-attention distribution is represented as: Among them, W Q t W is a learnable projection matrix from text features to query vectors. k vW is a learnable projection matrix from visual features to key vectors. Q v W is a learnable projection matrix from visual features to the query vector. k t d is a learnable projection matrix from text features to key vectors; k This is the scaling factor, typically taken as the dimension of the query / key vector.

[0042] The mutual construction operation includes: performing matrix multiplication of the first cross-attention distribution and the transpose of the second cross-attention distribution to obtain the intra-modal association matrix on the text side; performing matrix multiplication of the second cross-attention distribution and the transpose of the first cross-attention distribution to obtain the intra-modal association matrix on the visual side; the resulting intra-modal association matrix on the text side is: The obtained intramodal correlation matrix on the visual side is: .

[0043] To ensure that the fine-grained cross-modal association structure obtained through steps S1 to S4 can be internalized into the representation learning of the multimodal named entity recognition model of this invention, a consistency constraint loss based on KL divergence is designed; M t and M v We consider the text and image self-attention distributions as ideal, respectively, and encourage the current self-attention distribution of the multimodal named entity recognition model to approach them. This is achieved through the following loss functions: , ;in, Q is the first KL divergence loss between the intramodal correlation matrix of the text and the text self-attention distribution. i t K i t Enhance text features respectively The self-attention query and key vector obtained through linear transformation; where, Q is the first KL divergence loss between the intramodal correlation matrix of the text and the text self-attention distribution. j v K j v respectively enhance visual features The self-attention query and key vector are obtained through linear transformation; this loss function can minimize the difference between the cross-modal alignment-induced association distribution and the intra-modal self-attention distribution.

[0044] Step S5: Fuse to obtain multimodal joint features; The enhanced text features and the enhanced visual features are fused according to the intramodal correlation matrix to obtain multimodal joint features. The fusion method can be weighted summation or linear transformation after concatenation. In this embodiment, the text features are concatenated with the visual features after weighting by the intramodal correlation matrix.

[0045] Step S6: Sequence decoding outputs entity category labels; The multimodal joint features are sequence decoded to output entity category labels for each word in the text; the entity categories include person names, place names, organization names, and other entity categories; a conditional random field is used as the sequence labeling output layer; given the input sequence X = ( x 1， x 2..., x n ) and possible label sequences y=( y 1, y 2..., y n The conditional probability of the label sequence y is: Where Y represents a predefined set of tags with BIO tagging patterns. S i ( y i-1 , y i , X )and Si ( y i-1 ' , y i ' , X ) is a potential function.

[0046] Step S7: Train the multimodal named entity recognition model of the present invention; This invention employs a multi-task collaborative optimization framework, driving the learning of a multimodal named entity recognition model by jointly minimizing two functionally complementary core loss functions: the main loss L for multimodal named entity recognition. MNER Auxiliary loss L with mutual attention alignment MCA Among them, the main loss L MNER For the negative log-likelihood loss in conditional random field decoding, for each image-text pair in the training samples, given the input sequence X and the true label sequence y, the main loss is defined as: L MNER =log(P(Y|X)), where P(Y|X) is the conditional probability of the label sequence y; where the auxiliary loss is the sum of the first KL divergence loss and the second KL divergence loss, i.e.: L MCA = + This auxiliary loss prompts the current self-attention distribution of the multimodal named entity recognition model to converge towards the ideal association matrix generated through cross-modal mutual construction, achieving fine-grained semantic alignment. The total loss function L is expressed as: L = -(L MNER +λ·L MCA =-(log(P(Y|X))+λ·( + In this model, λ is a hyperparameter used to balance the impact of the two losses on the training of the multimodal named entity recognition model. In this embodiment, λ is set to 0.1. The AdamW optimizer is used for training, with a fixed learning rate of 3e-5 and weight decay. The optimal multimodal named entity recognition model is dynamically saved based on the F1 score on the validation set, and the performance of the optimal multimodal named entity recognition model is finally reported on the test set.

[0047] like Figure 3 As shown in the figure, an embodiment of the present invention provides a fine-grained multimodal named entity recognition system for extracting knowledge from flight control experiments, which is used to implement the fine-grained multimodal named entity recognition method for extracting knowledge from flight control experiments. This fine-grained multimodal named entity recognition system for extracting knowledge from flight control experiments includes a data acquisition module 100, a feature extraction module 101, a tag prototype memory module 102, a semantic enhancement module 103, a cross-modal alignment module 104, a feature fusion module 105, and a decoding output module 106.

[0048] The data acquisition module 100 is used to acquire multimodal data from flight control test documents, including text data and image data.

[0049] The feature extraction module 101 is used to extract the text feature sequence of the text data and the visual feature sequence of the image data. Specifically, the module uses a pre-trained BERT model to extract text features and a pre-trained ResNet model to extract image visual features. The visual feature sequence includes local object-level features and global image features.

[0050] The tag prototype memory module 102 is used to store multimodal prototype representations of entities of each category extracted from the training set by a clustering algorithm; the specific construction process of this module is the same as step 3 in the method implementation, and will not be repeated here.

[0051] The semantic enhancement module 103 is used to perform semantic enhancement on the text feature sequence and the visual feature sequence based on the tag prototype memory module, respectively, to obtain enhanced text features and enhanced visual features. The module adopts a gating mechanism, taking the prototype features as input, calculating the gating parameters through linear transformation and sigmoid activation function, and performing element-wise weighted summation on the prototype features and the original features according to the gating parameters to obtain enhanced text features or enhanced visual features.

[0052] The cross-modal alignment module 104 is used to calculate a cross-modal attention distribution based on the enhanced text features and the enhanced visual features, and to generate an intra-modal correlation matrix through the cross-modal attention distribution interoperation; specifically, it calculates the first text-to-visual cross-attention distribution A. t→v The second cross-attention distribution A between vision and text v→t Then, the text-side modal correlation matrix M is obtained through the mutual construction operation. t and the intramodal correlation matrix M of the visual side v .

[0053] The feature fusion module 105 is used to fuse the enhanced text features and the enhanced visual features according to the intramodal correlation matrix to obtain multimodal joint features.

[0054] The decoding output module 106 is used to perform sequence decoding on the multimodal joint features and output the entity category label of each word in the text; this module uses a conditional random field to perform globally optimal label sequence decoding.

[0055] The above modules work together to achieve named entity recognition of multimodal data in flight control test documents. Specifically, the connection between the modules is as follows: the output of the data acquisition module 100 is connected to the input of the feature extraction module 101, and the data acquisition module 100 transmits the acquired multimodal data (text data and image data) to the feature extraction module 101; the output of the feature extraction module 101 is connected to the input of the semantic enhancement module 103, and the feature extraction module 101 transmits the extracted text feature sequences and visual feature sequences to the semantic enhancement module 103 respectively; the output of the tag prototype memory module 102 is connected to the input of the semantic enhancement module 103, and the tag prototype memory module 102 transmits the stored multimodal prototype representations of entities of each category to the semantic enhancement module 103; the output of the semantic enhancement module 103 is connected to the cross-modal alignment module. At the input of module 104, the semantic enhancement module 103 transmits the enhanced text features and enhanced visual features to the cross-modal alignment module 104; the output of the cross-modal alignment module 104 is connected to the input of the feature fusion module 105, which transmits the generated intra-modal correlation matrix to the feature fusion module 105; the semantic enhancement module 103 also directly transmits the enhanced text features and enhanced visual features to the feature fusion module 105, which receives the enhanced features from the semantic enhancement module 103 and the intra-modal correlation matrix from the cross-modal alignment module 104 and fuses them; the output of the feature fusion module 105 is connected to the input of the decoding output module 106, which transmits the fused multimodal joint features to the decoding output module 106; finally, the decoding output module 106 outputs the entity category label.

[0056] During the model training phase, each module is jointly supervised by the loss function and its parameters are updated through backpropagation. During the inference phase, data flows through each module in sequence, and finally outputs entity category labels. This ensures that the system can complete feature extraction, semantic enhancement, cross-modal alignment, feature fusion, and sequence decoding of multimodal data in sequence, and finally output accurate entity recognition results.

[0057] The effectiveness of this invention will be verified through experiments below.

[0058] To verify the effectiveness of this invention, a systematic experiment was conducted on two authoritative benchmark datasets in the field of multimodal named entity recognition: Twitter-2015 and Twitter-2017. The experimental hardware configuration consisted of an Intel Xeon Silver4110 processor and an NVIDIA RTX 2080 Ti graphics card. The evaluation metrics used were precision, recall, and F1 score.

[0059] 1. The main experimental results are as follows; The performance of the CM-MNER model of this invention was compared with that of multiple single-modal baseline models (BiLSTM-CRF, CNN-BiLSTM-CRF, HBiLSTM-CRF, BERT, BERT-CRF) and multi-modal baseline models (ACBC, GHC, GVATT-BERT-CRF, AdaCAN-BERT-CRF, MT-BERT-CRF, OCSGA, Object-AGBAN, ATTR-MMKG-MNER, RpBERT, UMT, UMGF, MAF, HVPNeT, R-GCN, MRC-MNER, SMVAE, MNER-QG, DebiasCL, ICKA, etc.); the experimental results are shown in Table 1 below.

[0060] Table 1: Performance comparison of different competing baselines on the Twitter-2015 and Twitter-2017 datasets (%)

[0061] As shown in Table 1, the CM-MNER model of this invention achieves overall F1 scores of 75.77% and 87.68% on the Twitter-2015 and Twitter-2017 datasets, respectively, significantly outperforming all compared multimodal baseline models and achieving state-of-the-art performance. Particularly on the Twitter-2017 dataset, compared to DebiasCL and ICKA, the F1 score of CM-MNER is improved by 0.84% ​​and 0.84%, respectively. Experimental results demonstrate that this invention effectively solves the guidance problem when text information is insufficient and the cross-modal "one-to-many" alignment challenge through the tag prototype memory module and the cross-modal alignment module.

[0062] 2. The results of the ablation experiment are as follows; To verify the necessity of each innovative module, an ablation experiment was designed. The changes in model performance were observed by removing the tag prototype memory module (w / o LGPC) and the cross-modal alignment module (w / o MCA) one by one. The experimental results are shown in Table 2 below.

[0063] Table 2: Ablation studies of MNER (%)

[0064] Ablation experiments show that removing any key sub-module leads to a significant decrease in model performance. After removing the tag prototype memory module, the F1 scores of Twitter-2015 and Twitter-2017 decreased by 1.15% and 0.93%, respectively. After removing the cross-modal alignment module, the F1 scores decreased by 0.48% and 0.59%, respectively. The performance degradation was more pronounced when both modules were removed simultaneously. This verifies the indispensability of the tag prototype memory module and the cross-modal alignment module in the framework of this invention.

[0065] 3. Results of cross-domain experiments; To verify the model's generalization ability, cross-domain experiments were conducted: training on Twitter-2017 and testing on Twitter-2015 (Twitter-2017→Twitter-2015), and training on Twitter-2015 and testing on Twitter-2017 (Twitter-2015→Twitter-2017); the experimental results are shown in Table 3.

[0066] Table 3: Performance comparison across datasets (%)

[0067] As can be seen from Table 3, the performance of the CM-MNER model of this invention in cross-domain scenarios is significantly better than the comparative baselines UMT and UMGF, indicating that the invention has a strong generalization ability.

[0068] Based on the above experiments, the following conclusions can be drawn: (1) The tag prototype memory module proposed in this invention extracts the most representative multimodal prototypes from all training instances of each entity category through unsupervised clustering, providing reliable cross-modal semantic guidance for the CM-MNER model and effectively solving the problem of lack of reference in existing methods when text information is insufficient.

[0069] (2) The cross-modal alignment module proposed in this invention performs consistent smoothing and noise reduction on the original attention weights through closed computation of bidirectional attention paths (i.e., mutual construction operation), explicitly models the complex "many-to-many" correspondence between text words and visual regions, and improves the robustness of cross-modal alignment.

[0070] (3) The two modules have a synergistic effect: the tag prototype memory module filters noise based on global data distribution and provides high-purity features for the cross-modal alignment module. The association matrix generated by the latter is less affected by accidental noise, thus significantly improving the accuracy of the final identification.

[0071] It is understood that the above specific description of the present invention is only for illustrating the present invention and is not limited to the technical solutions described in the embodiments of the present invention. Those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention to achieve the same technical effect; as long as the use needs are met, they are all within the protection scope of the present invention.

Claims

1. A fine-grained multimodal named entity recognition method for knowledge extraction in flight control experiments, characterized in that, Includes the following steps: S1: Obtain multimodal data from the flight control test document, wherein the multimodal data includes text data and image data; S2: Extract the text feature sequence of the text data and the visual feature sequence of the image data respectively; S3: Based on the pre-built tag prototype memory, semantic enhancement is performed on the text feature sequence and the visual feature sequence respectively to obtain enhanced text features and enhanced visual features. The tag prototype memory stores multimodal prototype representations of entities of each category extracted from the training set by a clustering algorithm. S4: Based on the enhanced text features and the enhanced visual features, calculate the cross-modal attention distribution, and generate the intra-modal correlation matrix through the mutual construction operation of the cross-modal attention distribution; The cross-modal attention distribution includes a first cross-attention distribution from text to vision and a second cross-attention distribution from vision to text; The mutual construction operation includes: Multiply the first cross-attention distribution with the transpose of the second cross-attention distribution by matrix multiplication to obtain the intramodal correlation matrix on the text side; Multiply the second cross-attention distribution with the transpose of the first cross-attention distribution to obtain the intramodal correlation matrix on the visual side; It also includes: calculating the first KL divergence loss between the intramodal correlation matrix of the text side and the text self-attention distribution; Calculate the second KL divergence loss between the intramodal correlation matrix of the visual side and the visual self-attention distribution; The first KL divergence loss and the second KL divergence loss are used as auxiliary losses and jointly optimized with the main loss to optimize the multimodal named entity recognition model; S5: Based on the intramodal correlation matrix, the enhanced text features and the enhanced visual features are fused to obtain multimodal joint features; S6: Perform sequence decoding on the multimodal joint features and output the entity category label of each word in the text.

2. The fine-grained multimodal named entity recognition method for knowledge extraction in flight control experiments according to claim 1, characterized in that, In step S3, the tag prototype memory is constructed through the following steps: S31: Obtain multimodal entity instances from the training set. Each multimodal entity instance includes text words, entity category labels, and associated images. S32: Map each text word to the semantic space through the encoder to obtain a set of semantic vectors; S33: Divide the semantic vector set according to entity category, and perform clustering operation on the semantic vector set of each entity category to obtain K cluster centers; S34: Select a preset number of neighboring samples for each cluster center to construct an auxiliary semantic library for that category. The auxiliary semantic libraries for all categories together constitute the label prototype memory library.

3. The fine-grained multimodal named entity recognition method for knowledge extraction in flight control experiments according to claim 2, characterized in that, In step S33, the clustering operation uses the K-Means++ algorithm, and the number of clusters K is automatically selected using the silhouette coefficient maximization criterion.

4. The fine-grained multimodal named entity recognition method for knowledge extraction in flight control experiments according to claim 1, characterized in that, In step S3, the semantic enhancement step includes: The dynamic fusion weights between prototype features and original features derived from the tag prototype memory are calculated using a gating mechanism. The prototype features and the original features are weighted and fused according to the dynamic fusion weights to obtain enhanced text features and enhanced visual features.

5. The fine-grained multimodal named entity recognition method for knowledge extraction in flight control experiments according to claim 1, characterized in that, In step S2, the text feature sequence is extracted using a pre-trained BERT model, and the visual feature sequence is extracted using a pre-trained ResNet model. The visual feature sequence includes local object-level features and global image features.

6. The fine-grained multimodal named entity recognition method for knowledge extraction in flight control experiments according to claim 4, characterized in that, The gating mechanism includes: taking the prototype feature as input, calculating the gating parameters through linear transformation and sigmoid activation function, and performing element-wise weighted summation of the prototype feature and the original feature according to the gating parameters to obtain enhanced text features or enhanced visual features.

7. The fine-grained multimodal named entity recognition method for knowledge extraction in flight control experiments according to claim 1, characterized in that, In step S1, the flight control test document includes one or more of the following: test outline, test plan, test process record, test log, and test conclusion; the image data includes one or more of the following: statistical charts, system structure diagrams, and flowcharts. In step S6, the entity categories include person names, place names, organization names, and other entity categories; the sequence decoding uses a conditional random field to perform globally optimal label sequence decoding.

8. A fine-grained multimodal named entity recognition system for knowledge extraction in flight control experiments, characterized in that, The fine-grained multimodal named entity recognition method for extracting knowledge from flight control experiments according to any one of claims 1 to 7 includes a data acquisition module, a feature extraction module, a tag prototype memory module, a semantic enhancement module, a cross-modal alignment module, a feature fusion module, and a decoding output module. The data acquisition module is used to acquire multimodal data from flight control test documents, including text data and image data. The feature extraction module is used to extract the text feature sequence of the text data and the visual feature sequence of the image data; The tag prototype memory module is used to store multimodal prototype representations of entities of each category extracted from the training set by a clustering algorithm. The semantic enhancement module is used to perform semantic enhancement on the text feature sequence and the visual feature sequence based on the tag prototype memory module, respectively, to obtain enhanced text features and enhanced visual features. The cross-modal alignment module is used to calculate the cross-modal attention distribution based on the enhanced text features and the enhanced visual features, and to generate an intra-modal correlation matrix through the mutual construction operation of the cross-modal attention distribution; The feature fusion module is used to fuse the enhanced text features and the enhanced visual features according to the intramodal correlation matrix to obtain multimodal joint features; The decoding output module is used to perform sequence decoding on the multimodal joint features and output the entity category label of each word in the text.

Images