Open domain multi-mode identification method for aviation security hotspot events

Abstract

The invention belongs to the technical field of open domain multi-modal classification, and provides an open domain multi-modal identification method for aviation safety hot events. According to the method, the text features and the image features of the aviation safety hot events are extracted, meanwhile, a memory enhancement module is introduced, a scene guide pseudo event filter is designed, final classification is obtained by combining source events and pseudo events, and more effective help is provided for coping with and managing the aviation safety hot events. Compared with the prior art, the multi-modal aviation security hotspot event classification task in a complex scene can be effectively solved. Meanwhile, the influence of unknown events on classification is considered, known events are distinguished from the unknown events, and the problem that the classification precision is reduced by the unknown events is avoided.

Description

Technical Field

[0001] The present invention relates to the field of open domain multimodal technology, and in particular to an open domain multimodal recognition method for aviation safety hotspot events. Background Art

[0002] Timely and effective information processing of aviation crisis events is crucial for accident response, analysis and recovery. At present, the information processing of aviation safety hotspot events mainly relies on cross-domain expert experience and long-term professional investigations, so it is unable to cope with the rapidly growing text and image data, which restricts the response and management capabilities of aviation safety hotspot events. In addition, the classification method based on known events has obvious shortcomings in dealing with potential crisis events. Traditional classification methods may misclassify new events and fail to accurately capture the occurrence of unknown events. Therefore, how to generalize known events to unknown events and achieve accurate classification of unknown events has become an urgent problem to be solved. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes an open domain multimodal identification method for aviation safety hotspot events. This method constructs a comprehensive multimodal aviation safety hotspot event dataset and uses the dataset to conduct in-depth event analysis and classification, thereby effectively completing the complex multimodal aviation safety hotspot event classification task. In addition, the method also has the ability to accurately classify aviation safety hotspot events of known categories and can identify unknown events, thereby providing more comprehensive and accurate support for responding to and managing crisis events.

[0004] The technical solution of the present invention is as follows:

[0005] An open domain multimodal identification method for aviation safety hotspot events comprises the following steps:

[0006] Step S1: Construct a multimodal aviation safety hotspot event dataset;

[0007] Step S2: Establish a BERT model and a ResNet model to extract text features and image features from the multimodal aviation safety hotspot event dataset, respectively, to obtain a source event set;

[0008] Step S3: constructing a Transformer network model, in which the multimodal cross attention module is used to fuse text features and image features to obtain a first fused feature, and the memory enhancement module in the Transformer network model is used to process the first fused feature to obtain a second fused feature, so as to improve the accuracy of the representation of aviation safety hotspot events;

[0009] Step S4: Based on the source event set, a random event simulator is used to generate a pseudo event set, and a scene-guided pseudo event filter is used to obtain a third fusion feature;

[0010] Step S5: Design an unknown event classifier to classify aviation safety hotspot events;

[0011] Step S6: Use the multimodal aviation safety hotspot event dataset in step S1 to train the BERT model, ResNet model, Transformer network model, scenario-guided pseudo-event filter, and unknown event classifier in steps S2 to S5. After completion, they are used for classification and identification of aviation safety hotspot events.

[0012] Preferably, step S1 specifically includes: collecting posts about aviation safety hot events through the Internet, each post including data in two modes: text features and image features, and obtaining a multimodal aviation safety hot event dataset through manual annotation, filtering and sampling.

[0013] Preferably, the text feature F extracted by the BERT model in step S2 t It is expressed as follows:

[0014] F t =BERT(P t )∈R n·d

[0015] Among them, the event text P t =(w1,w2,…,w n ), w1,w2,…,w n represents the text in the event text, n represents the length of the event text, and the corresponding text feature F t =(t1,t2,…,t n ), d represents the dimension of the output text features.

[0016] Preferably, the image feature F extracted by the ResNet model in step S2 v It is expressed as follows:

[0017] F v =σ(ResNet(P v )·W v )∈R n·d

[0018] Among them, P v represents the event image, W v represents the weight matrix, and σ represents the ReLU function.

[0019] Preferably, the multimodal cross attention module in step S3 focuses on three feature matrices, which are respectively represented as query Q, key K and value V, and the obtained first fused feature is represented as:

[0020] MultiAtt(F v ,F t ,F t )=Att(F v ·W Q ,F t W K ,F t ·W V )

[0021] MultiAtt(F t ,F v ,F v )=Att(F t ·W Q ,F v ·W K ,F v ·W V )

[0022] Among them, MultiAtt() represents the multi-head cross attention function, Att() represents the attention function, and W Q , W K Represents the weight matrix.

[0023] Preferably, the second fusion feature method obtained by the memory enhancement module in step S3 is as follows:

[0024] (1) Expand the key and value set of self-attention into a learnable memory matrix and get

[0025]

[0026]

[0027] Among them, MemAtt() represents the memory learning function, and [:] represents the connection operation;

[0028] (2) Linear transformation and residual operation are combined to combine the cross-modal features of various aviation safety hotspot events:

[0029]

[0030] in, represents the cross-modal combined features with image features as query, represents the cross-modal combined features with text features as the query, Layernormal() represents the normalization layer, bZ A learnable parameter representing the bias vector;

[0031] (3) The TextCNN model is used to aggregate and reduce regional information, and the second fusion feature is obtained as follows:

[0032]

[0033] Among them, TextCNN() represents the text feature aggregation network.

[0034] Preferably, the step S4 specifically includes:

[0035] Step S4-1: using a random event simulator to recombine the image features and text features belonging to different event categories in the source event set to obtain a pseudo event set;

[0036] Step S4-2: Establish a scene-guided pseudo-event filter, which is composed of a scene attention module and a cross-event shielding module. The scene attention module identifies and retains valid data that can represent unknown events, and the cross-event shielding module filters out data that is similar to known categories and is insufficient to represent unknown events;

[0037] Step S4-3: Use the TextCNN model to perform regional information aggregation and dimensionality reduction, and obtain the third fusion feature as follows:

[0038]

[0039] Among them, MaskAtt() represents the cross-event masking function, F i represents the fusion features of known events, Fused features representing pseudo events.

[0040] Preferably, in step S4-3,

[0041]

[0042] Among them, Sim() represents the similarity matrix, softmax() represents the normalized exponential function, and e represents the scaling factor.

[0043] Preferably, step S5 comprises the following steps:

[0044] Step S5-1: The unknown event classifier is expressed as:

[0045]

[0046] in, Represents the predicted class label, argmax() represents the maximum value function, W c represents the standard event classifier, Wa represents the pseudo event classifier, C represents the number of known categories, and A represents the number of unknown categories;

[0047] The loss function l1 of the unknown event classifier is expressed as:

[0048]

[0049] Among them, y i represents the real class label, N represents the number of original scene samples, N S Indicates the number of supporting scene samples;

[0050] Step S5-2: Setting the update rules for pseudo event labels as follows:

[0051]

[0052] in, Indicates, U() indicates uniform distribution, p=max(f i W a ) is the probability threshold to ensure the appropriate distribution of samples;

[0053] The recognition loss function l2 for identifying unknown pseudo event labels is:

[0054]

[0055] Step S5-3: Combine the source event set and the pseudo event set to perform label classification:

[0056] The mixed classification loss function L of the unknown event classifier is:

[0057] L=l1+αl2

[0058] Among them, α is the optimal trade-off parameter;

[0059]

[0060] Among them, Y i Indicates the category to which the predicted event belongs.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. The present invention proposes an open domain multimodal recognition method for aviation safety hotspot events, which adopts a memory enhancement module to which prior knowledge in each modality is added to improve the cross-modal feature fusion in the Transformer network model, thereby achieving a more accurate representation of aviation safety hotspot events and improving the accuracy of recognition.

[0063] 2. The present invention proposes an open domain multimodal recognition method for aviation safety hotspot events, designs a scene guidance mechanism to support the pseudo-event generation process, and effectively simulates the distribution of unknown events. Based on the scene attention module and the cross-event shielding module, the supporting scenes of simulated pseudo-events are selected as valid samples of unknown events, and the unknown events are distinguished from the samples of known classes. Unknown events are used to simulate unknown events in the end-to-end solution, thereby expanding the training set and introducing unknown events. This method takes into account the impact of unknown events on classification, distinguishes known events from unknown events, and avoids the problem of unknown events reducing the classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. The features and advantages of the present invention can be more clearly understood by referring to the drawings. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0065] Figure 1 It is a flow chart of an open domain multimodal identification method of aviation safety hotspot events of the present invention.

[0066] Figure 2 It is a structural diagram of the memory enhancement module.

[0067] Figure 3 It is a visualization diagram of pseudo event sample classification in feature space.

[0068] Figure 4 It is a data flow diagram of an open domain multimodal identification method for aviation safety hotspot events of the present invention. DETAILED DESCRIPTION

[0069] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0070] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0071] The present invention proposes an open domain multimodal recognition method for aviation safety hotspot events. Figure 1 , Figure 4As shown in the figure, the text features and image features of aviation safety hotspot events are first extracted, and then the multimodal cross-attention module in the Transformer network model is used to merge the multimodal features; a memory enhancement module is added to the multimodal Transformer network model to better represent aviation safety hotspot events by balancing the features extracted from each single modality and the features fused across modalities; a valid pseudo-event set is generated through a scenario-guided simulation mechanism to solve the task of classifying unknown aviation safety hotspot events in an open domain, and invalid samples of the generation model are avoided in an end-to-end solution; an unknown event classifier based on source events and pseudo-events is designed to improve the recognition of pseudo-events for each unknown event class and avoid over-focusing on any specific category.

[0072] Example 1

[0073] Fatal aviation accidents (such as plane crashes) are considered as typical safety events in the field of aviation transportation and are characterized as "aviation safety hotspot events". The open domain multimodal recognition method proposed in this invention is used for identification and classification, as follows:

[0074] Step 1: Construct a multimodal aviation safety hotspot event dataset;

[0075] Due to the special nature of aviation accidents, information processing is a highly specialized field. In order to conduct in-depth research and respond to crisis events, a multimodal aviation safety hotspot event dataset named AirCrisisMMD is constructed. The dataset is collected from the Internet and manually annotated. The data comes from 9 fatal aviation accidents recorded by the Aviation Safety Network (ASN) since 2010. The specific 9 categories of labels include: 4U9525, ET302, JT610, MH17, MH370, MU5735, PS752, QZ8501, SJ182.

[0076] Keywords of each event are used to search for posts from the date of the event to two years after the event. Each post includes data in two modes: text and image. Based on the original data of text and image pairs, a multimodal aviation safety hotspot event dataset is obtained by filtering and sampling. The specific processing process includes:

[0077] (1) In order to create a comprehensive multimodal aviation safety hotspot event dataset, it is necessary to eliminate data with missing information of only one specific modality to ensure that each sample in the dataset has multimodal characteristics;

[0078] (2) Split a post containing a text paragraph and multiple images into several new samples. Specifically, if the same text corresponds to multiple images, it is converted into multiple image and text pairs;

[0079] (3) Discard all image and text pairs containing only emoticons to avoid interference from uninformative characters and ensure that the text information in the dataset has actual semantic value;

[0080] (4) By analyzing the features of different modalities, we identify and deduplicate image and text pairs with similar contents to generate a comprehensive feature vector. By calculating the distance between feature vectors, we remove entries with high similarity.

[0081] Step 2: Extract text features and image features of aviation safety hot events to obtain the source event set.

[0082] In order to understand the word connections between short sentences in aviation safety hotspot event texts, the BERT model is used to learn complex semantic representations. Given a single event text, represented as P t =(w1,w2,…,w n ), w1,w2,…,w n represents the text in the event text, n represents the length of the event text, and the corresponding text feature F t =(t1,t2,…,t n ), which is defined as follows:

[0083] F t =BERT(P t )∈R n·d

[0084] Where d represents the dimension of the output text features.

[0085] Secondly, the ResNet model is used to obtain image features to maintain more layers of feature representation and better understand the semantic information in event images. v The size is adjusted to 224×224 pixels. Secondly, the penultimate pooling layer V=(v1,v2,…,v n ) extracts image region features, and this layer has n image regions. To fit the model, the image features are further converted to the same dimension as the text features through a linear layer:

[0086] F v =σ(ResNet(P v )·W v )∈R n·d

[0087] Where W v represents the weight matrix, σ represents the ReLU function, the number of image regions is set to r, and the dimension of the image features after linear transformation is also set to d.

[0088] Step 3: Multimodal feature fusion of aviation safety hotspot events.

[0089] In order to effectively filter redundant information in multimodal data and capture the relationship between different modalities, a multimodal cross-attention module based on the Transformer network model is introduced to enhance the fusion effect of image features and text features. The cross-attention mechanism between image and text is used to generate similarity scores to represent the correlation between modalities and help extract high-quality complementary information. The above information is crucial for learning the important weights of different text and image regions.

[0090] The cross-attention mechanism is the core component of Transformer, which relies on the pairwise relationship between the query and the key. By calculating the similarity score between the query and the key, the weighted sum of the corresponding value Value is determined. The above design enables the module to capture the relationship between different modalities more accurately. Therefore, the multimodal cross-attention module consists of two shielding units, which process text features and image features respectively.

[0091] All intra-modal and cross-modal interactions between text features and image features are modeled by scaled dot product attention without using recurrence. The multi-head attention in the Transformer network model focuses on three feature matrices, denoted as query Q, key K, and value V. The dot product between query Q and key K is calculated to obtain the similarity score, which is weighted summed with value V. For scaled dot product attention, it can be expressed as:

[0092]

[0093] Where e is the scaling factor. Given a set of image regions extracted from the input image, the attention mechanism can be used to obtain a permutation-invariant encoding of the image regions through the self-attention operation used in the Transformer network model. On this basis, the query Q, key K, and value V are obtained by linearly projecting the input features. Therefore, for any feature F, the projection of the input feature can be defined as:

[0094]

[0095] in, The three are learnable weight matrices with the same dimension. They reflect the correlation between modalities, on which the model can extract high-level complementary information and learn key points of different text and image regions. The self-attention mechanism is essentially an encoding of pairwise relationships between elements in the input data. When image regions (or features from image regions) are used as the input set, the self-attention mechanism is able to encode the relationships between regions in the image before describing the image content, providing a basis for understanding the entire image. However, this feature of self-attention has obvious limitations. Because everything relies entirely on pairwise similarities, it is impossible to establish a priori knowledge model for the relationship between image regions.

[0096] The feature vector of one modality is taken as the query, and the feature vector of the other modality is converted into keys and values. Therefore, the first fused feature is written as cross attention, and the formula is as follows:

[0097] MultiAtt(F v ,F t ,F t )=Att(F v ·W Q ,F t ·W K ,F t ·W V )

[0098] MultiAtt(F t ,F v ,F v )=Att(F t ·W Q ,F v ·W K ,F v ·W V ).

[0099] Step 4: Feature representation based on memory enhancement module.

[0100] By obtaining better feature representation through the memory enhancement module in the Transformer network model, the information exploration of each mode is maximized. For example, when processing text features, isolated, key words can be identified and retained, while the above words are often mistakenly filtered out because they show low similarity when compared with image regions. In this way, the memory enhancement module helps to correct errors and ensure that important information in the text is not overlooked. At the same time, before the cross-attention operation, the key and value sets for self-attention are expanded into additional slots to embed prior knowledge, thereby providing the model with a richer background information library. The above slots are set as learnable memory matrices, represented as and

[0101]

[0102] Where MemAtt() denotes the memory learning function and [:] denotes the concatenation operation. Intuitively, by adding learnable keys and values, attention can be used to retrieve learned knowledge that has not yet been embedded. At the same time, the formula leaves the query set unchanged. The memory enhancement module can be applied in a multi-headed manner, in which case the memory enhancement operation is repeated h times, using a different weight matrix W for each head. Q , W K , W V and different learnable memory matrices Then, the results from different heads are concatenated and linear projection is used.

[0103] Linear transformation and residual operation combined with layer normalization are used to combine the cross-modal features of various aviation safety hotspot events:

[0104]

[0105]

[0106] in, represents the cross-modal combined features with image features as query, represents the cross-modal combined features with text features as the query, Layernormal() represents the normalization layer, b Z A learnable parameter representing the bias vector;

[0107] The TextCNN model is used to aggregate and reduce regional information, and the second fusion feature is obtained as follows:

[0108]

[0109] Here, features are extracted using three different sizes of convolutional kernels and a fully connected layer, such as Figure 2 shown.

[0110] Step 5: Scenario-guided support event representation learning.

[0111] Based on the source event set, a random event simulator is used to generate a pseudo event set, and a scene-guided pseudo event filter is used to obtain a third fusion feature.

[0112] (1) Random pseudo-event simulation. To handle unknown events, the basic idea is to randomly simulate some pseudo-event samples to simulate potential unknown events. Based on the source event set, a random event simulator is used to recombine image features and text features that originally belonged to different event categories. This recombination breaks the original correspondence and forms a new text-image pair as a pseudo-event. i =(t i ,v i )'s posts (x i ,y i ), randomly select another with feature x j =(t j ,v j )'s posts (x j ,y j ) in different classes, where y j ≠y i Then, a pseudo event is generated The sample of this class label Unknown, new features In this simulation, the new training set D tr It consists of two parts, namely the previous source event set, which has N samples and is renamed to D os The original set of events, and the additional pseudo-event set D ps , which contains N p samples.

[0113] (2) Establish a scene-guided pseudo-event filter. Due to the randomness of the above process, the sample features of pseudo-events may still be scattered in the feature space of known classes and cannot explore the space of unknown classes. Therefore, such events are called invalid samples of pseudo-events, so it is necessary to remove invalid samples of pseudo-events. This embodiment adopts a scene attention module and a cross-event shielding module. The function of the scene attention module is to identify and retain valid samples that can represent unknown events, namely "support scenes". The above-mentioned support scenes are simulated samples that are significantly different from known category samples based on feature similarity metrics. In this way, it is ensured that pseudo-event samples are not just randomly generated, but have the potential to explore the space of unknown categories; the cross-event shielding module is responsible for filtering out samples that are too similar to known categories and are not sufficient to represent unknown events. The total number of support scenes is recorded as N s , where N s ≤N p The classification of valid samples and invalid samples of pseudo events in feature space is as follows: Figure 3 shown.

[0114] First, the memory-enhanced feature representation of pseudo events as follows:

[0115]

[0116] Using the sample features of the pseudo-event as the query and the sample features of the known event as the key value, cross-event feature retrieval is performed based on the following formula:

[0117]

[0118] In the formula and F i Represent the fusion features of pseudo events and known events respectively. By querying pseudo events to retrieve the features of source events, the output of cross-event attention can extract information from both events. And the cross-event masking module is used to modify the scene attention module. Calculation and F i The similarity matrix between them is used to evaluate the correlation.

[0119]

[0120] Given a specific threshold for supporting scenario u, the elements sim in the similarity matrix i,j Convert to binary value as follows:

[0121]

[0122] Then, the feature vectors with higher similarity scores are masked, and the corresponding samples of pseudo events are determined as unsupported scenes. The feature vectors with lower scores are retained, and the corresponding samples are determined as supported scenes. Therefore, the proposed scene attention module and cross-event masking module are rewritten as follows:

[0123]

[0124] (3) The TextCNN model is used to aggregate and reduce regional information, and the third fusion feature is obtained as follows:

[0125]

[0126] Among them, MaskAtt() represents the cross-event masking function, F i represents the fusion features of known events, Fused features representing pseudo events.

[0127] Step 6: Unknown event classifier based on source events and pseudo events.

[0128] (1) Label allocation for aviation safety hotspot events.

[0129] For the traditional known aviation safety hotspot event classifier task, the classifier W c The second fusion feature f of any eventi Project to the corresponding class In the label set {1,2,…,C}. The definition is as follows:

[0130]

[0131] However, since the model has no exposure to instances in the unknown event set, if the classifier W is directly c If it is applied to the recognition of unknown events, all unknown events will be predicted into known categories, resulting in unsatisfactory performance of unknown event recognition. Therefore, considering the pseudo event support scenario, the pseudo event classifier W is introduced a , supports dividing scenes into several pseudo-classes.

[0132]

[0133] The number of unknown classes is denoted as A (A>1), which is an adjustable parameter.

[0134] Since the original decision boundary of the known event classifier may cross the boundary of the multi-class pseudo classifier, the original decision boundary should be pushed back to a tighter boundary. Therefore, by adding the support scenario of pseudo events, the new unknown event classifier can be rewritten as follows:

[0135]

[0136] in, Represents the class label, which can be a known event or an unknown event. The corresponding loss function l1 is as follows:

[0137]

[0138] (2) Update of pseudo event labels.

[0139] Since the distribution of supporting scenarios for synthetic pseudo-events is unknown and it is difficult to determine the actual number of pseudo-event categories, the following rules are used to update the labels of pseudo-events, denoted as

[0140]

[0141] where p = max(f i W a ) is the probability threshold to ensure proper distribution of samples. U() represents uniform distribution. Specifically, labels are distributed from {C+1, C+2, …, C+A}, with the aim of improving the recognition of pseudo events for each unknown event class and avoiding excessive attention to any particular class. In addition, classifier calibration is used to avoid data imbalance for unknown classes. Therefore, the recognition loss function l2 for identifying unknown pseudo event labels is:

[0142]

[0143] (3) Combine label classification of source events and pseudo events.

[0144] By using mixed events with pseudo labels, the boundaries of seen categories are effectively narrowed and the distribution of unseen categories that may exist in the feature space is modeled. To achieve this goal, a method for calculating mixed classification loss is used to balance the distribution of seen and unseen categories in the feature space. Therefore, combining closed domain and open domain classification losses, the overall objective function combines l1 and l2 to improve classification performance. The loss function of the unknown event classifier is:

[0145] L=l1+αl2

[0146] Among them, α is the optimal trade-off parameter. The identified unknown classes are all set to a single label, denoted as C+1, otherwise, the events are classified according to the known labels, as shown in the following formula:

[0147]

[0148] In summary, the proposed open domain multimodal recognition method can distinguish known events from unknown events of aviation safety hotspot events, and classify known events with detailed labels.

[0149] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0150] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0151] In the present invention, the terms "first", "second", "third", "fourth" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The term "plurality" refers to two or more, unless otherwise clearly defined.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An open domain multimodal identification method for aviation safety hotspot events, characterized in that: The following steps are involved: Step S1: Construct a multimodal aviation safety hotspot event dataset; Step S2: Establish a BERT model and a ResNet model to extract text features and image features from the multimodal aviation safety hotspot event dataset, respectively, to obtain a source event set; Step S3: constructing a Transformer network model, in which the multimodal cross attention module is used to fuse text features and image features to obtain a first fused feature, and the memory enhancement module in the Transformer network model is used to process the first fused feature to obtain a second fused feature, so as to improve the accuracy of the representation of aviation safety hotspot events; Step S4: Based on the source event set, a random event simulator is used to generate a pseudo event set, and a scene-guided pseudo event filter is used to obtain a third fusion feature; Step S5: Design an unknown event classifier to classify aviation safety hotspot events; Step S6: Use the multimodal aviation safety hotspot event dataset in step S1 to train the BERT model, ResNet model, Transformer network model, scenario-guided pseudo-event filter, and unknown event classifier in steps S2 to S5. After completion, they are used for classification and identification of aviation safety hotspot events.

2. The open domain multimodal identification method for aviation safety hotspot events according to claim 1 is characterized in that: The step S1 specifically includes: collecting posts related to aviation safety hot events through the Internet, each post including data in two modes: text features and image features, and obtaining a multimodal aviation safety hot event dataset through manual annotation, filtering and sampling.

3. The open domain multimodal identification method for aviation safety hotspot events according to claim 1 is characterized in that: The text feature F extracted by the BERT model in step S2 t It is expressed as follows: F t =BERT(P t )∈R n·d Among them, the event text P t =(w1,w2,…,w n ), w1,w2,…,w n represents the text in the event text, n represents the length of the event text, and the corresponding text feature F t =(t1,t2,…,t n ), d represents the dimension of the output text features.

4. The open domain multimodal identification method for aviation safety hotspot events according to claim 3 is characterized in that: The image feature F extracted by the ResNet model in step S2 v It is expressed as follows: F v =σ(ResNet(P v )·W v )∈R n·d Among them, P v represents the event image, W v represents the weight matrix, and σ represents the ReLU function.

5. The open domain multimodal identification method for aviation safety hotspot events according to claim 4 is characterized in that: The multimodal cross attention module in step S3 focuses on three feature matrices, represented as query Q, key K and value V respectively, and the obtained first fused feature is represented as: MultiAtt(F v ,F t ,F t )=Att(F v ·W Q ,F t ·W K ,F t ·W V ) MultiAtt(F t ,F v ,F v )=Att(F t ·W Q ,F v ·W K ,F v ·W V ) Among them, MultiAtt() represents the multi-head cross attention function, Att() represents the attention function, and W Q , W K Represents the weight matrix.

6. The open domain multimodal identification method of aviation safety hotspot events according to claim 5 is characterized in that: The second fusion feature method obtained by the memory enhancement module in step S3 is as follows: (1) Expand the key and value set of self-attention into a learnable memory matrix and get Among them, MemAtt() represents the memory learning function, and [:] represents the connection operation; (2) Linear transformation and residual operation are combined to combine the cross-modal features of various aviation safety hotspot events: in, represents the cross-modal combined features with image features as query, represents the cross-modal combined features with text features as the query, Layernormal() represents the normalization layer, b Z A learnable parameter representing the bias vector; (3) The TextCNN model is used to aggregate and reduce regional information, and the second fusion feature is obtained as follows: Among them, TextCNN() represents the text feature aggregation network.

7. The open domain multimodal identification method for aviation safety hotspot events according to claim 6 is characterized in that: The step S4 specifically includes: Step S4-1: using a random event simulator to recombine image features and text features belonging to different event categories in the source event set to obtain a pseudo event set; Step S4-2: Establish a scene-guided pseudo-event filter, which is composed of a scene attention module and a cross-event shielding module. The scene attention module identifies and retains valid data that can represent unknown events, and the cross-event shielding module filters out data that is similar to known categories and is insufficient to represent unknown events; Step S4-3: Use the TextCNN model to perform regional information aggregation and dimensionality reduction, and obtain the third fusion feature as follows: Among them, MaskAtt() represents the cross-event masking function, F i represents the fusion features of known events, Fused features representing pseudo events.

8. The open domain multimodal identification method for aviation safety hotspot events according to claim 7 is characterized in that: In the step S4-3, Among them, Sim() represents the similarity matrix, softmax() represents the normalized exponential function, and e represents the scaling factor.

9. The open domain multimodal identification method of aviation safety hotspot events according to claim 8 is characterized in that: The step S5 comprises the following steps: Step S5-1: The unknown event classifier is expressed as: in, Represents the predicted class label, argmax() represents the maximum value function, W c represents the standard event classifier, W a represents the pseudo event classifier, C represents the number of known categories, and A represents the number of unknown categories; The loss function l1 of the unknown event classifier is expressed as: Among them, y i represents the true class label, N represents the number of original scene samples, N S Indicates the number of supporting scene samples; Step S5-2: Setting the update rules for pseudo event labels as follows: in, Indicates, U() indicates uniform distribution, p=max(f i W a ) is the probability threshold to ensure the appropriate distribution of samples; The recognition loss function l2 for identifying unknown pseudo event labels is: Step S5-3: Combine the source event set and the pseudo event set to perform label classification: The mixed classification loss function L of the unknown event classifier is: L＝l1+αl2 Among them, α is the optimal trade-off parameter; Among them, Y i Indicates the category to which the predicted event belongs.

Images