An open domain multi-modal recognition method for aviation safety hotspot events

Abstract

The present application belongs to the technical field of open domain multi-modal classification, and proposes an open domain multi-modal recognition method for aviation safety hotspot events. The method extracts text features and image features of aviation safety hotspot events, introduces a memory enhancement module, designs a scene guide pseudo-event filter, and combines source events and pseudo events to obtain the final classification, providing more effective help for responding to and managing aviation safety hotspot events. Compared with the prior art, the present application can effectively solve the multi-modal aviation safety hotspot event classification task in a complex scene. At the same time, the influence of unknown events on classification is considered, and known events and unknown events are distinguished, avoiding the problem of reducing the classification accuracy of unknown events.

Description

Technical Field

[0001] This invention relates to the field of open-domain multimodal technology, and in particular to an open-domain multimodal identification method for aviation safety hotspot events. Background Technology

[0002] Timely and effective information processing for aviation crisis events is crucial for incident response, analysis, and recovery. Currently, information processing for aviation safety hotspots relies primarily on cross-disciplinary expert experience and long-term professional investigations, which is insufficient to handle the rapidly increasing volume of text and image data, thus limiting response and management capabilities for aviation safety hotspots. Furthermore, classification methods based on known events have significant shortcomings in handling potential crisis events. Traditional classification methods may misclassify new events and fail to accurately capture the occurrence of unknown events. Therefore, how to generalize from known events to unknown events and achieve accurate classification of unknown events has become an urgent problem to be solved. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes an open-domain multimodal identification method for aviation safety hotspot events. This method constructs a comprehensive multimodal aviation safety hotspot event dataset and utilizes this dataset for in-depth event analysis and classification, thereby effectively completing complex multimodal aviation safety hotspot event classification tasks. Furthermore, this method possesses the ability to accurately classify known categories of aviation safety hotspot events and can identify unknown events, thus providing more comprehensive and accurate support for responding to and managing crisis events.

[0004] The specific technical solution of the present invention is as follows:

[0005] An open-domain multimodal identification method for aviation safety hotspot events includes the following steps:

[0006] Step S1: Construct a multimodal aviation safety hotspot event dataset;

[0007] Step S2: Build BERT and ResNet models to extract text features and image features from the multimodal aviation safety hotspot event dataset, respectively, to obtain the source event set;

[0008] Step S3: Construct a Transformer network model, in which the multimodal cross-attention module is used to fuse text features and image features to obtain the first fused feature. The memory enhancement module in the Transformer network model is used to process the first fused feature to obtain the second fused feature, so as to improve the accuracy of the representation of aviation safety hot events.

[0009] Step S4: Based on the source event set, a pseudo event set is generated using a random event simulator, and a third fusion feature is obtained by using a scene-guided pseudo event filter;

[0010] Step S5: Design an unknown event classifier to classify aviation safety hotspot events;

[0011] Step S6: Use the multimodal aviation safety hotspot event dataset from Step S1 to train the BERT model, ResNet model, Transformer network model, scene-guided pseudo-event filter, and unknown event classifier from Steps S2 to S5. Once completed, it can be used for the classification and identification of aviation safety hotspot events.

[0012] Preferably, step S1 specifically includes: collecting posts about aviation safety hot topics via the Internet, each post including data in two modalities: text features and image features, and obtaining a multimodal aviation safety hot topic dataset through manual annotation, filtering and sampling.

[0013] Preferably, the text features F extracted by the BERT model in step S2 are... t It is expressed as follows:

[0014] F t =BERT(P t )∈R n·d

[0015] Among them, the event text P t =(w1,w2,…,w n ), w1, w2, ..., w n This represents the text in the event text, where n represents the length of the event text, and the corresponding text feature F. t =(t1,t2,…,t n ), where d represents the dimension of the output text features.

[0016] Preferably, the image features F extracted by the ResNet model in step S2 v It is expressed as follows:

[0017] F v =σ(ResNet(P) v )·W v )∈R n·d

[0018] Among them, P v Represents an event image, W v Let σ represent the weight matrix, and σ represent the ReLU function.

[0019] Preferably, the multimodal cross-attention module in step S3 focuses on three feature matrices, represented by query Q, key K, and value V, respectively, and the resulting first fusion feature is represented as:

[0020] MultiAtt(F v ,F t ,F t ) = Att(F v ·W Q ,F t W K ,F t ·W V )

[0021] MultiAtt(F t ,F v ,F v ) = Att(F t ·W Q ,F v ·W K ,F v ·W V )

[0022] Where MultiAtt() represents the multi-head cross-attention function, Att() represents the attention function, and W Q W K Represents the weight matrix.

[0023] Preferably, the second fusion feature obtained by the memory enhancement module in step S3 is as follows:

[0024] (1) Expand the set of keys and values ​​of self-attention into a learnable memory matrix. and get

[0025]

[0026]

[0027] Where MemAtt() represents the memory learning function, and [:] represents the join operation;

[0028] (2) A combination layer of linear transformation and residual operation is used to combine the cross-modal characteristics of various aviation safety hotspot events:

[0029]

[0030] in, This represents a cross-modal combination of features using image features as the query. This represents a cross-modal combination of text features as the query, where Layernormal() represents the normalization layer, and bZ The learnable parameters representing the bias vector;

[0031] (3) Using the TextCNN model to aggregate and reduce the dimension of regional information, the second fusion feature is obtained as follows:

[0032]

[0033] TextCNN() represents a text feature aggregation network.

[0034] Preferably, step S4 specifically includes:

[0035] Step S4-1: Use a random event simulator to recombine the image features and text features belonging to different event categories in the source event set to obtain a pseudo event set;

[0036] Step S4-2: Establish a scene-guided pseudo-event filter, consisting of a scene attention module and a cross-event masking module. The scene attention module identifies and retains valid data that can represent unknown events, while the cross-event masking module filters out data that is similar to known categories or is insufficient to represent unknown events.

[0037] Step S4-3: Use the TextCNN model to aggregate and reduce the dimensionality of region information, and obtain the third fusion feature as follows:

[0038]

[0039] Where MaskAtt() represents the cross-event masking function, F i This represents the fusion characteristics of known events. This indicates the fusion characteristics of pseudo-events.

[0040] Preferably, in step S4-3,

[0041]

[0042] Where Sim() represents finding the similarity matrix, softmax() represents the normalization exponential function, and e represents the scaling factor.

[0043] Preferably, step S5 includes the following steps:

[0044] Step S5-1: Represent the unknown event classifier as follows:

[0045]

[0046] in, The predicted class label is represented by argmax(), which is the function to find the maximum value. c W represents the standard event classifier.a Let C represent the pseudo-event classifier, where C represents the number of known categories and A represents the number of unknown categories.

[0047] The loss function l1 for the unknown event classifier is expressed as:

[0048]

[0049] Among them, y i The class label represents the actual class label, and N represents the number of original scene samples. S This indicates the number of supporting scenario samples;

[0050] Step S5-2: Set the update rules for pseudo-event tags as follows:

[0051]

[0052] in, Let U() represent a uniform distribution, and p = max(f i W a () is the probability threshold that ensures an appropriate distribution of the sample;

[0053] The recognition loss function l2 for identifying unknown pseudo-event labels is:

[0054]

[0055] Step S5-3: Combine the source event set and pseudo-event set for tag classification:

[0056] The hybrid classification loss function L for the unknown event classifier is:

[0057] L=l1+αl2

[0058] Where α is the optimal trade-off parameter;

[0059]

[0060] Among them, Y i This indicates the category to which the predicted event belongs.

[0061] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0062] 1. The present invention proposes an open-domain multimodal recognition method for aviation safety hotspot events, which employs a memory enhancement module. This module incorporates prior knowledge from each modality to improve cross-modal feature fusion in the Transformer network model, thereby achieving a more accurate representation of aviation safety hotspot events and improving recognition accuracy.

[0063] 2. This invention proposes an open-domain multimodal recognition method for aviation safety hotspot events. It designs a scene guidance mechanism to support the pseudo-event generation process, effectively simulating the distribution of unknown events. Based on a scene attention module and a cross-event masking module, it selects supporting scenes of simulated pseudo-events as effective samples for unknown events and distinguishes unknown events from samples of known classes. Unknown events are used to simulate unknown events in end-to-end solutions, thereby expanding the training set and introducing unknown events. This method considers the impact of unknown events on classification, distinguishing known events from unknown events to avoid the problem of unknown events reducing classification accuracy. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly introduced below. The features and advantages of the present invention can be more clearly understood by referring to the accompanying drawings. The accompanying drawings are schematic and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart of an open-domain multimodal identification method for aviation safety hotspot events according to the present invention.

[0066] Figure 2 This is a schematic diagram of the memory enhancement module.

[0067] Figure 3 This is a visual diagram illustrating the classification of pseudo-event samples in the feature space.

[0068] Figure 4 This is a data flow diagram of an open-domain multimodal identification method for aviation safety hotspot events according to the present invention. Detailed Implementation

[0069] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0070] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0071] This invention proposes an open-domain multimodal identification method for aviation safety hotspot events. For example... Figure 1 , Figure 4As shown, firstly, textual and image features of aviation safety hotspot events are extracted. Then, the multimodal cross-attention module in the Transformer network model is used to merge the multimodal features. A memory enhancement module is added to the multimodal Transformer network model to better represent aviation safety hotspot events by balancing the features extracted from each single modality and the features fused across modalities. Through a scenario-guided simulation mechanism, an effective set of pseudo-events is generated to solve the classification task of unknown aviation safety hotspot events in the open domain, avoiding invalid samples generated in the end-to-end solution. An unknown event classifier based on source events and pseudo-events is designed to improve the recognition of pseudo-events for each unknown event class and avoid over-focusing on any particular category.

[0072] Example 1

[0073] Fatal aviation accidents (such as plane crashes) are considered typical safety incidents in the air transport sector and are classified as "aviation safety hotspots." The open-domain multimodal recognition method proposed in this invention is used for identification and classification, as detailed below:

[0074] Step 1: Construct a multimodal aviation safety hotspot event dataset;

[0075] Due to the unique nature of aviation accidents, information processing is a highly specialized field. To facilitate in-depth research and response to crisis events, a multimodal aviation safety hotspot dataset, named AirCrisisMMD, was constructed. This dataset was collected from the internet and manually labeled. The data originates from nine fatal aviation accidents recorded by the Aviation Safety Network (ASN) since 2010, specifically 9 categories of labels: 4U9525, ET302, JT610, MH17, MH370, MU5735, PS752, QZ8501, and SJ182.

[0076] Posts from the date of an event to two years after it were searched using keywords for each event. Each post included both text and image data. Based on the raw text and image data, filtering and sampling were performed to obtain a multimodal aviation safety hotspot event dataset. Specific processing steps included:

[0077] (1) In order to create a comprehensive multimodal aviation safety hotspot event dataset, it is necessary to eliminate data with missing information that only has one specific modality and ensure that each sample in the dataset has multimodal characteristics;

[0078] (2) Posts containing a text paragraph and multiple images are divided into several new samples. Specifically, if the same text corresponds to multiple images, it is converted into multiple image-text pairs;

[0079] (3) Discard all image and text pairs containing only emojis in the text to avoid interference from non-informative characters and ensure that the text information in the dataset has actual semantic value;

[0080] (4) By analyzing the features of different modalities, similar image and text pairs are identified and deduplicated to generate a comprehensive feature vector. By calculating the distance between the feature vectors, highly similar entries are removed.

[0081] Step 2: Extract text and image features from aviation safety hotspot events to obtain the source event set.

[0082] To understand word connections between short sentences in texts of aviation safety hotspot events, a BERT model is used to learn complex semantic representations. Given a single event text, represented as P... t =(w1,w2,…,w n ), w1, w2, ..., w n This represents the text in the event text, where n represents the length of the event text, and the corresponding text feature F. t =(t1,t2,…,t n ), which is defined as follows:

[0083] F t =BERT(P t )∈R n·d

[0084] Where d represents the dimension of the output text features.

[0085] Secondly, a ResNet model is used to acquire image features to maintain more layers of feature representation and better understand the semantic information in the event image. First, the event image P to be published... v The size was adjusted to 224×224 pixels. Secondly, the penultimate pooling layer of the ResNet model, V=(v1,v2,…,v…), was used. n This layer extracts image region features; it contains n image regions. To fit the model, a linear layer further transforms the image features into the same dimension as the text features.

[0086] F v =σ(ResNet(P) v )·W v )∈R n·d

[0087] Among them W v Let r represent the weight matrix, σ represent the ReLU function, r represent the number of image regions, and d represent the dimension of the image features after linear transformation.

[0088] Step 3: Multimodal feature fusion of aviation safety hotspot events.

[0089] To effectively filter redundant information in multimodal data and capture the relationships between different modalities, a multimodal cross-attention module based on a Transformer network model is introduced to enhance the fusion of image and text features. A cross-attention mechanism between images and text is employed to generate similarity scores, representing the correlation between modalities and helping to extract high-quality complementary information. This information is crucial for learning the important weights of different text and image regions.

[0090] Cross-attention is a core component of the Transformer, relying on the pairwise relationship between queries and keys. It determines the weighted sum of corresponding values ​​by calculating similarity scores between queries and keys. This design allows the module to more accurately capture the relationships between different modalities. Therefore, the multimodal cross-attention module consists of two masking units, processing text features and image features respectively.

[0091] All intramodal and crossmodal interactions between text and image features are modeled using scaled dot-product attention without recursion. The multi-head attention in the Transformer network model focuses on three feature matrices, denoted as query Q, key K, and value V. The dot product between query Q and key K is computed to obtain a similarity score, which is then weighted with value V. The scaled dot-product attention can be represented as:

[0092]

[0093] Where e is the scaling factor. Given a set of image regions extracted from the input image, the permutation-invariant encoding of the image regions can be obtained using the self-attention operation used in the Transformer network model. Based on this, the query Q, key K, and value V are obtained by linearly projecting the input features. Therefore, for any feature F, the projection of the input features can be defined as:

[0094]

[0095] in, These three are learnable weight matrices of the same dimension. They reflect the correlation between modalities, upon which the model can extract high-level complementary information and learn key points of different text and image regions. Self-attention mechanisms are essentially encoding pairwise relationships between elements in the input data. When using image regions (or features from image regions) as the input set, self-attention mechanisms can encode the relationships between regions in the image before describing the image content, providing a foundation for understanding the entire image. However, this characteristic of self-attention has significant limitations. Because everything relies entirely on pairwise similarity, it cannot build a priori knowledge model of the relationships between image regions.

[0096] The feature vectors of one modality are used as the query, and the feature vectors of the other modality are converted into keys and values. Therefore, the first fused feature is written as a cross-attention expression, as follows:

[0097] MultiAtt(F v ,F t ,F t ) = Att(F v ·W Q ,F t ·W K ,F t ·W V )

[0098] MultiAtt(F t ,F v ,F v ) = Att(F t ·W Q ,F v ·W K ,F v ·W V ).

[0099] Step 4: Feature representation based on memory enhancement module.

[0100] By using a memory augmentation module in the Transformer network model, better feature representations are obtained, maximizing information exploration for each mode. For example, when processing text features, isolated keywords can be identified and retained, words that are often incorrectly filtered out due to their low similarity to image regions. In this way, the memory augmentation module helps correct errors, ensuring that important information in the text is not overlooked. Simultaneously, prior to the cross-attention operation, the key-value set used for self-attention is expanded into additional slots to embed prior knowledge, thus providing the model with a richer background information base. These slots are respectively set as learnable memory matrices, denoted as follows: and

[0101]

[0102] Here, MemAtt() represents the memory learning function, and [:] represents the join operation. Intuitively, by adding learnable keys and values, and then using attention, learned knowledge that is not yet embedded can be retrieved. Simultaneously, the formula keeps the query set unchanged. The memory enhancement module can be applied in a multi-head manner, in which case the memory enhancement operation is repeated h times, using a different weight matrix W for each head. Q W K W V and different learnable memory matrices Then, the results from the different heads are concatenated and linear projection is used.

[0103] By combining linear transformation and residual operations with layer normalization, the cross-modal features of various aviation safety hotspot events are combined:

[0104]

[0105]

[0106] in, This represents a cross-modal combination of features using image features as the query. This represents a cross-modal combination of text features as the query, where Layernormal() represents the normalization layer, and b Z The learnable parameters representing the bias vector;

[0107] The TextCNN model is used for region information aggregation and dimensionality reduction to obtain the second fusion feature as follows:

[0108]

[0109] Here, features are extracted using three different sized convolutional kernels and a fully connected layer, such as... Figure 2 As shown.

[0110] Step 5: Scenario-guided learning of supporting event representations.

[0111] Based on the source event set, a pseudo event set is generated using a random event simulator, and a third fusion feature is obtained by using a scene-guided pseudo event filter.

[0112] (1) Random Pseudo-Event Simulation. To handle unknown events, the basic idea is to randomly simulate some pseudo-event samples to simulate potential unknown events. Based on the source event set, a random event simulator is used to recombine image features and text features that originally belonged to different event categories. This recombination breaks the original correspondence, forming new text-image pairs as pseudo-events. For any event with specific characteristics... i =(t i ,v i ) post (x i ,y i ), randomly select another one with feature x j =(t j ,v j ) post (x j ,y j In different classes, where y j ≠y i Then, generate a pseudo-event. The sample, this type of label Unknown, new features During this simulation, the new training set D tr It consists of two parts: the previous source event set, which has N samples, is renamed to D. os The original set, and the additional pseudo-event set D ps It contains N p One sample.

[0113] (2) Establish a scene-guided pseudo-event filter. Due to the randomness of the above process, the sample features of pseudo-events may still be scattered within the feature space of known classes, and unable to explore the space of unknown classes. Therefore, such events are called invalid samples of pseudo-events, and it is necessary to remove invalid samples of pseudo-events. This embodiment uses a scene attention module and a cross-event masking module. The role of the scene attention module is to identify and retain valid samples that can represent unknown events, i.e., "supporting scenes". The above-mentioned supporting scenes are simulated samples that are significantly different from known class samples based on feature similarity measurement. In this way, it is ensured that pseudo-event samples are not just randomly generated, but have the potential to explore the space of unknown classes; the cross-event masking module is responsible for filtering out samples that are too similar to known classes and are insufficient to represent unknown events. The total number of supporting scenes is denoted as N. s Where N satisfies s ≤N p The classification of pseudo-event valid samples and invalid samples in the feature space is as follows: Figure 3 As shown.

[0114] First, the memory enhancement features of pseudo-events. as follows:

[0115]

[0116] Using the sample features of pseudo-events as the query and the sample features of known events as the key, cross-event feature retrieval is performed based on the following formula:

[0117]

[0118] In the formula and F i These represent the fused features of pseudo-events and known events, respectively. Features of the source event are retrieved by querying pseudo-events, and the output of cross-event attention can extract information from both types of events. Furthermore, the scene attention module is modified using a cross-event masking module. (Calculation...) and F i The similarity matrix between them is used to assess the correlation.

[0119]

[0120] Given a specific threshold for supporting scenario u, the elements sim in the similarity matrix are... i,j Convert to binary value as follows:

[0121]

[0122] Then, feature vectors with high similarity scores are masked, and the corresponding samples of pseudo-events are identified as unsupported scenarios. Feature vectors with lower scores are retained, and the corresponding samples are identified as supporting scenarios. Therefore, the proposed scene attention module and cross-event masking module are rewritten as follows:

[0123]

[0124] (3) Using the TextCNN model to aggregate and reduce the dimension of regional information, the third fusion feature is obtained as follows:

[0125]

[0126] Where MaskAtt() represents the cross-event masking function, F i This represents the fusion characteristics of known events. This indicates the fusion characteristics of pseudo-events.

[0127] Step 6: Unknown event classifier based on source events and pseudo-events.

[0128] (1) Labeling of aviation safety hotspots.

[0129] For traditional known aviation safety hotspot event classifier tasks, the classifier W is used. c The second fusion feature f of any eventi Projected to the corresponding class In the tag set {1,2,…,C}. For The definition is as follows:

[0130]

[0131] However, since the model does not encounter instances in the set of unknown events, directly applying the classifier W... c When applied to the identification of unknown events, all unknown events will be predicted into known categories, resulting in unsatisfactory performance in unknown event identification. Therefore, considering scenarios supporting pseudo-events, a pseudo-event classifier W is introduced. a It supports dividing scenarios into several pseudo-classes.

[0132]

[0133] The number of unknown classes is denoted as A (A>1), which is an adjustable parameter.

[0134] Since the original decision boundary of the known event classifier may cross the boundaries of multiple pseudo-classifiers, the original decision boundary should be pushed back to a tighter boundary. Therefore, by adding support scenarios for pseudo-events, a new unknown event classifier can be rewritten, as shown below:

[0135]

[0136] in, The class label represents either a known event or an unknown event. The corresponding loss function l1 is as follows:

[0137]

[0138] (2) Update pseudo-event tags.

[0139] Since the supporting scenario distribution of synthetic pseudo-events is unknown, it is difficult to determine the actual number of pseudo-event categories. Therefore, the following rules are used to update the labels of pseudo-events, denoted as...

[0140]

[0141] Where p = max(f) i W a ) is the probability threshold to ensure an appropriate distribution of samples. U() represents a uniform distribution. Specifically, the labels range from {C+1, C+2, ..., C+A}, with the aim of improving the identification of pseudo-events for each unknown event class and avoiding over-attention to any particular class. Furthermore, classifier calibration is used to avoid data imbalance in the unknown class. Therefore, the recognition loss function l2 for identifying unknown pseudo-event labels is:

[0142]

[0143] (3) Combine the label classification of source events and pseudo-events.

[0144] By employing mixed events with pseudo-labels, the boundary range of seen categories is effectively narrowed, and the distribution of potentially unseen categories in the feature space is modeled. To achieve this goal, a method for calculating the mixed classification loss is used to balance the distribution of seen and unseen categories in the feature space. Therefore, by combining closed-domain and open-domain classification losses, the overall objective function combines l1 and l2 to improve classification performance. The loss function for the unknown event classifier is:

[0145] L=l1+αl2

[0146] Where α is the optimal trade-off parameter. All identified unknown classes are assigned a single label, denoted as C+1; otherwise, events are classified according to known labels, as shown in the following formula:

[0147]

[0148] In summary, the proposed open-domain multimodal recognition method can distinguish between known and unknown events in aviation safety hotspots, and further classify the known events with detailed labels.

[0149] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0150] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0151] In this invention, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" refers to two or more unless otherwise expressly defined.

[0152] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An open-domain multimodal identification method for aviation safety hotspot events, characterized in that, Includes the following steps: Step S1: Construct a multimodal aviation safety hotspot event dataset; Step S2: Build BERT and ResNet models to extract text features and image features from the multimodal aviation safety hotspot event dataset, respectively, to obtain the source event set; Step S3: Construct a Transformer network model, in which the multimodal cross-attention module is used to fuse text features and image features to obtain the first fused feature. The memory enhancement module in the Transformer network model is used to process the first fused feature to obtain the second fused feature, so as to improve the accuracy of the representation of aviation safety hot events. Step S4: Based on the source event set, a pseudo event set is generated using a random event simulator, and a third fusion feature is obtained by using a scene-guided pseudo event filter; The scene-guided pseudo-event filter consists of a scene attention module and a cross-event masking module. The scene attention module identifies and retains valid data that can represent unknown events, while the cross-event masking module filters out data that is similar to known categories and is insufficient to represent unknown events. The TextCNN model is used to aggregate and reduce the dimensionality of regional information to obtain the third fusion feature. Step S5: Design an unknown event classifier to classify aviation safety hotspot events; Step S6: Use the multimodal aviation safety hotspot event dataset from Step S1 to train the BERT model, ResNet model, Transformer network model, scene-guided pseudo-event filter, and unknown event classifier from Steps S2 to S5. Once completed, it can be used for the classification and identification of aviation safety hotspot events.

2. The open-domain multimodal identification method for aviation safety hotspot events according to claim 1, characterized in that, Step S1 specifically includes: collecting posts about aviation safety hot topics via the Internet, with each post including data in two modalities: text features and image features, and obtaining a multimodal aviation safety hot topic dataset through manual annotation, filtering, and sampling.

3. The open-domain multimodal identification method for aviation safety hotspot events according to claim 1, characterized in that, In step S2, the text features F extracted by the BERT model t It is expressed as follows: F t =BERT(P t )∈R n·d Among them, the event text P t =(w1,w2,…,w n ), w1, w2, ..., w n This represents the text in the event text, where n represents the length of the event text, and the corresponding text feature F. t =(t1,t2,…,t n ), where d represents the dimension of the output text features.

4. The open-domain multimodal identification method for aviation safety hotspot events according to claim 3, characterized in that, The image features F extracted by the ResNet model in step S2 v It is expressed as follows: F v =σ(ResNet(P v )·W v )∈R n·d Among them, P v Represents an event image, W v Let σ represent the weight matrix, and σ represent the ReLU function.

5. The open-domain multimodal identification method for aviation safety hotspot events according to claim 4, characterized in that, The multimodal cross-attention module in step S3 focuses on three feature matrices, represented by query Q, key K, and value V, respectively. The resulting first fusion feature is represented as: MultiAtt(F v ,F t ,F t )=Att(F v ·W Q ,F t ·W K ,F t ·W V ) MultiAtt(F t ,F v ,F v )=Att(F t ·W Q ,F v ·W K ,F v ·W V ) Where MultiAtt() represents the multi-head cross-attention function, Att() represents the attention function, and W Q W K Represents the weight matrix.

6. The open-domain multimodal identification method for aviation safety hotspot events according to claim 5, characterized in that, The second fusion feature method obtained by the memory enhancement module in step S3 is as follows: (1) Expand the set of keys and values ​​for self-attention into a learnable memory matrix. and get Where MemAtt() represents the memory learning function, and [:] represents the join operation; (2) A combination layer of linear transformation and residual operation is used to combine the cross-modal characteristics of various aviation safety hotspot events: in, This represents a cross-modal combination of features using image features as the query. This represents a cross-modal combination of text features as the query, where Laternormal() represents the normalization layer, and b Z The learnable parameters representing the bias vector; (3) Using the TextCNN model to aggregate and reduce the dimension of regional information, the second fusion feature is obtained as follows: TextCNN() represents a text feature aggregation network.

7. The open-domain multimodal identification method for aviation safety hotspot events according to claim 6, characterized in that, In step S4, a random event simulator is used to recombine image features and text features belonging to different event categories in the source event set to obtain a pseudo event set; Third fusion feature The expression is: Where MaskAtt() represents the cross-event masking function, F i This represents the fusion characteristics of known events. This indicates the fusion characteristics of pseudo-events.

8. The open-domain multimodal identification method for aviation safety hotspot events according to claim 7, characterized in that, In step S4-3, Where Sim() represents finding the similarity matrix, softmax() represents the normalization exponential function, and e represents the scaling factor.

9. The open-domain multimodal identification method for aviation safety hotspot events according to claim 8, characterized in that, Step S5 includes the following steps: Step S5-1: Represent the unknown event classifier as follows: in, The predicted class label is represented by argmax(), which is the function to find the maximum value. c W represents the standard event classifier. a Let C represent the pseudo-event classifier, where C represents the number of known categories and A represents the number of unknown categories. The loss function l1 for the unknown event classifier is expressed as: Among them, y i The class label represents the actual class label, and N represents the number of original scene samples. S This indicates the number of supporting scenario samples; Step S5-2: Set the update rules for pseudo-event tags as follows: Where U() represents a uniform distribution, p = max(f i W a () is the probability threshold that ensures an appropriate distribution of the sample; The recognition loss function l2 for identifying unknown pseudo-event labels is: Step S5-3: Combine the source event set and pseudo-event set for tag classification: The hybrid classification loss function L for the unknown event classifier is: L＝l1+αl2 Where α is the optimal trade-off parameter; Among them, Y i This indicates the category to which the predicted event belongs.

Images