Emergency classification method, system and device based on multi-modal feature fusion

Abstract

The invention provides an emergency classification method, system and device based on multi-modal feature fusion, and mainly solves the problem of non-ideal classification effect caused by incomplete feature information representation in the existing emergency classification method. The method comprises the following steps of: firstly, constructing an emergency sample data set which is represented by multi-modal data and is provided with a category label, and preprocessing the emergency sample data set; then, feature extraction is carried out on text data and image data in the sample by utilizing a BERT model and a ViT model; thirdly, constructing a multi-modal feature fusion module based on a multi-head self-attention mechanism to perform early fusion on the extracted text features and image features, and constructing a feedforward neural network model to perform learning on a single text feature vector, a single image feature vector and a feature vector after early fusion respectively and then add the single text feature vector, the single image feature vector and the feature vector after early fusion to realize late fusion; and finally, constructing a classifier to learn and classify the feature vectors after the late fusion to obtain a final classification result.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and emergency classification, and provides an emergency classification method, system and device based on multimodal feature fusion. Background Art

[0002] In recent years, emergencies have occurred frequently both at home and abroad. The derivative nature of emergencies poses a huge potential threat to society during the communication process. Therefore, how to quickly and effectively identify and classify emergency information has become a current research hotspot.

[0003] Compared with pure text news of emergencies, multimodal media data containing pictures can attract more people's attention. Recent studies have paid more and more attention to the use of multimodal (text + image) information to identify and classify emergencies. At present, most multimodal emergency classification models either simply concatenate and fuse different modal data or perform additive fusion in terms of feature fusion, which cannot fully capture the complementarity between different modal features, thus greatly affecting the final classification results. Using multimodal feature fusion technology to classify emergencies has become the mainstream research in the field of emergency classification. However, how to more cleverly fuse different modal features to further improve the classification accuracy is still a research hotspot in this field.

[0004] The existing emergency classification methods have the following disadvantages:

[0005] (1) Most classification methods only use single text modal features for research, which does not fully describe emergency information and is limited to one feature. They are unable to capture the relationship between modalities and it is difficult to achieve a high classification accuracy.

[0006] (2) Existing multimodal feature fusion methods only perform simple splicing and fusion of different modal features without fully considering the interdependence between different modalities. It is difficult to capture effective information between different modal data, thus affecting the classification results of emergencies.

[0007] (3) In terms of feature fusion, only the early fusion or late fusion method is used. When used alone, the effective feature information of the emergency event is easily ignored, resulting in inaccurate classification results and other problems.

[0008] Due to these shortcomings, there are still problems such as low classification accuracy, poor effect, and insufficient stability in the field of emergency classification. Summary of the invention

[0009] (1) Technical issues to be solved

[0010] In view of the above-mentioned problems, the present invention aims to propose a method, system and device for emergency event classification based on multimodal feature fusion, which combines early fusion and late fusion, and can fully capture the interdependence between text and image features, so as to make up for the limitations of simple feature splicing. At the same time, a more mature deep learning model is used to extract and learn features of different modal data, and the complementarity between the two modal data is utilized to improve the accuracy of emergency event classification.

[0011] (2) Technical solution

[0012] In order to achieve the above object, the present invention adopts the following technical solution:

[0013] A method for emergency classification based on multimodal feature fusion includes the following steps:

[0014] Step 1: By obtaining the text data and related image data of emergency news from the official website of emergencies, the collected graphic data are divided into five categories: natural disasters, accidents and disasters, public health, social security, and non-emergency events according to the "GB / T 35561-2017 Classification and Coding of Emergency Events" specification issued by the state, and an emergency sample data set represented by multimodal data is constructed and preprocessed;

[0015] Step 2: Use the BERT model and the ViT model to extract the features of text modality data and image modality data from the samples respectively;

[0016] Step 3: Use the multi-head self-attention mechanism to build a multimodal feature fusion module to perform early fusion of the extracted text features and image features;

[0017] Step 4: Based on the early fusion feature vector obtained in step 3, a feedforward neural network model is constructed to learn the text feature vector, image feature vector and feature vector after early fusion to obtain feature vectors with three dimensions of 128 each. The three are added and fused to obtain the feature vector after late fusion, i.e. the final feature fusion result;

[0018] Step 5: Construct a classifier to learn and classify the final fused feature vector, and divide it into five categories: natural disasters, accidental disasters, public health, social security, and non-emergency events.

[0019] In the above technical solution, the specific steps of acquiring and preprocessing data in step 1 are:

[0020] A data set was constructed to obtain emergency information from the official website of emergencies, including event title, release time, event text, event source, and corresponding image data in the event. In terms of data preprocessing, two modal data containing text and corresponding images were retained, and the ratio of text and image data samples was 1:1. At the same time, irrelevant content such as emoticons and emojis were removed, and images with low resolution were deduplicated and deleted. Event sample data were denoised and encoded. According to the GB / T35561-2017 Classification and Coding of Emergency Events issued by the state, the collected graphic data were labeled by category. According to the occurrence process, nature and mechanism of emergencies, emergencies were divided into five categories (natural disasters, accidental disasters, public health, social security, and non-emergency events).

[0021] In the above technical solution, the specific steps of extracting features from different modal data in step 2 are:

[0022] The text feature extractor adopts the BERT model and uses the BERT-base with 16 self-attention heads to extract text features. First, the emergency text is input for word embedding operations, including character embedding, segment embedding and position embedding. Among them, the character embedding converts the text into a vector representation of characters through a lookup table, and marks the beginning and end of the text with [CLS] and [SEP]; segment embedding marks the global semantic information of the text and merges it with the character-level vector; position embedding marks the context information and outputs the final word embedding representation; secondly, the word embedding representation result is input into the Encoder layer for self-attention processing and feedforward neural network calculation, and the emergency text feature vector is output after normalization, and the feature vector dimension is 768; the image feature extractor adopts the ViT model, and the input image size is adjusted to 224 ×224; divide the image into patches of fixed size, the patch size is 16×16, then each image will generate 224×224 / 16×16=19 patches, that is, the length of the input sequence is 196, and the dimension of each patch is 16×16×3=768; after a linear projection layer with a dimension of 768×N (N=768), 196 tokens are output, and the dimension of each token is 768; add a special character CLS, and the sequence dimension is 197×768; consider the position relationship between each patch of the image and add position encoding, use the multi-head attention mechanism to mine the relationship between different patches, and perform image classification, extract the feature vector input by the last classification layer and use it as the image feature representation result, the extracted image feature vector dimension is 768.

[0023] In the above technical solution, the specific steps of step 3 are:

[0024] Early fusion of feature vectors of different modalities is performed, and a multimodal feature fusion module is constructed to fuse the extracted text feature vectors and image feature vectors. The multimodal feature fusion module uses a multi-head self-attention mechanism. The query comes from one modality, while the key and value come from another modality. That is, if Q comes from text, K and V come from attached images, then the attention value calculated using Q and K can be used as a similarity measure between text and image, and then the image is weighted, and vice versa. Cross-fusion operations are performed on text features and image features respectively, and the fusion results obtained by the two multi-head self-attention multimodal feature fusion modules are added to fully capture the interdependence between different modalities, thereby obtaining early feature fusion results.

[0025] In the above technical solution, the specific steps of step 4 are:

[0026] For the text feature vectors, image feature vectors and feature vectors after early fusion obtained in steps 2 and 3, a feedforward neural network model is constructed, which has 1 input layer, 1 output layer and 3 hidden layers; the 3 hidden layers contain 1024, 512 and 256 neurons respectively, and the output layer contains 128 neurons; at the same time, PReLU is used as the activation function after each hidden layer, and the three feature vectors are learned separately by this network to obtain feature vectors with dimensions of 128, and then the three are added and fused to obtain the late fusion result, that is, the final fusion result.

[0027] In the above technical solution, the specific steps of step 5 are:

[0028] The classifier is a fully connected layer constructed by an input layer and an output layer. The input layer contains 128 neurons and the output layer contains 5 neurons. Sigmoid is used as the activation function of the fully connected layer to obtain a feature vector with a dimension of 5. Finally, the softmax layer is used to classify and predict the obtained 5-dimensional feature vector to obtain the final classification result.

[0029] An emergency classification system based on multimodal feature fusion, comprising:

[0030] Data acquisition and preprocessing module: used to acquire and process multimodal emergency event data, encode sample data, and classify the collected graphic data into five categories: natural disasters, accidents and disasters, public health, social security, and non-emergency events according to the "GB / T 35561-2017 Classification and Coding of Emergency Events" specification issued by the state;

[0031] Multimodal feature extraction module: used to extract features from the processed multimodal emergency event data, using the BERT model and the ViT model to extract text features and image features respectively;

[0032] Multimodal feature fusion module: used to fuse the extracted feature vectors of different modalities, build a multimodal feature fusion module based on multi-head self-attention mechanism to perform early fusion of text and image features, and then build a feedforward neural network model to learn and add the text feature vector, image feature vector and feature vector after early fusion to achieve late fusion, so as to obtain the final fusion result;

[0033] Classification module: Construct a classifier to learn and classify the final fused feature vector, and divide it into five categories: natural disasters, accidental disasters, public health, social security, and non-emergency events.

[0034] In the above technical solution, the specific implementation of the data acquisition and preprocessing module is:

[0035] A data set was constructed to obtain emergency information from the official website of emergencies, including event title, release time, event text, event source, and corresponding image data in the event. In terms of data preprocessing, two modal data containing text and corresponding images were retained, and the ratio of text and image data samples was 1:1. At the same time, irrelevant content such as emoticons and emojis were removed, and images with low resolution were deduplicated and deleted. Event sample data were denoised and encoded. According to the GB / T35561-2017 Classification and Coding of Emergency Events issued by the state, the collected graphic data were labeled by category. According to the occurrence process, nature and mechanism of emergencies, emergencies were divided into five categories (natural disasters, accidental disasters, public health, social security, and non-emergency events).

[0036] In the above technical solution, the specific implementation of the multimodal feature extraction module is:

[0037] The text feature extractor adopts the BERT model and uses the BERT-base with 16 self-attention heads to extract text features. First, the emergency text is input for word embedding operations, including character embedding, segment embedding and position embedding. Among them, the character embedding converts the text into a vector representation of characters through a lookup table, and marks the beginning and end of the text with [CLS] and [SEP]; segment embedding marks the global semantic information of the text and merges it with the character-level vector; position embedding marks the context information and outputs the final word embedding representation; secondly, the word embedding representation result is input into the Encoder layer for self-attention processing and feedforward neural network calculation, and the emergency text feature vector is output after normalization, and the feature vector dimension is 768; the image feature extractor adopts the ViT model, and the input image size is adjusted to 224 ×224; divide the image into patches of fixed size, the patch size is 16×16, then each image will generate 224×224 / 16×16=19 patches, that is, the length of the input sequence is 196, and the dimension of each patch is 16×16×3=768; after a linear projection layer with a dimension of 768×N (N=768), 196 tokens are output, and the dimension of each token is 768; add a special character CLS, and the sequence dimension is 197×768; consider the position relationship between each patch of the image and add position encoding, use the multi-head attention mechanism to mine the relationship between different patches, and perform image classification, extract the feature vector input by the last classification layer and use it as the image feature representation result, the extracted image feature vector dimension is 768.

[0038] In the above technical solution, the specific implementation of the multimodal feature fusion module is:

[0039] A multimodal feature fusion module is constructed to fuse the extracted text feature vector and image feature vector. The multimodal feature fusion module uses a multi-head self-attention mechanism. The query comes from one modality, while the key and value come from another modality. That is, if Q comes from text, K and V come from attached images, then the attention value calculated using Q and K can be used as a similarity measure between text and image, and then the image is weighted, and vice versa; cross-fusion operations are performed on text features and image features respectively, and the fusion results obtained by the two multi-head self-attention multimodal feature fusion modules are added to fully capture the interdependence between different modalities. , thus obtaining the early feature fusion result; for the obtained text feature vector, image feature vector and feature vector after early fusion, a feedforward neural network model is constructed, in which there is 1 input layer, 1 output layer and 3 hidden layers; the 3 hidden layers contain 1024, 512 and 256 neurons respectively, and the output layer contains 128 neurons; at the same time, PReLU is used as the activation function after each hidden layer, and the network is used to learn the three feature vectors respectively to obtain feature vectors with dimensions of 128, and then the three are added and fused to achieve late fusion, so as to obtain the final fusion result.

[0040] In the above technical solution, the specific implementation of the classification module is:

[0041] The classifier is a fully connected layer constructed by an input layer and an output layer. The input layer contains 128 neurons and the output layer contains 5 neurons. Sigmoid is used as the activation function of the fully connected layer to obtain a feature vector with a dimension of 5. Finally, the softmax layer is used to classify and predict the obtained 5-dimensional feature vector to obtain the final classification result.

[0042] A device for classifying emergency events based on multimodal feature fusion, wherein the device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the steps of the method or system for classifying emergency events based on multimodal feature fusion described in any one of the above technical solutions are implemented.

[0043] Compared with the prior art, the present invention provides a method, system and device for emergency event classification based on multimodal feature fusion, which has the following beneficial effects and advantages:

[0044] 1. The present invention provides a method for classifying emergencies based on multimodal feature fusion, which characterizes the different features of emergencies from two perspectives: text modality and image modality.

[0045] 2. The present invention constructs a multimodal feature fusion module based on a multi-head self-attention mechanism to achieve early fusion of different modal features, so that different modal data can be integrated and supplemented with each other to improve the accuracy and richness of information contained in emergencies. Compared with existing methods, it can better capture the interdependence between different modalities.

[0046] 3. The present invention combines early fusion with late fusion, constructs a multimodal feature fusion module based on a multi-head self-attention mechanism to achieve early fusion, and constructs a feedforward neural network model to learn and add text feature vectors, image feature vectors, and feature vectors after early fusion to achieve late fusion. This fusion method solves the problem that there are differences in the recognition effects of different event categories corresponding to data features of different modalities, and can give play to the advantages of local classifiers. Classification is performed according to the final fused feature representation, so as to obtain better classification results. Compared with the existing methods, combining the two fusion methods can more effectively give play to the dependency and complementarity between different modal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the process provided by the present invention.

[0048] Figure 2 It is a structural schematic diagram of the feedforward neural network model provided by the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, an example is now given to illustrate the specific steps as follows:

[0050] This embodiment provides a method for emergency event classification based on multimodal feature fusion, comprising the following steps:

[0051] Step 1: By obtaining the text data and related image data of emergency news from the official network of emergencies, the collected image and text data are divided into five categories: natural disasters, accidents and disasters, public health, social security, and non-emergency events according to the "GB / T 35561-2017 Classification and Coding of Emergency Events" specification issued by the state, and a sample dataset of emergency events represented by multimodal data is constructed and preprocessed;

[0052] Step 2: Use the BERT model and the ViT model to extract the features of text modality data and image modality data from the samples respectively;

[0053] Step 3: Use the multi-head self-attention mechanism to build a multimodal feature fusion module to perform early fusion of the extracted text features and image features;

[0054] Step 4: Based on the early fusion feature vector obtained in step 3, a feedforward neural network model is constructed to learn the text feature vector, image feature vector and feature vector after early fusion to obtain feature vectors with three dimensions of 128 each. The three are added and fused to obtain the feature vector after late fusion, i.e. the final feature fusion result;

[0055] Step 5: Construct a classifier to learn and classify the final fused feature vector, and divide it into five categories: natural disasters, accidental disasters, public health, social security, and non-emergency events.

[0056] In this embodiment, the specific steps of acquiring and preprocessing data in step 1 are:

[0057] A data set was constructed to obtain emergency information from the official website of emergencies, including event title, release time, event text, event source, and corresponding image data in the event. In terms of data preprocessing, two modal data containing text and corresponding images were retained, and the ratio of text and image data samples was 1:1. At the same time, irrelevant content such as emoticons and emojis were removed, and images with low resolution were deduplicated and deleted. Event sample data were denoised and encoded. According to the GB / T35561-2017 Classification and Coding of Emergency Events issued by the state, the collected graphic data were labeled by category. According to the occurrence process, nature and mechanism of emergencies, emergencies were divided into five categories (natural disasters, accidental disasters, public health, social security, and non-emergency events).

[0058] In this embodiment, the specific steps of extracting features from different modal data in step 2 are:

[0059] The text feature extractor adopts the BERT model and uses the BERT-base with 16 self-attention heads to extract text features. First, the emergency text is input for word embedding operations, including character embedding, segment embedding and position embedding. Among them, the character embedding converts the text into a vector representation of characters through a lookup table, and marks the beginning and end of the text with [CLS] and [SEP]; segment embedding marks the global semantic information of the text and merges it with the character-level vector; position embedding marks the context information and outputs the final word embedding representation; secondly, the word embedding representation result is input into the Encoder layer for self-attention processing and feedforward neural network calculation, and the emergency text feature vector is output after normalization, and the feature vector dimension is 768; the image feature extractor adopts the ViT model, and the input image size is adjusted to 224 ×224; divide the image into patches of fixed size, the patch size is 16×16, then each image will generate 224×224 / 16×16=19 patches, that is, the length of the input sequence is 196, and the dimension of each patch is 16×16×3=768; after a linear projection layer with a dimension of 768×N (N=768), 196 tokens are output, and the dimension of each token is 768; add a special character CLS, and the sequence dimension is 197×768; consider the position relationship between each patch of the image and add position encoding, use the multi-head attention mechanism to mine the relationship between different patches, and perform image classification, extract the feature vector input by the last classification layer and use it as the image feature representation result, the extracted image feature vector dimension is 768.

[0060] In this embodiment, the specific steps of step 3 are:

[0061] Early fusion of feature vectors of different modalities is performed, and a multimodal feature fusion module is constructed to fuse the extracted text feature vectors and image feature vectors. The multimodal feature fusion module uses a multi-head self-attention mechanism. The query comes from one modality, while the key and value come from another modality. That is, if Q comes from text, K and V come from attached images, then the attention value calculated using Q and K can be used as a similarity measure between text and image, and then the image is weighted, and vice versa. Cross-fusion operations are performed on text features and image features respectively, and the fusion results obtained by the two multi-head self-attention multimodal feature fusion modules are added to fully capture the interdependence between different modalities, thereby obtaining early feature fusion results.

[0062] In this embodiment, the specific steps of step 4 are:

[0063] For the text feature vectors, image feature vectors and feature vectors after early fusion obtained in steps 2 and 3, a feedforward neural network model is constructed, which has 1 input layer, 1 output layer and 3 hidden layers; the 3 hidden layers contain 1024, 512 and 256 neurons respectively, and the output layer contains 128 neurons; at the same time, PReLU is used as the activation function after each hidden layer, and the three feature vectors are learned separately by this network to obtain feature vectors with dimensions of 128, and then the three are added and fused to obtain the late fusion result, that is, the final fusion result.

[0064] In this embodiment, the specific steps of step 5 are:

[0065] The classifier is a fully connected layer constructed by an input layer and an output layer. The input layer contains 128 neurons and the output layer contains 5 neurons. Sigmoid is used as the activation function of the fully connected layer to obtain a feature vector with a dimension of 5. Finally, the softmax layer is used to classify and predict the obtained 5-dimensional feature vector to obtain the final classification result.

[0066] This embodiment provides an emergency classification system based on multimodal feature fusion, including the following modules:

[0067] Data acquisition and preprocessing module: used to acquire and process multimodal emergency event data, encode sample data, and classify the collected graphic data into five categories: natural disasters, accidents and disasters, public health, social security, and non-emergency events according to the "GB / T 35561-2017 Classification and Coding of Emergency Events" specification issued by the state;

[0068] Multimodal feature extraction module: used to extract features from the processed multimodal emergency event data, using the BERT model and the ViT model to extract text features and image features respectively;

[0069] Multimodal feature fusion module: used to fuse the extracted feature vectors of different modalities, build a multimodal feature fusion module based on multi-head self-attention mechanism to perform early fusion of text and image features, and then build a feedforward neural network model to learn and add the text feature vector, image feature vector and feature vector after early fusion to achieve late fusion, so as to obtain the final fusion result;

[0070] Classification module: Construct a classifier to learn and classify the final fused feature vector, and divide it into five categories: natural disasters, accidental disasters, public health, social security, and non-emergency events.

[0071] In this embodiment, the data acquisition and preprocessing module is specifically implemented as follows:

[0072] A data set was constructed to obtain emergency information from the official website of emergencies, including event title, release time, event text, event source, and corresponding image data in the event. In terms of data preprocessing, two modal data containing text and corresponding images were retained, and the ratio of text and image data samples was 1:1. At the same time, irrelevant content such as emoticons and emojis were removed, and images with low resolution were deduplicated and deleted. Event sample data were denoised and encoded. According to the GB / T35561-2017 Classification and Coding of Emergency Events issued by the state, the collected graphic data were labeled by category. According to the occurrence process, nature and mechanism of emergencies, emergencies were divided into five categories (natural disasters, accidental disasters, public health, social security, and non-emergency events).

[0073] In this embodiment, the multimodal feature extraction module is specifically implemented as follows:

[0074] The text feature extractor adopts the BERT model and uses the BERT-base with 16 self-attention heads to extract text features. First, the emergency text is input for word embedding operations, including character embedding, segment embedding and position embedding. Among them, the character embedding converts the text into a vector representation of characters through a lookup table, and marks the beginning and end of the text with [CLS] and [SEP]; segment embedding marks the global semantic information of the text and merges it with the character-level vector; position embedding marks the context information and outputs the final word embedding representation; secondly, the word embedding representation result is input into the Encoder layer for self-attention processing and feedforward neural network calculation, and the emergency text feature vector is output after normalization, and the feature vector dimension is 768; the image feature extractor adopts the ViT model, and the input image size is adjusted to 224 ×224; divide the image into patches of fixed size, the patch size is 16×16, then each image will generate 224×224 / 16×16=19 patches, that is, the length of the input sequence is 196, and the dimension of each patch is 16×16×3=768; after a linear projection layer with a dimension of 768×N (N=768), 196 tokens are output, and the dimension of each token is 768; add a special character CLS, and the sequence dimension is 197×768; consider the position relationship between each patch of the image and add position encoding, use the multi-head attention mechanism to mine the relationship between different patches, and perform image classification, extract the feature vector input by the last classification layer and use it as the image feature representation result, the extracted image feature vector dimension is 768.

[0075] In this embodiment, the multimodal feature fusion module is specifically implemented as follows:

[0076] A multimodal feature fusion module is constructed to fuse the extracted text feature vector and image feature vector. The multimodal feature fusion module uses a multi-head self-attention mechanism. The query comes from one modality, while the key and value come from another modality. That is, if Q comes from text, K and V come from attached images, then the attention value calculated using Q and K can be used as a similarity measure between text and image, and then the image is weighted, and vice versa; cross-fusion operations are performed on text features and image features respectively, and the fusion results obtained by the two multi-head self-attention multimodal feature fusion modules are added to fully capture the interdependence between different modalities. , thus obtaining the early feature fusion result; for the obtained text feature vector, image feature vector and feature vector after early fusion, a feedforward neural network model is constructed, in which there is 1 input layer, 1 output layer and 3 hidden layers; the 3 hidden layers contain 1024, 512 and 256 neurons respectively, and the output layer contains 128 neurons; at the same time, PReLU is used as the activation function after each hidden layer, and the network is used to learn the three feature vectors respectively to obtain feature vectors with dimensions of 128, and then the three are added and fused to achieve late fusion, so as to obtain the final fusion result.

[0077] In this embodiment, the classification module is specifically implemented as follows:

[0078] The classifier is a fully connected layer constructed by an input layer and an output layer. The input layer contains 128 neurons and the output layer contains 5 neurons. Sigmoid is used as the activation function of the fully connected layer to obtain a feature vector with a dimension of 5. Finally, the softmax layer is used to classify and predict the obtained 5-dimensional feature vector to obtain the final classification result.

[0079] The specific implementation process of the system described in this embodiment includes the following steps:

[0080] Step 1: Construct a data set. Since social media platforms have corresponding management for the authorization of user data, before obtaining data, you need to apply for an API developer account in accordance with the official website regulations, thereby obtaining the user serial number, user password, access serial number and access password, and then obtain data through Python crawlers. According to the "GB / T 35561-2017 Classification and Coding of Emergency Events" issued by the state, the collected graphic data are categorized and labeled. According to the occurrence process, nature and mechanism of emergencies, emergencies are divided into five categories (natural disasters, accidents and disasters, public health, social security, and other emergencies). A total of 15,181 data samples were obtained, which were divided into training sets and test sets in a ratio of 8:2;

[0081] Step 2: Preprocess the samples in the dataset, including the following steps:

[0082] Step 2.1: Filter, deduplicate and reduce noise of text data, and remove abnormal data collected, such as garbled text or images that cannot be displayed normally. For special language symbols such as emoticons, use the string "emoticon" to replace the text;

[0083] Step 2.2: De-duplicate the image data. Use the approximate duplicate image detection algorithm based on local perceptual hashing (LSH) to delete duplicate images in the dataset. Use the image clarity enhancement model provided by Baidu Smart Cloud to remove noise in the image, optimize the image texture, and improve the image clarity. For images with smaller sizes and images with all text in the dataset, use manual verification to delete them.

[0084] Step 2.3: Encode the data, match the text and image one by one, and annotate the corresponding data of the two modalities with the same code;

[0085] Step 3: Extract features from text and image data. The text data is extracted and learned using the BERT model, and the image features are extracted and learned using the ViT model.

[0086] Step 4: Fusion of the two modal features obtained in step 3, including the following steps:

[0087] Step 4.1: Construct a multimodal feature fusion module based on a multi-head self-attention mechanism to fuse the extracted text feature vector and image feature vector to obtain an early fusion result;

[0088] Step 4.2: Construct a feedforward neural network model, learn and add the text feature vector, image feature vector and feature vector after early fusion to achieve late fusion, so as to obtain the final fusion result;

[0089] Step 5: Input the multimodal feature vector obtained by fusion in step 4 into the constructed classifier, learn and classify it into five categories: natural disasters, accidental disasters, public health, social security, and non-emergency events, and output the final classification results.

[0090] Based on the above embodiments, the present invention also provides an emergency event classification device based on multimodal feature fusion, wherein the device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. At the same time, when the processor executes the computer program, the steps of the emergency event classification method or system based on multimodal feature fusion described in any one of the above embodiments are implemented.

[0091] Compared with the prior art, the present invention provides a method, system and device for emergency event classification based on multimodal feature fusion, which has the following beneficial effects and advantages:

[0092] 1. The present invention provides an emergency classification method based on multimodal feature fusion, which characterizes the different characteristics of emergencies from two perspectives: text modality and image modality.

[0093] 2. The present invention constructs a multimodal feature fusion module based on a multi-head self-attention mechanism to achieve early fusion of different modal features, so that different modal data can be integrated and supplemented with each other to improve the accuracy and richness of information contained in emergencies. Compared with existing methods, it can better capture the interdependence between different modalities.

[0094] 3. The present invention combines early fusion with late fusion, constructs a multimodal feature fusion module based on a multi-head self-attention mechanism to achieve early fusion, and constructs a feedforward neural network model to learn and add text feature vectors, image feature vectors, and feature vectors after early fusion to achieve late fusion. This fusion method solves the problem that there are differences in the recognition effects of different event categories corresponding to data features of different modalities, and can give play to the advantages of local classifiers. Classification is performed according to the final fused feature representation, so as to obtain better classification results. Compared with the existing methods, combining the two fusion methods can more effectively give play to the dependency and complementarity between different modal features.

Claims

1. A method, system and device for emergency event classification based on multimodal feature fusion, characterized in that: The following steps are involved: Step 1: By obtaining the text data and related image data of emergency news from the official website of emergencies, the collected graphic data are divided into five categories: natural disasters, accidents and disasters, public health, social security, and non-emergency events according to the "GB / T 35561-2017 Classification and Coding of Emergency Events" specification issued by the state, and an emergency sample data set represented by multimodal data is constructed and preprocessed; Step 2: Use the BERT model and the ViT model to extract the features of text modality data and image modality data from the samples respectively; Step 3: Use the multi-head self-attention mechanism to build a multimodal feature fusion module to perform early fusion of the extracted text features and image features; Step 4: Based on the early fusion feature vector obtained in step 3, a feedforward neural network model is constructed to learn the text feature vector, image feature vector and feature vector after early fusion to obtain feature vectors with three dimensions of 128 each. The three are added and fused to obtain the feature vector after late fusion, i.e. the final feature fusion result; Step 5: Construct a classifier to learn and classify the final fused feature vector, and divide it into five categories: natural disasters, accidental disasters, public health, social security, and non-emergency events.

2. The emergency event classification method based on multimodal feature fusion according to claim 1 is characterized in that: The step 1 comprises: constructing a data set, obtaining emergency information including event title, release time, event text, event source and corresponding image data in the event from the official website of the emergency; in terms of data preprocessing, retaining two modal data containing text and corresponding images, with the ratio of text and image data samples being 1:1, while removing irrelevant content such as emoticons and emoticons, deduplicating and deleting images with low resolution, and performing noise reduction and encoding processing on the event sample data; labeling the collected graphic data according to the GB / T 35561-2017 Classification and Coding of Emergency Events issued by the state; and classifying the emergency events into five categories (natural disasters, accidental disasters, public health, social security, and non-emergency events) according to the occurrence process, nature and mechanism of the emergency events.

3. The emergency event classification method based on multimodal feature fusion according to claim 1 is characterized in that: The step 3 includes: early fusion of feature vectors of different modalities, building a multimodal feature fusion module, fusing the extracted text feature vector and image feature vector, the multimodal feature fusion module uses a multi-head self-attention mechanism, the query comes from one modality, and the key and value come from another modality, that is, if Q comes from text, K and V come from additional images, then the attention value calculated using Q and K can be used as a similarity measure between the text and the image, and then the image is weighted, and vice versa; cross-fusion operations are performed on text features and image features respectively, and the fusion results obtained by the two multi-head self-attention multimodal feature fusion modules are added to fully capture the mutual dependence between different modalities, thereby obtaining an early feature fusion result.

4. The emergency event classification method based on multimodal feature fusion according to claim 3 is characterized in that: The step 4 comprises: constructing a feedforward neural network model for the text feature vector, the image feature vector and the feature vector after early fusion obtained in steps 2 and 3, wherein the model comprises one input layer, one output layer and three hidden layers; the three hidden layers respectively comprise 1024, 512 and 256 neurons, and the output layer comprises 128 neurons; at the same time, using PReLU as the activation function after each hidden layer, and using this network to learn the three feature vectors respectively to obtain feature vectors with dimensions of 128, and then adding and fusing the three to obtain a late fusion result, i.e., a final fusion result.

5. The emergency event classification method based on multimodal feature fusion according to claim 4 is characterized in that: The step 5 includes: the classifier is a fully connected layer constructed by an input layer and an output layer, wherein the input layer contains 128 neurons and the output layer contains 5 neurons, and sigmoid is used as the activation function of the fully connected layer to obtain a feature vector with a dimension of 5, and finally a softmax layer is used to perform classification prediction on the obtained 5-dimensional feature vector to obtain the final classification result.

6. An emergency classification system based on multimodal feature fusion, characterized in that: include: Data acquisition and preprocessing module: used to acquire and process multimodal emergency event data, encode sample data, and classify the collected graphic data into five categories: natural disasters, accidents and disasters, public health, social security, and non-emergency events according to the "GB / T 35561-2017 Classification and Coding of Emergency Events" specification issued by the state; Multimodal feature extraction module: used to extract features from the processed multimodal emergency event data, using the BERT model and the ViT model to extract text features and image features respectively; Multimodal feature fusion module: used to fuse the extracted feature vectors of different modalities, build a multimodal feature fusion module based on multi-head self-attention mechanism to perform early fusion of text and image features, and then build a feedforward neural network model to learn and add the text feature vector, image feature vector and feature vector after early fusion to achieve late fusion, so as to obtain the final fusion result; Classification module: Construct a classifier to learn and classify the final fused feature vector, and divide it into five categories: natural disasters, accidental disasters, public health, social security, and non-emergency events.

7. An emergency event classification device based on multimodal feature fusion, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the emergency event classification method or system based on multimodal feature fusion described in any one of claims 1 to 6 are implemented.

Images