Open world event detection method based on hybrid universal domain adaptive algorithm

Abstract

The invention discloses an open world event detection method based on a hybrid universal domain adaptive algorithm. The open world event detection method comprises the following steps: (1) acquiring open world event data; carrying out feature extraction to obtain source domain features and target domain features, and carrying out MMoE fusion to obtain multi-modal features; (2) calculating to obtain a source domain prototype; (3) calculating cosine similarity of the target domain features and the source domain prototype to obtain a similarity matrix, and then inputting the similarity matrix into an unbalanced optimal transmission formula for solving; (4) selecting samples of a common class; (5) taking the maximum value of each row of the similarity matrix, and comparing the maximum value with the fuzzy boundary; (6) calculating a common class loss function, and calculating a reconstruction loss function; (7) calculating a weighted clustering network loss function, calculating a sub-clustering network loss function and a target domain loss function, and calculating a contrast learning loss function to obtain a total training loss function; (8) obtaining an effective event detection effect; according to the invention, unknown category data can be detected.

Description

Technical Field

[0001] The present invention relates to the technical field of event detection, and in particular to an open-world event detection method based on a hybrid general domain adaptive algorithm. Background Art

[0002] Existing event detection methods can be broadly categorized into unimodal and multimodal event detection. Unimodal event detection primarily focuses on textual information within posts. The existing PP-GCN algorithm innovatively combines weighted meta-path instance similarity with semantic text representation, using graph convolutional networks to accurately classify social events in heterogeneous information networks.

[0003] Most multimodal event detection methods have made great progress in modal fusion. Existing technologies fuse text and image modalities through visual geometry group network 16 (VGG16) and neural network (CNN) to learn joint representations for social event detection. Although these methods can achieve good event detection effects on data of existing event categories, they cannot perform relevant detection and classification on data of newly occurring event categories. Summary of the Invention

[0004] Purpose of the invention: The purpose of the present invention is to provide an open-world event detection method based on a hybrid general domain adaptation algorithm, which solves the problem that current event detection cannot effectively integrate multimodal data information, and solves the problem that unknown category data cannot be identified and detected from newly arrived data.

[0005] Technical solution: The open-world event detection method based on a hybrid general domain adaptation algorithm described in the present invention includes the following steps: (1) Obtaining open world event data including source domain event data and target domain event data; (2) The images and texts corresponding to the source domain event data and the target domain event data are respectively extracted by the feature extractor to obtain source domain features and target domain features, and the features of the two modalities are fused using MMoE to obtain new multimodal features; (3) The source domain features are passed through the classifier to obtain the prediction component, and then the source domain prototype cross entropy is calculated using the source domain label and the classifier parameters to obtain the source domain prototype; (4) The cosine similarity between the target domain features and the source domain prototype is calculated to obtain the similarity matrix, and then the similarity matrix is ​​input into the unbalanced optimal transmission formula to obtain the optimal solution of the allocation matrix.

[0006] (5) Normalize the optimal solution of the allocation matrix to obtain the optimal solution of the normalized allocation matrix. Then, according to the property of the statistical mean, obtain the confidence score of the target sample and the confidence score of the source domain prototype from the optimal solution of the normalized allocation matrix. Then, apply pseudo labels to the target domain samples by the statistical mean method to select samples of the common class. (6) Take the maximum value of each row of the similarity matrix and compare the maximum value with the fuzzy limit; (7) Repeat steps (4)-(5) until high-confidence target domain samples are obtained, and pseudo labels are assigned, and the common class loss function is calculated.

[0007] (8) The target domain features are reduced in dimension by the trained variable autoencoder VAE to obtain new target domain features, and the reconstruction loss function is calculated based on the reconstructed features; (9) The target domain features after dimensionality reduction are fed into the clustering network to obtain cluster soft assignments; then the sub-network corresponding to each cluster uses the data corresponding to the cluster to generate the corresponding sub-cluster soft assignments; the soft assignments are obtained using the YES Gaussian mixture model GMM; and the KL divergence is used to calculate the weighted clustering network loss function. ; Calculate the sub-clustering network loss function using sub-cluster soft assignment; Obtain the target domain loss function using the weighted clustering network loss function and the sub-clustering network loss function; (10) Compare the neighboring samples of the cluster corresponding to the target domain features with the corresponding cluster prototypes and the target domain prototypes to calculate the comparative learning loss function; (11) The total training loss function is obtained using the weighted source domain prototype cross entropy, the common class loss function, the target domain loss function, the reconstruction loss function and the contrastive learning loss function; (12) Input the target domain data into the clustering model to obtain the label and the corresponding target domain data to update the queue; (13) Repeat steps (2)-(12) until the model converges and stabilizes, and effective event detection results are obtained.

[0008] Furthermore, step (2) is as follows: the source domain event data and target domain events The corresponding picture and text The corresponding features are extracted by feature extractors respectively, and the masked autoencoder is used for image modality data. The formula is as follows: ; in is a masked autoencoder model, is the image data, is the image feature; For text, the BERT model is used, and the formula is as follows: ; in, is the text feature extractor model, is text data, It is a text feature; Text features and image features Splicing, let the spliced ​​feature be x, and obtain the multi-task relationship in the data through the new MmoE, iMMoE, to obtain the multimodal feature after fusion , the calculation formula is as follows: ; in, is a hyperparameter, is the number of downstream tasks, is the output of the i-th expert, is the kth output of the i-th gate, t is the number of tokens, represents the token attention for task k.

[0009] Furthermore, step (3) is as follows: the source domain features Through the prototype-based classifier: obtain the prediction component, and then use the source domain label and the classifier parameters to calculate the source domain prototype cross entropy , get the source domain prototype ,in is the number of categories of the source domain prototype, It is A source domain prototype.

[0010] Furthermore, step (4) is as follows: the target domain features and source domain prototype Calculate cosine similarity to obtain similarity matrix , then input the similarity matrix into the unbalanced optimal transmission formula to solve and obtain the optimal solution of the allocation matrix ; Among them, the solution formula for the unbalanced optimal transmission formula is as follows: ; in, is the optimal allocation matrix to be solved, is the similarity matrix, is the matrix trace calculation, is a constant greater than zero, is the entropy regularization term, It is a soft penalty item. is the KL divergence, and is a simple shape vector; in solving When the similarity matrix is , yes , yes ; where B is the batch size, It is a vector of dimension B, where all elements are 1; is the number of source domain prototypes, The dimension is , a vector whose elements are all 1.

[0011] Furthermore, step (5) is as follows: the optimal solution of the allocation matrix Normalize to get the optimal solution of the normalized allocation matrix , and then according to the properties of the statistical mean, the optimal solution is obtained from the normalized distribution matrix Get the confidence score of the target sample and the confidence score of the source domain prototype , the calculation formula is as follows: ; ; Then, pseudo labels are applied to the target domain samples by using the statistical mean method. Select samples of the common class; when and When =1, it is considered to be a public sample with high confidence; is the number of source domain prototypes, and B is the batch size.

[0012] Furthermore, step (6) is specifically as follows: samples above the fuzzy limit are considered to be positive samples, and samples below the fuzzy limit are considered to be negative samples; then an adaptive filling mechanism is used to fill in the samples so that the samples in the target domain are balanced, that is, when there are many positive samples, negative samples are synthesized in the form of mixed feature vectors for filling; when there are many negative samples, the common class samples selected in step (5) are used as positive samples for filling.

[0013] Furthermore, the calculation formula of step (7) is as follows: ; in, is the i-th target domain sample, is the pseudo label of the i-th target domain sample, is the cross entropy loss function.

[0014] Furthermore, step (8) is as follows: the target domain features The new target domain features are obtained by reducing the dimension through the trained variable autoencoder VAE , and the target domain features Features of reconstruction Calculate the reconstruction loss function The calculation formula is as follows: ; Where B is the batch size, is the dimension of the input data, is the dimension after dimensionality reduction, is the original input data, is the reconstructed output data, is the mean of the d-th dimension of the n-th sample, is the variance of the dth dimension of the nth sample.

[0015] Furthermore, step (9) is as follows: the target domain features after dimension reduction are fed into the clustering network In the cluster soft assignment , the calculation formula is as follows: ; in is the input data, is the soft assignment that represents each target sample being assigned to cluster k; Then the sub-network corresponding to each cluster Use the data corresponding to this cluster Generate corresponding subclass cluster soft allocation , the calculation formula is as follows: ; in, is the sample data belonging to each cluster k, is the soft allocation for each cluster class; Reuse cluster soft allocation To learn the soft distribution obtained in the Yes-Gaussian mixture model GMM , use KL divergence to calculate the clustering network loss function , the formula is as follows: ; Using sub-clustering network Obtained Calculate the sub-clustering network loss function , the formula is as follows: ; in, is the target sample, is the average value of sub-cluster j in cluster k, Represents the L2 paradigm; then the weighted clustering network loss function is used and sub-clustering network loss function Get the target domain loss function .

[0016] Furthermore, in step (10), the contrastive learning loss function , the formula is as follows: ; in is the temperature parameter.

[0017] Beneficial Effects: Compared with the existing technology, the present invention has the following significant advantages: The new MMoE is used to learn cross-modal semantic relationships between multimodal features. Deep clustering is used to simulate the distribution of learning target domain features, making it possible to learn the target domain representation without relying on prior knowledge. In order to improve the global distinguishability and local consistency of the target domain representation, the sample itself, its neighbors, and the prototype that matches it are placed in the embedding space, and finally, representation learning is performed through contrastive learning to improve the performance of the model in the target domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the present invention; Figure 2 It is the multimodal feature fusion module of the present invention. DETAILED DESCRIPTION

[0019] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0020] like Figure 1 As shown, an embodiment of the present invention provides an open world event detection method based on a hybrid general domain adaptive algorithm, comprising the following steps: S1: Construct an open-world event dataset. One dataset is source data, which is past data, and the other is target data, which is newly arrived data. The source data has corresponding category labels, while the target data does not. The target data contains both event categories that have appeared in the source domain and event categories that have not appeared in the source domain. Each open-world event dataset includes at least two matching open-world event modal data.

[0021] S2, the source domain event data and target domain events The corresponding picture and text The corresponding features are extracted by feature extractors, such as Figure 2 As shown, for image modality data, a masked autoencoder is used. The specific calculation formula is as follows: ; in is a masked autoencoder model, is the image data, It is a picture feature.

[0022] For text modality, the BERT model is used, and the formula is as follows: ; in is the text feature extractor model, is text data, It is a text feature.

[0023] Text features and image features Splice them together, assuming that the spliced ​​features are , through the new MMoE (iMMoE) to obtain the multi-task relationship in the data to obtain the multimodal features after fusion , the calculation formula is as follows: ; in is a hyperparameter, is the number of downstream tasks, is the output of the i-th expert, is the kth output of the i-th gate, t is the number of tokens, represents the token attention for task k.

[0024] S3, the source domain features The prediction component is obtained through a prototype-based classifier (i.e., only one fully connected layer), and then the source domain label and the classifier parameters are used to calculate the source domain prototype cross entropy , get the source domain prototype ,in is the number of categories of the source domain prototype, It is A source domain prototype.

[0025] S4, the target domain features and source domain prototype Calculate cosine similarity to obtain similarity matrix , then input the similarity matrix into the unbalanced optimal transmission formula to solve and obtain the optimal solution of the allocation matrix , the solution formula for the unbalanced optimal transmission formula is as follows: ; in, is the optimal allocation matrix to be solved, is the similarity matrix, is the matrix trace calculation, is a constant greater than zero, is the entropy regularization term, It is a soft penalty item. is the KL divergence, and Is a simple shape vector. When the similarity matrix is , yes , yes . Where B is the batch size, It is a vector of dimension B, where all elements are 1. is the number of source domain prototypes, The dimension is , a vector whose elements are all 1.

[0026] S5, allocate the optimal solution of the matrix Normalize to get the optimal solution of the normalized allocation matrix , and then according to the properties of the statistical mean, the optimal solution is obtained from the normalized distribution matrix Get the confidence score of the target sample and the confidence score of the source domain prototype , the calculation formula is as follows: ; ; Then, pseudo labels are applied to the target domain samples by using the statistical mean method. Select samples of the common class.

[0027] Only when and When =1, it is considered to be a public sample with high confidence. is the number of source domain prototypes, and B is the batch size.

[0028] S6, similarity matrix Take the maximum value for each row of Compare. Set to 50%, above this limit Considered to be a positive sample, below the limit If the target domain sample is large, it is considered a negative sample. To balance the samples in the target domain, an adaptive padding mechanism is used. When there are many positive samples, negative samples are synthesized using a mixed feature vector for padding. This means finding the source domain prototype with the minimum similarity to the target sample and synthesizing this minimum similarity with the target sample. When there are many negative samples, the common class samples selected in step S5 are used as positive samples for padding.

[0029] S7: After filling until the samples are balanced, repeat steps S4-S5 to obtain high-confidence target domain samples and assign pseudo labels , calculate the common class loss function The calculation formula is as follows: ; in is the i-th target domain sample, is the pseudo label of the i-th target domain sample, is the cross entropy loss function.

[0030] S8, the target domain features The new target domain features are obtained by reducing the dimension through the trained variable autoencoder VAE , and the target domain features Features of reconstruction Calculate the reconstruction loss function The calculation formula is as follows: ; Where B is the batch size, is the dimension of the input data, is the dimension after dimensionality reduction, is the original input data, is the reconstructed output data, is the mean of the d-th dimension of the n-th sample, is the variance of the dth dimension of the nth sample.

[0031] This step removes redundant dimensions from high-dimensional features and extracts effective dimensional features.

[0032] S9, send the target domain features after dimension reduction into the clustering network In the cluster soft assignment The calculation formula is as follows:

[0033] in is the input data, is the soft assignment that represents each target sample being assigned to cluster k.

[0034] Then the sub-network corresponding to each cluster Use the data corresponding to this cluster Generate corresponding subclass cluster soft allocation The calculation formula is as follows:

[0035] in is the sample data belonging to each cluster k, is the soft assignment for each cluster class.

[0036] Using cluster soft assignment To learn the soft distribution obtained from the Yes-Gaussian mixture model (GMM) , use KL divergence to calculate the clustering network loss function .

[0037]

[0038] Using sub-clustering network Obtained To calculate the sub-clustering network loss function ,

[0039] in is the target sample, is the average value of sub-cluster j in cluster k, Represents the L2 paradigm. After the soft assignment is obtained through the clustering network, the target domain prototype is obtained. However, the number of target domain prototypes will continue to split / aggregate each time. If the two sub-clusters of the target domain prototype are relatively independent and of equal size, they will be divided according to The probability of accepting the split, The calculation formula is as follows:

[0040] in is the gamma function, is the target sample of each cluster, , is the marginal likelihood. The same is true for merging, with The probability of accepting to be merged.

[0041] The final weighted clustering network loss function and sub-clustering network loss function Get the target domain loss function .

[0042] S10, in order to make the samples of the target domain more globally distinguishable and locally consistent, the contrast loss method is used for integration. The target domain data is input into the clustering model to obtain the label and the corresponding target domain data are added to the memory queue. In order to achieve local consistency, the target domain features are taken out from the memory queue. and the neighboring samples of the corresponding cluster , the corresponding cluster prototype Considered as a positive sample, the two are combined to obtain , in order to achieve global discriminability, the target sample will not be matched Cluster prototype Other samples are considered negative samples. , calculate the contrastive learning loss function The formula is as follows:

[0043] in is the temperature parameter.

[0044] S11, using weighted source domain prototype cross entropy , public class loss function , target domain loss function , reconstruction loss function and contrastive learning loss function Get the total training loss function .

[0045] ; and are the weight parameters that control the loss function training, which are 0.1 and 0.5 respectively, based on the total training loss function Backpropagation updates the gradient parameters.

[0046] S12: Input the target domain data into the clustering model, and update the labels and corresponding target domain data into the memory queue.

[0047] S13, repeating steps S2-S12 until the model converges and stabilizes, and inputting the target domain data into the open world event detection model based on the hybrid general domain adaptation algorithm to obtain effective event detection results.

[0048] The present invention selects 4 related models for comparison with the model of the present invention. The model of the present invention is applied to the CrisisMMD dataset. The CrisisMMD multimodal dataset includes 16097 text information and 18126 image information, which contains 7 categories in total. Several categories are regarded as data from the past as known categories, and the other categories are regarded as new categories that have not been discovered in the past as unknown categories. The four methods are quantitatively compared using three evaluation indicators: Know ACC known class accuracy, Unkown ACC unknown class accuracy and mutual information NMI.

[0049] Table 1 Performance of different models on the CrisisMMD dataset

[0050] HCRC and HISEvent are event detection methods. Although they can detect unknown events in an unsupervised manner, they are limited to events with text information and fail to integrate event detection with modal information such as images. VGG16+CNN and SCBD are multimodal event detection methods. Although they effectively combine multiple modal information for further fusion, they cannot detect new events of unknown categories. As shown in Table 1, when the number of unknown categories in the target data varies, the results show that the proposed model outperforms other models on the CrisisMMD dataset, demonstrating better open-world event detection capabilities.

Claims

1. A method for open world event detection based on a hybrid general domain adaptation algorithm, characterized in that: The following steps are involved: (1) Obtaining open world event data including source domain event data and target domain event data; (2) The images and texts corresponding to the source domain event data and the target domain event data are respectively extracted by the feature extractor to obtain source domain features and target domain features, and the features of the two modalities are fused using MMoE to obtain new multimodal features; (3) The source domain features are passed through the classifier to obtain the prediction component, and then the source domain prototype cross entropy is calculated using the source domain label and the classifier parameters to obtain the source domain prototype; (4) Calculate the cosine similarity between the target domain features and the source domain prototypes to obtain a similarity matrix, and then input the similarity matrix into the unbalanced optimal transmission formula to obtain the optimal solution of the allocation matrix; (5) Normalize the optimal solution of the allocation matrix to obtain the optimal solution of the normalized allocation matrix. Then, according to the attribute of the statistical mean, obtain the confidence score of the target sample and the confidence score of the source domain prototype from the optimal solution of the normalized allocation matrix. Then, apply pseudo labels to the target domain samples by the statistical mean method to select samples of the common class. (6) Take the maximum value of each row of the similarity matrix and compare the maximum value with the fuzzy boundary; (7) Repeat steps (4)-(5) until high-confidence target domain samples are obtained, and pseudo labels are assigned, and the common class loss function is calculated; (8) The target domain features are reduced in dimension through the trained variable autoencoder VAE to obtain new target domain features, and the reconstruction loss function is calculated based on the reconstructed features; (9) The target domain features after dimensionality reduction are sent to the clustering network to obtain cluster soft assignment; then the sub-network corresponding to each cluster uses the data corresponding to the cluster to generate the corresponding sub-cluster soft assignment; the soft assignment is obtained using the Yes-Gaussian mixture model GMM; the KL divergence is used to calculate the weighted clustering network loss function; the sub-clustering network loss function is calculated using the sub-cluster soft assignment; the target domain loss function is obtained using the weighted clustering network loss function and the sub-clustering network loss function; (10) Compare the neighboring samples of the cluster corresponding to the target domain features with the corresponding cluster prototypes and the target domain prototypes to calculate the comparative learning loss function; (11) The total training loss function is obtained using the weighted source domain prototype cross entropy, the common class loss function, the target domain loss function, the reconstruction loss function and the contrastive learning loss function; (12) Input the target domain data into the clustering model, obtain the label, and update the corresponding target domain data into the queue; (13) Repeat steps (2)-(12) until the model converges and stabilizes, and an effective event detection effect is obtained.

2. The open world event detection method based on a hybrid general domain adaptive algorithm according to claim 1, characterized in that: Step (2) is as follows: and target domain events The corresponding picture and text The corresponding features are extracted by feature extractors respectively, and the masked autoencoder is used for the image modality data. The formula is as follows: ； in is a masked autoencoder model, is the image data, is the image feature; For text, the BERT model is used, and the formula is as follows: ； in, is the text feature extractor model, is text data, It is a text feature; Text Features and image features Splicing, let the spliced ​​feature be x, and the new MmoE, iMMoE, obtains the multi-task relationship in the data to obtain the fused multimodal feature , the calculation formula is as follows: ； in, is a hyperparameter, is the number of downstream tasks, is the output of the ith expert, is the kth output of the ith gate, t is the number of tokens, represents the token attention for task k.

3. The open world event detection method based on a hybrid general domain adaptive algorithm according to claim 1, characterized in that: Step (3) is as follows: transform the source domain features Through the prototype-based classifier: obtain the prediction component, and then use the source domain label and the parameters of the classifier to calculate the source domain prototype cross entropy , get the source domain prototype ,in is the number of categories of the source domain prototype, It is A source domain prototype.

4. The open world event detection method based on a hybrid general domain adaptive algorithm according to claim 1, characterized in that: Step (4) is as follows: and source domain prototype Calculate cosine similarity to obtain similarity matrix Then the similarity matrix is ​​input into the unbalanced optimal transmission formula to obtain the optimal solution of the allocation matrix ; Among them, the solution formula of the unbalanced optimal transmission formula is as follows: ； in, is the optimal allocation matrix to be solved, is the similarity matrix, is the matrix trace calculation, is a constant greater than zero, is the entropy regularization term, is a soft penalty item. is the KL divergence, and is a simple shape vector; in solving When the similarity matrix is , yes , yes ; where B is the batch size, is a vector of dimension B, where all elements are 1; is the number of source domain prototypes, The dimension is , a vector whose elements are all 1.

5. The open world event detection method based on a hybrid general domain adaptive algorithm according to claim 1, characterized in that: Step (5) is as follows: assign the optimal solution of the allocation matrix Normalize to get the optimal solution of the normalized allocation matrix , and then according to the properties of the statistical mean, the optimal solution is obtained from the normalized allocation matrix Get the confidence score of the target sample and the confidence score of the source domain prototype , the calculation formula is as follows: ； ； Then, the pseudo-label is applied to the target domain samples by using the statistical mean method. Select samples of the common class; when and When =1, it is considered to be a public sample with high confidence; is the number of source domain prototypes and B is the batch size.

6. The open world event detection method based on a hybrid general domain adaptive algorithm according to claim 1, characterized in that: Step (6) is as follows: samples above the fuzzy limit are considered positive samples, and samples below the fuzzy limit are considered negative samples; then an adaptive filling mechanism is used to fill in samples so that the samples in the target domain remain balanced. That is, when there are many positive samples, negative samples are synthesized in the form of mixed feature vectors for filling; when there are many negative samples, the common class samples selected in step (5) are used as positive samples for filling.

7. The open world event detection method based on a hybrid general domain adaptive algorithm according to claim 1, characterized in that: The calculation formula for step (7) is as follows: ； in, is the i-th target domain sample, is the pseudo label of the i-th target domain sample, is the cross entropy loss function.

8. The open world event detection method based on a hybrid general domain adaptive algorithm according to claim 1, characterized in that: Step (8) is as follows: The trained variable autoencoder VAE is used to reduce the dimension and obtain new target domain features. , and the target domain features Features with reconstruction Calculate the reconstruction loss function ; The calculation formula is as follows: ； Where B is the batch size, is the dimension of the input data, is the dimension after dimensionality reduction, is the original input data, is the reconstructed output data, is the mean of the dth dimension of the nth sample, is the variance of the dth dimension of the nth sample.

9. The open world event detection method based on hybrid general domain adaptive algorithm according to claim 1, characterized in that: Step (9) is as follows: Send the reduced-dimensional target domain features into the clustering network In the cluster soft assignment , the calculation formula is as follows: ； in is the input data, is the soft assignment that represents each target sample being assigned to cluster k; Then the subnetwork corresponding to each cluster Use the data corresponding to this cluster Generate corresponding subclass cluster soft allocation , the calculation formula is as follows: ； in, is the sample data belonging to each cluster class k, is the soft allocation for each cluster class; Reuse Cluster Soft Assignment To learn the soft distribution obtained in the Yes-Gaussian mixture model GMM , use KL divergence to calculate the clustering network loss function , the formula is as follows: ； Using sub-clustering networks What you get Calculate the sub-clustering network loss function , the formula is as follows: ； in, is the target sample, is the average value of subcluster j in cluster k, represents the L2 paradigm; then the weighted clustering network loss function is used And sub-clustering network loss function Get the target domain loss function .

10. The open world event detection method based on hybrid general domain adaptive algorithm according to claim 1, characterized in that: In step (10), the contrastive learning loss function , the formula is as follows: ； in is the temperature parameter.

Images