Open domain social event classification methods, devices, electronic equipment and storage media

Abstract

This invention provides an open-domain social event classification method, apparatus, electronic device, and storage medium, relating to the field of artificial intelligence technology. The method includes: acquiring target data of social events; inputting the target data of social events into an open-domain social event classification model, and outputting the classification category corresponding to the target data of social events; wherein the classification category is determined by multimodal fusion features obtained by fusing at least two modal features of the target data of social events, the multimodal fusion features being used to characterize the cross-modal semantic relationships between the various modal features, and the open-domain classification loss function of the open-domain social event classification model being determined based on a hybrid classification loss function, an unknown class focus loss function, and an uncertainty loss function. This invention can achieve open-domain social event classification, including known and unknown classes, based on cross-modal semantic relationships.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an open-domain social event classification method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of the internet and multimedia, people can receive and send social media messages online through various social media platforms. Meanwhile, trending events on social media can spread rapidly around the world through user interactions such as forwarding and commenting. The proliferation of social media events with multiple modalities, such as images, videos, and text, on the internet underscores the importance of automatically mining and summarizing trending social topics from massive amounts of social media data. How to mine and classify social events is also one of the core issues in information recommendation, search engines, and other intelligent systems. Therefore, research on the classification of social events is of great significance.

[0003] In existing technologies, social event classification methods mainly rely on the assumption of a closed world, that is, the tested category must appear in the training process. Social event classification methods generally include: (1) traditional manual methods, which classify social events by manually extracting features. However, manual feature extraction is highly subjective and cannot obtain deep semantic features, resulting in low classification accuracy; (2) deep neural network methods, which simply connect features of different modalities to form multimodal event features and then classify social events. However, they ignore the fine-grained feature information of different modalities and cross-modal semantic relationships, resulting in low classification accuracy. Summary of the Invention

[0004] This invention provides an open-domain social event classification method, apparatus, electronic device, and storage medium to address the shortcomings of low classification accuracy in existing technologies. Based on cross-modal semantic relationships, it enables open-domain social event classification, including known and unknown classes.

[0005] This invention provides an open-domain social event classification method, comprising:

[0006] Acquire target data on social events;

[0007] Input the target data of social events into the open-domain social event classification model, and output the classification category corresponding to the target data of social events;

[0008] The classification category is determined by a multimodal fusion feature obtained by fusing at least two modal features of the social event target data. The multimodal fusion feature is used to characterize the cross-modal semantic relationship between the modal features. The open-domain classification loss function of the open-domain social event classification model is determined based on a hybrid classification loss function, an unknown class focus loss function, and an uncertainty loss function.

[0009] According to the open-domain social event classification method provided by the present invention, the open-domain social event classification model is trained based on the following steps:

[0010] Construct a social event training dataset, wherein each social event training data in the social event training dataset includes at least two matching social event modal data and the modal label corresponding to the social event training data, and the social event training dataset includes training data of known classes and training data of unknown classes;

[0011] For each of the social event training data, determine the modal features corresponding to each of the social event modal data.

[0012] Each modal feature is input into a multimodal masking network to determine the multimodal fusion feature after the fusion of each modal feature;

[0013] Based on the training data of each social event, the labels corresponding to each social event training data, and the multimodal fusion features, the initial open-domain social event classification model is trained under supervision to obtain the open-domain social event classification model.

[0014] According to the open-domain social event classification method provided by the present invention, the unknown class training data is obtained based on the following steps:

[0015] Based on the known training data, determine the mixed sample pool;

[0016] From the mixed sample pool, randomly select any two training data of the known classes without replacement;

[0017] Interchange the social event modal data corresponding to any two of the known class training data to determine the unknown class training data.

[0018] According to the open-domain social event classification method provided by the present invention, determining the multimodal fusion feature after fusing the modal features includes:

[0019] For each of the modal features, determine the modal fusion feature used as the query feature;

[0020] The modal fusion features are fused together to obtain the multimodal fusion features.

[0021] According to the open-domain social event classification method provided by the present invention, determining the modality fusion feature used as the query modality feature includes:

[0022] For each modal feature, a key-value pair corresponding to the modal feature is determined based on a multi-head attention mechanism;

[0023] Based on the mask matrix and the key-value pairs, the modal features are determined as the modal fusion features for querying, and the mask matrix is ​​used to filter redundant information in the modal fusion features.

[0024] According to the open-domain social event classification method provided by the present invention, the mask matrix is ​​determined based on the following steps:

[0025] Determine the semantic similarity between each of the modal features;

[0026] Each semantic similarity is compared with a similarity threshold to determine the comparison result;

[0027] Based on the comparison results, the mask matrix corresponding to each modal feature is determined.

[0028] According to the open-domain social event classification method provided by the present invention, the open-domain classification loss function is determined based on the following steps:

[0029] Determine the known class classification score corresponding to the known class training data and the unknown class classification score corresponding to the unknown class training data, respectively;

[0030] The mixed classification loss function is determined based on the known class classification scores and the unknown class classification scores;

[0031] Based on the classification score of the unknown class, determine the focus loss function for the unknown class;

[0032] Based on the known class classification scores, determine the uncertainty loss function;

[0033] Determine the weights corresponding to the hybrid classification loss function, the unknown class focus loss function, and the uncertainty loss function, respectively.

[0034] Based on the weights, the mixed classification loss function, the unknown class focus loss function, and the uncertainty loss function are weighted and summed to determine the open domain classification loss function.

[0035] The present invention also provides an open-domain social event classification device, comprising:

[0036] The acquisition module is used to acquire target data on social events;

[0037] The output module is used to input the target data of social events into the open domain social event classification model and output the classification category corresponding to the target data of social events.

[0038] The classification category is determined by a multimodal fusion feature obtained by fusing at least two modal features of the social event target data. The multimodal fusion feature is used to characterize the cross-modal semantic relationship between the modal features. The open-domain classification loss function of the open-domain social event classification model is determined based on a hybrid classification loss function, an unknown class focus loss function, and an uncertainty loss function.

[0039] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the open domain social event classification methods described above.

[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the open-domain social event classification method as described above.

[0041] The open-domain social event classification method, apparatus, electronic device, and storage medium provided by this invention acquire social event target data and fuse at least two modal features of the social event target data. By utilizing the fine-grained semantic associations between different modal features, the at least two modal features are fused to obtain multimodal fusion features, thereby obtaining cross-modal semantic relationships between each modal feature. This further determines the classification category corresponding to the social event target data. Different modal features complement and enhance each other to improve classification accuracy. At the same time, the open-domain loss function in the open-domain social event classification model used for classification combines a hybrid classification loss function, an unknown class focus loss function, and an uncertainty loss function to achieve open-domain social event classification for events of known or unknown categories. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the open-domain social event classification method provided by the present invention;

[0044] Figure 2 This is a schematic diagram of the training process of the open-domain social event classification model provided by the present invention;

[0045] Figure 3 This is a schematic diagram of the process for determining multimodal fusion features provided by the present invention;

[0046] Figure 4 This is a schematic diagram of the classification results of the open-domain social event classification method provided by the present invention;

[0047] Figure 5 This is a schematic diagram of the structure of the open-domain social event classification device provided by the present invention;

[0048] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0050] When classifying social events, on the one hand, existing traditional manual feature extraction methods rely heavily on human subjectivity and cannot uncover deep semantic features. Furthermore, multimodal neural networks in deep neural networks often simply connect features from different modalities to form multimodal event features, ignoring fine-grained feature information from different modalities and cross-modal semantic relationships. Both of these feature extraction methods lead to low classification accuracy. On the other hand, existing social event classification methods primarily rely on the assumption of a closed world, meaning the predicted category must have appeared during training. Therefore, when new or unknown categories of events appear in the test, existing social event classification methods cannot identify these new or unknown categories and incorrectly classify them as known categories, thus reducing classification accuracy.

[0051] To address the issue of low classification accuracy mentioned above, embodiments of the present invention provide an open-domain social event classification method. Figure 1 This is a flowchart illustrating the open-domain social event classification method provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0052] Step 110: Obtain target data for social events.

[0053] Optionally, the aforementioned target data for social events can be social event data of different modalities obtained from social media, including: one-to-one corresponding image information and text information, that is, the image information and text information indicate the same social event. For example, the image information is: "Fires rage in the city of San Lorenzo in Argentina's Santa Fe province, an environmental disaster that has been continuing since February," which translates to "Fires rage in the city of San Lorenzo in Argentina's Santa Fe province, an environmental disaster that has been continuing since February." The image information can be obtained by acquiring image data from social media or by segmenting the acquired video data into frames. The text information can be obtained by transcribing text data or audio data into speech, capturing subtitles from video data, etc., and this embodiment of the invention does not limit this.

[0054] It should be noted that the language categories of the above text information include, but are not limited to: Chinese, English, German, Russian, French, Japanese, Italian, Spanish, and Portuguese. This embodiment of the invention does not impose any limitations on these languages.

[0055] Step 120: Input the target data of the social event into the open domain social event classification model, and output the classification category corresponding to the target data of the social event;

[0056] The classification category is determined by a multimodal fusion feature obtained by fusing at least two modal features of the social event target data. The multimodal fusion feature is used to characterize the cross-modal semantic relationship between the modal features. The open-domain classification loss function of the open-domain social event classification model is determined based on a hybrid classification loss function, an unknown class focus loss function, and an uncertainty loss function.

[0057] Specifically, after inputting the target data of social events into the open-domain social event classification model, feature extraction can be performed on the input target data to obtain the corresponding image features and text features. To further capture the cross-modal semantic relationships between different modal features, this embodiment of the invention fuses the acquired image features and text features to obtain multimodal fusion features, thereby achieving mutual complementarity and enhancement between different modal features. Based on the fused multimodal fusion features, the classification category corresponding to the target data of the social event is determined. Simultaneously, this open-domain social event classification model is trained using an open-domain classification loss function determined by a hybrid classification loss function, an unknown class focus loss function, and an uncertainty loss function. The hybrid classification loss function can be cross-entropy loss, making the predicted values ​​of the open-domain social event classification model closer to the true values. The unknown class focus loss function and the uncertainty loss function tilt the prediction focus towards the unknown class training data during training, enabling the trained open-domain social event classification model to classify open-domain social events of known or unknown categories.

[0058] It should be noted that the categories of the aforementioned social event target data can be categories known during the initial training of the open-domain social event classification model, or new event categories that did not appear during training. In practical applications, new event categories are constantly emerging. The goal of open-domain social event classification is to be able to classify both known and unknown social events during testing.

[0059] Optionally, before inputting the target data of social events into the open-domain social event classification model, the initial open-domain social event classification model needs to be trained to obtain the open-domain social event classification model. Figure 2 This is a schematic diagram of the training process of the open-domain social event classification model provided by the present invention, as shown below. Figure 2 As shown, the open-domain social event classification model is trained based on the following steps:

[0060] Construct a social event training dataset, wherein each social event training data in the social event training dataset includes at least two matching social event modal data and the modal label corresponding to the social event training data, and the social event training dataset includes training data of known classes and training data of unknown classes;

[0061] For each of the social event training data, determine the modal features corresponding to each of the social event modal data.

[0062] Each modal feature is input into a multimodal masking network to determine the multimodal fusion feature after the fusion of each modal feature;

[0063] Based on the training data of each social event, the labels corresponding to each social event training data, and the multimodal fusion features, the initial open-domain social event classification model is trained under supervision to obtain the open-domain social event classification model.

[0064] Specifically, training data for known classes in a social event training dataset can be constructed by acquiring current social media data. Alternatively, training data for unknown classes can be constructed based on the known class training data and a cross-modal event mixing mechanism, and the classification scores for each social event training data point can be determined. After constructing the social event training dataset, it is input into a pre-built initial open-domain social event classification model, and the multimodal fusion features f corresponding to the known class training data are extracted. i Multimodal fusion features of training data of unknown class Further open-domain classification is used to update the parameters of the initial open-domain social event classification model based on the open-domain classification loss function, which consists of an uncertainty loss function, a mixed classification loss function, and an unknown class focus loss function, thereby obtaining an open-domain social event classification model to improve classification accuracy.

[0065] Optionally, such as Figure 2 As shown, the unknown class training data is obtained based on the following steps:

[0066] Based on the known training data, determine the mixed sample pool;

[0067] From the mixed sample pool, randomly select any two training data of the known classes without replacement;

[0068] Interchange the social event modal data corresponding to any two of the known class training data to determine the unknown class training data.

[0069] Specifically, to satisfy open-domain classification and capture the potential distribution of unknown social events, in this embodiment of the invention, after constructing training data for known classes, a portion of the known class training data is randomly selected as a mixed sample pool. For each social event in the mixed sample pool, another category of social events is randomly selected. Any two known class training data sets include labels, image information, and text information. The image and text information of any two known class training data sets are then interchanged to generate unknown class training data. This unknown class training data does not belong to any known category of events; that is, the labels of the unknown class training data change, but it has the same text or image data distribution as the known class training data. Through the generated unknown class training data, unknown social events are learned and identified.

[0070] Optionally, after determining the training data for the known and unknown classes, the classification scores for the known classes under the hybrid mechanism can be further determined. And unknown class classification score Based on the classification score of the unknown class The pseudo-labels corresponding to the unknown class training data can be determined using the rules shown in equation (1). Equation (1) is:

[0071]

[0072] Where p represents the probability of using the unknown category with the maximum confidence, U(·) represents a uniform distribution, K represents the number of heads in the multi-head virtual classification network, y represents the label sampled from the uniform distribution U(·), and C is the number of known categories.

[0073] Optionally, the initial open-domain social event classification model described above includes: a modal feature extractor, a multimodal masking network, and an open-domain classifier. The modal feature extractor is used to extract image feature representations and text feature representations corresponding to the social event training data. Figure 3 This is a schematic diagram of the process for determining multimodal fusion features provided by the present invention, such as... Figure 3 As shown, after determining the image and text information corresponding to the target data or training data of a social event, it is necessary to extract the image feature representation corresponding to the image information and the text feature representation corresponding to the text information, respectively, as the modal features corresponding to the target data of the social event. Therefore, in this embodiment of the invention, two deep learning networks, BERT and ResNet, are pre-trained. After training, the text information is input into the deep learning network BERT, and the image information is input into the deep learning network ResNet. The image information and text information are mapped to the feature space through their respective deep learning networks to determine the image feature representation corresponding to the image information and the text feature representation corresponding to the text information.

[0074] Optionally, the multimodal masking network can be a multimodal masking Transformer network, which includes a feature fusion module and a multimodal masking module. The feature fusion module includes two encoders and a fusion unit. The two encoders are used to generate the image modality fusion features f corresponding to the image feature representations. i vt The text modality fusion feature f corresponding to the text feature representation i tv The multimodal masking module is used to remove redundant information in image feature representation and text feature representation.

[0075] Optionally, the open-domain classifier may include a closed-domain classifier and a multi-head virtual classification network, wherein both the closed-domain classifier and the multi-head virtual classification network can be softmax classifiers. When training the open-domain classifier, after constructing training data for known and unknown classes, the initial closed-domain classifier and the multi-head virtual classification network can be trained together using the known and unknown class training data. Alternatively, after constructing training data for known classes, the initial closed-domain classifier can be trained first using the known class training data to obtain a trained closed-domain classifier, and then, after constructing training data for known and unknown classes, the trained closed-domain classifier and the multi-head virtual classification network can be trained together using the known and unknown class training data.

[0076] Optionally, such as Figure 3 As shown, after determining each modal feature, determining the multimodal fusion feature after fusing the modal features includes:

[0077] For each of the modal features, determine the modal fusion feature used as the query feature;

[0078] The modal fusion features are fused together to obtain the multimodal fusion features.

[0079] Specifically, to further capture cross-modal semantic relationships between different modal features, in this embodiment of the invention, taking the modal features including image feature representation and text feature representation as an example, the image feature representation and text feature representation can be input into a multimodal masking network respectively. Based on a multi-head attention mechanism, the text modal fusion feature f, which is used as the query text feature representation, is determined respectively. i tv Image modality fusion features f, which are represented by image features as queries i vt And for text modality fusion features f i tv Image modality fusion features f i vt The fusion is performed to obtain the multimodal fusion feature f. i By mapping the features of each modality to a common semantic representation space, a common representation of cross-modal data can be obtained.

[0080] Optionally, such as Figure 3 As shown, determining the modal feature as the modal fusion feature for the query includes:

[0081] For each modal feature, a key-value pair corresponding to the modal feature is determined based on a multi-head attention mechanism;

[0082] Based on the mask matrix and the key-value pairs, the modal features are determined as the modal fusion features for querying, and the mask matrix is ​​used to filter redundant information in the modal fusion features.

[0083] Optionally, the mask matrix is ​​determined based on the following steps:

[0084] Determine the semantic similarity between each of the modal features;

[0085] Each semantic similarity is compared with a similarity threshold to determine the comparison result;

[0086] Based on the comparison results, the mask matrix corresponding to each modal feature is determined.

[0087] Specifically, modal features include image features. and text feature representation For example, multimodal fusion feature f i The determination includes the following steps:

[0088] 1) After determining the image feature representation and text feature representation, firstly, one of the encoders in the feature fusion module converts the image feature representation V... i These are respectively related to the two parameter matrices in the multi-head attention mechanism. and Multiplying them together yields the image feature representation V. i The corresponding key-value pairs, and the text feature representation T i and parameter matrix Multiplication is performed, and the text attention value corresponding to the text feature representation is obtained through Equation (2) using a multi-head attention mechanism. And based on the text attention value Equation (3) yields the text modality fusion feature f, which uses fine-grained text feature representation as the query. i tv Meanwhile, in determining the text modality fusion feature f i tv At that time, the attention mechanism is used to calculate the text feature representation T through equation (4). i and image feature representation V i semantic similarity between text and images And the semantic similarity between text and images The similarity threshold is compared to determine the text mask matrix MS corresponding to the text feature representation. tv Whether the corresponding position is 0 or 1, for example, if the semantic similarity between text and image... If the similarity is less than the similarity threshold, the corresponding position in the text mask matrix is ​​0; if the semantic similarity between text and image is less than the threshold, then the position is 0. If the similarity is greater than or equal to the similarity threshold, the corresponding position in the text mask matrix is ​​set to 1. By combining the text mask matrix with a multi-head attention mechanism, the text attention value is determined. Then, redundant information in the text feature representation is filtered out to obtain the text modality fusion feature f. i tv .in:

[0089] Equation (2) is:

[0090]

[0091] in, This represents the attention value from text words to image regions. Let T represent the learnable query parameter matrix, key parameter matrix, and value parameter matrix, respectively, and d represent the text feature representation. i and image feature representation V i The feature dimensions.

[0092] Equation (3) is:

[0093]

[0094] in, LayerNorm represents the text modality fusion feature, and LayerNorm represents the layer regularization function. and Let n represent the matrix parameters and vector parameters of the linear layer, respectively. t This indicates the number of words in the text.

[0095] Equation (4) is:

[0096]

[0097] in, T represents the similarity between text words and image regions, where d represents the text feature representation. i and image feature representation V i The feature dimensions.

[0098] 2) Another encoder in the feature fusion module represents the text features T. i These are respectively related to the two parameter matrices in the multi-head attention mechanism. and Multiplying them together yields the text feature representation T. i The corresponding key-value pairs, and the image feature representation V i With parameter matrix Multiply by equation (5) to obtain the image feature representation V. i The corresponding image attention value Att i vtAnd based on the image attention value The image modality fusion feature f, which is a fine-grained image feature representation used as the query, is obtained through equation (6). i vt Meanwhile, in determining the image modality fusion features f i vt At that time, the image feature representation V is calculated using equation (7). i Text feature representation T i Image-text semantic similarity between And the semantic similarity between images and text The image mask matrix MS corresponding to the image feature representation is compared with a similarity threshold to determine the image feature representation. vt In the image mask matrix MS vt Combined with multi-head attention mechanisms, by determining image attention values This leads to the image modality fusion feature f, which removes redundant information from the image feature representation. i vt .in:

[0099] Equation (5) is:

[0100]

[0101] in, For attention from image regions to text words, These represent the learnable query parameter matrix, key parameter matrix, and value parameter matrix, respectively, where d is the text feature representation T. i and image feature representation V i The feature dimensions.

[0102] Equation (6) is:

[0103]

[0104] Among them, f i vt For image modality fusion features, LayerNorm is the layer regularization function. and For the matrix and vector parameters of the linear layer, n v This represents the number of image regions.

[0105] Equation (7) is:

[0106]

[0107] in, Let d represent the similarity between image regions and text words, and let d represent the text feature representation T. i and image feature representation V i The feature dimensions.

[0108] 3) The fusion unit can fuse image modal features f i vt Text modality fusion features f i tv Multimodal fusion features f are obtained by splicing together the data. i .

[0109] Optionally, the open-domain classification loss function is determined based on the following steps:

[0110] Determine the known class classification score corresponding to the known class training data and the unknown class classification score corresponding to the unknown class training data, respectively;

[0111] The mixed classification loss function is determined based on the known class classification scores and the unknown class classification scores;

[0112] Based on the classification score of the unknown class, determine the focus loss function for the unknown class;

[0113] Based on the known class classification scores, determine the uncertainty loss function;

[0114] Determine the weights corresponding to the hybrid classification loss function, the unknown class focus loss function, and the uncertainty loss function, respectively.

[0115] Based on the weights, the mixed classification loss function, the unknown class focus loss function, and the uncertainty loss function are weighted and summed to determine the open domain classification loss function.

[0116] Specifically, after constructing training data for the unknown classes, the classification scores of the known classes under the influence of the hybrid mechanism can be further determined. And unknown class classification score Among them, W s and W u These are the classification parameters for known classes and unknown classes, respectively. Further, the classification scores for the known classes are used... And unknown class classification score Determine the open-domain loss function, which consists of a mixture classification loss function, an unknown class focus loss function, and an uncertainty loss function, where:

[0117] 1) Training data for known classes has corresponding labels, while training data for unknown classes has corresponding pseudo-labels. During training, the classification scores of known classes can be used. And unknown class classification score The predicted categories of the initial open-domain social event classification model are determined using equation (8). And based on prediction category The hybrid classification loss function is determined by equation (9). By utilizing training data of unknown classes with pseudo-labels, the boundaries of known class events can be effectively tightened, and the distribution of events of potential unknown classes in the feature space can be modeled to ensure that the predicted class is consistent with the label or pseudo-label. Wherein:

[0118] Equation (8) is:

[0119]

[0120] in, To predict classification probabilities for open domains, Given the classification scores of the known classes, Classify scores for unknown categories. It is a series function.

[0121] Equation (9) is:

[0122]

[0123] Among them, L m For mixed classification loss, Predict the classification probability for the open domain of the i-th sample belonging to the j-th class. For open-domain classification labels, N is the number of samples, C is the number of known categories, and K is the number of heads in the multi-head virtual classification network.

[0124] 2) Since the number of unknown class training data is smaller than that of known class training data in the social event training dataset, in order to balance the learning of the initial open-domain social event classification model on the unknown class training data and the known class training data, the focus loss function for the unknown class is determined by equations (10)-(12) to tilt the learning focus towards the unknown class training data during the training process, so as to improve the discrimination and representativeness of different unknown categories. Wherein:

[0125] Equation (10) is:

[0126]

[0127] in, Predict the classification probability for the unknown class. K represents the classification score for the unknown class, and K represents the number of heads in the multi-head virtual classification network.

[0128] Equation (11) is:

[0129]

[0130] in, The focus classification probability for the unknown class. Predict the classification probability of the open domain for the i-th mixed sample with the j-th unknown class. Let be the open-domain classification label of the i-th mixed sample, and C represents the number of known categories.

[0131] Equation (12) is:

[0132]

[0133] Among them, L f For the unknown class focus loss, N is the number of samples, and γ is the focus hyperparameter.

[0134] 3) To further distinguish between unknown and known events, the classification scores of the known events are used. Using equations (13) and (14), the uncertainty loss function is determined. By optimizing the uncertainty loss of known classes, the uncertainty of unknown class event features is improved. Wherein:

[0135] Equation (13) is:

[0136]

[0137] in, Predict the classification probability for a known class. C represents the classification score of the known classes, and C represents the number of known classes.

[0138] Equation (14) is:

[0139]

[0140] Among them, L e For known classes of uncertainty loss, Let N be the predicted classification probability of the j-th known category of the i-th mixed sample, where N is the number of samples and C is the number of known categories.

[0141] 4) After determining the mixed classification loss function, the unknown class focus loss function, and the uncertainty loss function, the weights of each of the three loss functions can be determined empirically. The open-domain classification loss function is then determined by weighted summation of the three loss functions as shown in equation (15). Equation (15) is:

[0142] Lo=λL m +αL f +βL e

[0143] Optionally, Figure 4 This is a schematic diagram illustrating the classification results of the open-domain social event classification method provided by this invention, as shown below. Figure 4As shown, after training the open-domain social event classification model, the target data of the social events to be tested, which simultaneously contain known and unknown classes, are input into the open-domain social event classification model for classification, resulting in partial social event target data corresponding to category 1, partial social event target data corresponding to category 2, partial social event target data corresponding to category 3, and partial social event target data corresponding to the unknown class. Among them, category 1, category 2, and category 3 are all known classes, that is, the categories that appear during the training process.

[0144] Table 1 Comparison results of the three classification methods

[0145]

[0146] Optionally, the open-domain social event classification method provided in this embodiment of the invention is applied to the CrisisMMD multimodal dataset. The CrisisMMD multimodal dataset includes 16,097 text messages and 18,126 image messages, containing a total of 7 categories. Several categories are selected as unknown classes, and the remaining categories are selected as known classes that appeared during training. Simultaneously, in the CrisisMMD multimodal dataset, a social event training dataset, a social event validation dataset, and a social event test dataset are constructed in a 7:1:2 ratio, and the hyperparameters in the open-domain social event classification model are determined using the social event validation dataset. Furthermore, the open-domain social event classification method (OWSEC) provided in this embodiment of the invention is compared with existing classification methods Multi-RC and SCBD under different numbers of unknown categories. Two evaluation metrics, accuracy (ACC) and Macro F1, are used to quantitatively compare the three classification methods, and the comparison results are shown in Table 1. The Multi-RC method performs classification learning by simply concatenating different modal features, while the SCBD method utilizes an attention mechanism to fuse multimodal features. As shown in Table 1, when the number of unknown categories in the target data of social events to be tested is different, the accuracy of the open domain social event classification method provided in this embodiment of the invention is better than that of the Multi-RC method and the SCBD method in both ACC and Macro F1.

[0147] It should be noted that the Macro F1 mentioned above can be calculated using the harmonic average of recall and precision (ACC).

[0148] The open-domain social event classification method provided by this invention acquires social event target data and fuses at least two modal features of the social event target data. By utilizing the fine-grained semantic associations between different modal features, the at least two modal features are fused to obtain multimodal fusion features, thereby obtaining cross-modal semantic relationships between each modal feature. This further determines the classification category corresponding to the social event target data. Different modal features complement and enhance each other to improve classification accuracy. At the same time, the open-domain loss function in the open-domain social event classification model used for classification combines a hybrid classification loss function, an unknown class focus loss function, and an uncertainty loss function to achieve open-domain social event classification for events of known or unknown categories.

[0149] The open-domain social event classification device provided by the present invention is described below. The open-domain social event classification device described below can be referred to in correspondence with the open-domain social event classification method described above.

[0150] This invention also provides an open-domain social event classification device. Figure 5 This is a schematic diagram of the open-domain social event classification device provided by the present invention, as shown below. Figure 5 As shown, the open-domain social event classification device 500 includes: an acquisition module 501 and an output module 502, wherein:

[0151] Module 501 is used to acquire target data of social events;

[0152] Output module 502 is used to input the social event target data into the open domain social event classification model and output the classification category corresponding to the social event target data;

[0153] The classification category is determined by a multimodal fusion feature obtained by fusing at least two modal features of the social event target data. The multimodal fusion feature is used to characterize the cross-modal semantic relationship between the modal features. The open-domain classification loss function of the open-domain social event classification model is determined based on a hybrid classification loss function, an unknown class focus loss function, and an uncertainty loss function.

[0154] The open-domain social event classification device provided by this invention acquires social event target data and fuses at least two modal features of the social event target data. By utilizing the fine-grained semantic associations between different modal features, the device fuses at least two modal features to obtain multimodal fusion features, thereby obtaining cross-modal semantic relationships between each modal feature. This further determines the classification category corresponding to the social event target data. Different modal features complement and enhance each other to improve classification accuracy. At the same time, the open-domain loss function in the open-domain social event classification model used for classification combines a hybrid classification loss function, an unknown class focus loss function, and an uncertainty loss function to achieve open-domain social event classification for events of known or unknown categories.

[0155] Optionally, the output module 502 is specifically used for:

[0156] The open-domain social event classification model is trained based on the following steps:

[0157] Construct a social event training dataset, wherein each social event training data in the social event training dataset includes at least two matching social event modal data and the modal label corresponding to the social event training data, and the social event training dataset includes training data of known classes and training data of unknown classes;

[0158] For each of the social event training data, determine the modal features corresponding to each of the social event modal data.

[0159] Each modal feature is input into a multimodal masking network to determine the multimodal fusion feature after the fusion of each modal feature;

[0160] Based on the training data of each social event, the labels corresponding to each social event training data, and the multimodal fusion features, the initial open-domain social event classification model is trained under supervision to obtain the open-domain social event classification model.

[0161] Optionally, the output module 502 is specifically used for:

[0162] The unknown class training data was obtained based on the following steps:

[0163] Based on the known training data, determine the mixed sample pool;

[0164] From the mixed sample pool, randomly select any two training data of the known classes without replacement;

[0165] Interchange the social event modal data corresponding to any two of the known class training data to determine the unknown class training data.

[0166] Optionally, the output module 502 is specifically used for:

[0167] The determination of the multimodal fusion feature after fusing the modal features includes:

[0168] For each of the modal features, determine the modal fusion feature used as the query feature;

[0169] The modal fusion features are fused together to obtain the multimodal fusion features.

[0170] Optionally, the output module 502 is specifically used for:

[0171] The step of determining the modal feature as the modal fusion feature for the query includes:

[0172] For each modal feature, a key-value pair corresponding to the modal feature is determined based on a multi-head attention mechanism;

[0173] Based on the mask matrix and the key-value pairs, the modal features are determined as the modal fusion features for querying, and the mask matrix is ​​used to filter redundant information in the modal fusion features.

[0174] Optionally, the output module 502 is specifically used for:

[0175] The mask matrix is ​​determined based on the following steps:

[0176] Determine the semantic similarity between each of the modal features;

[0177] Each semantic similarity is compared with a similarity threshold to determine the comparison result;

[0178] Based on the comparison results, the mask matrix corresponding to each modal feature is determined.

[0179] Optionally, the output module 502 is specifically used for:

[0180] The open-domain classification loss function is determined based on the following steps:

[0181] Determine the known class classification score corresponding to the known class training data and the unknown class classification score corresponding to the unknown class training data, respectively;

[0182] The mixed classification loss function is determined based on the known class classification scores and the unknown class classification scores;

[0183] Based on the classification score of the unknown class, determine the focus loss function for the unknown class;

[0184] Based on the known class classification scores, determine the uncertainty loss function;

[0185] Determine the weights corresponding to the hybrid classification loss function, the unknown class focus loss function, and the uncertainty loss function, respectively.

[0186] Based on the weights, the mixed classification loss function, the unknown class focus loss function, and the uncertainty loss function are weighted and summed to determine the open domain classification loss function.

[0187] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute an open-domain social event classification method, which includes:

[0188] Acquire target data on social events;

[0189] Input the target data of social events into the open-domain social event classification model, and output the classification category corresponding to the target data of social events;

[0190] The classification category is determined by a multimodal fusion feature obtained by fusing at least two modal features of the social event target data. The multimodal fusion feature is used to characterize the cross-modal semantic relationship between the modal features. The open-domain classification loss function of the open-domain social event classification model is determined based on a hybrid classification loss function, an unknown class focus loss function, and an uncertainty loss function.

[0191] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0192] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the open-domain social event classification method provided by the above methods, the method comprising:

[0193] Acquire target data on social events;

[0194] Input the target data of social events into the open-domain social event classification model, and output the classification category corresponding to the target data of social events;

[0195] The classification category is determined by a multimodal fusion feature obtained by fusing at least two modal features of the social event target data. The multimodal fusion feature is used to characterize the cross-modal semantic relationship between the modal features. The open-domain classification loss function of the open-domain social event classification model is determined based on a hybrid classification loss function, an unknown class focus loss function, and an uncertainty loss function.

[0196] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the open-domain social event classification method provided by the methods described above, the method comprising:

[0197] Acquire target data on social events;

[0198] Input the target data of social events into the open-domain social event classification model, and output the classification category corresponding to the target data of social events;

[0199] The classification category is determined by a multimodal fusion feature obtained by fusing at least two modal features of the social event target data. The multimodal fusion feature is used to characterize the cross-modal semantic relationship between the modal features. The open-domain classification loss function of the open-domain social event classification model is determined based on a hybrid classification loss function, an unknown class focus loss function, and an uncertainty loss function.

[0200] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0201] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An open-domain social event classification method, characterized in that, The method comprises: acquiring social event target data; inputting the social event target data into an open domain social event classification model to output a classification category corresponding to the social event target data; wherein the classification category is determined based on multi-modal fusion features of at least two modal features of the social event target data, the multi-modal fusion features are used to represent cross-modal semantic relationships between the modal features, and an open domain classification loss function of the open domain social event classification model is determined based on a mixed classification loss function, an unknown class focus loss function and an uncertainty loss function; the open domain social event classification model is obtained by training based on the following steps: constructing a social event training data set, each social event training data in the social event training data set comprising at least two matched social event modal data and a modal label corresponding to the social event training data, the social event training data set comprising known class training data and unknown class training data; determining modal features corresponding to each of the social event modal data for each of the social event training data; inputting each of the modal features into a multi-modal mask network to determine multi-modal fusion features after fusion of each of the modal features; supervising training of an initial open domain social event classification model based on each of the social event training data, labels corresponding to each of the social event training data and the multi-modal fusion features to obtain the open domain social event classification model; the determination of the multi-modal fusion features after fusion of each of the modal features comprises: determining a key-value pair corresponding to each of the modal features based on a multi-head attention mechanism; determining a modal fusion feature taking the modal feature as a query based on a mask matrix and the key-value pair, the mask matrix being used to filter redundant information in the modal fusion feature; fusing each of the modal fusion features to obtain the multi-modal fusion features. 2.The open-domain social event classification method of claim 1, wherein, the unknown class training data is obtained based on the following steps: determining a mixed sample pool based on the known class training data; randomly obtaining any two of the known class training data from the mixed sample pool without replacement; determining unknown class training data by interchanging the social event modal data corresponding to the any two of the known class training data. 3.The open-domain social event classification method of claim 1, wherein, the mask matrix is determined based on the following steps: determining semantic similarities between each of the modal features; comparing each of the semantic similarities with a similarity threshold to determine comparison results; determining a mask matrix corresponding to each of the modal features based on the comparison results. 4.The open-domain social event classification method of claim 1 or 2, wherein, the open domain classification loss function is determined based on the following steps: determining a known class classification score corresponding to the known class training data and an unknown class classification score corresponding to the unknown class training data; determining the mixed classification loss function based on the known class classification score and the unknown class classification score; determining the unknown class focus loss function based on the unknown class classification score; determining the uncertainty loss function based on the known class classification score. determine weights corresponding to the mixed classification loss function, the unknown class focal loss function and the uncertainty loss function respectively; based on the weights, perform weighted summation on the mixed classification loss function, the unknown class focal loss function and the uncertainty loss function to determine the open domain classification loss function.

5. An apparatus for implementing the open-domain social event classification method of any one of claims 1-4, characterized in that, comprise: an acquisition module, configured to acquire social event target data; an output module, configured to input the social event target data into an open domain social event classification model and output a classification category corresponding to the social event target data; wherein the classification category is determined based on multi-modal fusion features of at least two modal features of the social event target data, the multi-modal fusion features are used to represent cross-modal semantic relationships between the modal features, and an open domain classification loss function of the open domain social event classification model is determined based on a mixed classification loss function, an unknown class focal loss function and an uncertainty loss function.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the open domain social event classification method of any one of claims 1-4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the open domain social event classification method of any one of claims 1-4.

Images