Training method for social event classification model based on time series

Abstract

The present invention provides a training method for a social event classification model based on time series, which belongs to the field of artificial intelligence technology. The method includes: using a time series event knowledge network to divide the prediction classification evaluation results into past, current and future spatial event results, and determining the total loss value of the social event classification model based on the current space cross entropy loss, the past-future space co-activation constraint loss, the past event memory storage loss and the future event comparison mixed loss. The present invention classifies the prediction classification evaluation results obtained by the training model in time series according to the time series characteristics of the event, and introduces the current event knowledge learning, past event knowledge playback and future event knowledge preheating links when constructing the loss function of the social event classification model, so as to continuously learn the event data in an environment where the time series of social events continues to appear, avoid catastrophic forgetting of past event knowledge, and continuously classify incremental social events.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a training method for a social event classification model. Background Art

[0002] Today's social media contains a large amount of social event information composed of multimodal data. Different types of social events have different characteristics and forms of expression. Using these characteristics and forms of expression to classify social events is an important part of obtaining social event information from social media.

[0003] At present, the main method for classifying multimodal social events is to input the collected multimodal social event samples into the social event classification model, and update the loss function of the social event classification model according to the classification assessment prediction results and classification assessment result labels output by the social event classification model, so as to train the social event classification model and use the trained social event classification model for social event classification.

[0004] However, the current training methods of social event classification models do not take into account the impact of the temporal characteristics of multimodal social events on the classification assessment prediction results output by the social event classification model, making it difficult to further improve the performance of the social event classification model. Summary of the invention

[0005] The present invention provides a training method for a time-series-based social event classification model, which is used to solve the defect that the training of the social event classification model in the prior art does not take the time-series characteristics of multimodal social events into consideration.

[0006] In a first aspect, the present invention provides a training method for a time-series-based social event classification model, comprising:

[0007] The social event classification model includes a linear classifier and a temporal event knowledge network, and the training method includes:

[0008] Inputting the fusion features of multiple multimodal social event samples into the linear classifier, and outputting the prediction classification evaluation results corresponding to the multiple multimodal social event samples;

[0009] According to the occurrence time of the plurality of multimodal social event samples, the prediction classification evaluation results are divided into past spatial event results, current spatial event results and future spatial event results by using the temporal event knowledge network;

[0010] Calculate the total loss value of the social event classification model using the classification evaluation result labels corresponding to the multiple multimodal social event samples, the past spatial event results, the current spatial event results, and the future spatial event results, and optimize the model parameters of the social event classification model with the goal of minimizing the total loss value;

[0011] The total loss value is determined based on the current space cross entropy loss, the past-future space co-activation constraint loss, the past event memory storage loss and the future event contrast hybrid loss.

[0012] According to a training method for a time-series-based social event classification model provided by the present invention, the social event classification model further includes a picture feature extraction model, a text feature extraction model and a feature fusion model;

[0013] Accordingly, the fusion feature is obtained according to the following steps:

[0014] Inputting the multimodal social event sample into the image feature extraction model to obtain image features corresponding to the multimodal social event sample;

[0015] Inputting the multimodal social event sample into the text feature extraction model to obtain text features corresponding to the multimodal social event sample;

[0016] The image features and the text features are fused into fused features using the feature fusion model.

[0017] According to a training method for a time-series-based social event classification model provided by the present invention, the current spatial cross entropy loss is determined according to the following steps, including:

[0018] According to the classification and evaluation result labels of the current spatial event results and their corresponding multimodal social event samples, a normalized exponential function is calculated to obtain the current spatial cross entropy loss.

[0019] According to a training method for a time-series-based social event classification model provided by the present invention, the past-future space co-activation constraint loss is determined according to the following steps, including:

[0020] Converting the classification and evaluation result labels of the multimodal social event samples corresponding to the current spatial event result into a result vector, and determining a threshold value according to the inner product of the result vector and the vector of the current spatial event result;

[0021] Determining a past space classification assessment limit value based on the threshold value and the past space event result;

[0022] Determining a future space classification assessment limit value based on the threshold value and the future space event result;

[0023] The past-future spatial co-activation constraint loss is determined based on the past spatial classification assessment limit and the future spatial classification assessment limit.

[0024] According to a training method for a time-series-based social event classification model provided by the present invention, the past event memory storage loss is determined according to the following steps, including:

[0025] Inputting the fusion features of multiple past event samples in the memory storage body into the linear classifier to obtain the prediction classification evaluation results corresponding to the multiple past event samples;

[0026] The memory storage body includes a plurality of past event samples, and past prediction classification evaluation results and classification evaluation result labels corresponding to the past event samples; the past event samples are pre-extracted from the multimodal social event samples that have completed training and added to the memory storage body; the past prediction classification evaluation results are prediction classification evaluation results generated by the past event samples when training the social event classification model;

[0027] The past event memory storage loss is determined based on the predicted classification assessment results corresponding to the plurality of past event samples, the past predicted classification assessment results, and the classification assessment result labels.

[0028] According to a training method for a time-series-based social event classification model provided by the present invention, the method of determining the past event memory storage loss based on the prediction classification evaluation results corresponding to the multiple past event samples, the past prediction classification evaluation results and the classification evaluation result labels includes:

[0029] Calculating the past event knowledge distillation loss according to the Euclidean distance between the prediction classification evaluation result corresponding to the past event sample and the past prediction classification evaluation result;

[0030] Calculating a past event cross entropy loss based on a cross entropy loss between the predicted classification evaluation result corresponding to the past event sample and the classification evaluation result label;

[0031] The past event memory storage loss is determined based on the past event knowledge distillation loss and the past event cross entropy loss of all past event samples in the memory storage body.

[0032] According to a training method for a time-series-based social event classification model provided by the present invention, the future event contrast mixed loss is determined according to the following steps, including:

[0033] Modally mixing the image information of the plurality of multimodal social event samples with the text information of each of the past event samples, respectively, to obtain a first mixed sample set consisting of first mixed samples;

[0034] Modally mixing the text information of the plurality of multimodal social event samples with the image information of each of the past event samples, respectively, to obtain a second mixed sample set consisting of second mixed samples;

[0035] Inputting the fusion features of the first mixed sample into the linear classifier to obtain a corresponding first mixed prediction classification evaluation result;

[0036] Inputting the fusion features of the second mixed sample into the linear classifier to obtain a corresponding second mixed prediction classification evaluation result;

[0037] The future event comparison hybrid loss is determined based on the cosine similarity between the first hybrid prediction classification evaluation result and the second hybrid prediction classification evaluation result.

[0038] In a second aspect, the present invention further provides a social event classification method based on time series, comprising: inputting a multimodal social event to be classified into a social event classification model, and obtaining a classification assessment prediction result output by the social event classification model; the social event classification model comprises a linear classifier and a time series event knowledge network;

[0039] The social event classification model is trained according to any of the above-mentioned training methods for the time series-based social event classification model.

[0040] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements a training method for a time-series-based social event classification model as described above, or implements a time-series-based social event classification method as described above.

[0041] In a fourth aspect, the present invention also provides a transient computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for a time-series-based social event classification model as described above, or implements a time-series-based social event classification method as described above.

[0042] The training method of the time-series-based social event classification model provided by the present invention performs time-series classification on the predictive classification evaluation results obtained by the training model according to the time-series characteristics of the events, and introduces current event knowledge learning, past event knowledge playback and future event knowledge preheating links when constructing the loss function of the social event classification model, so as to continuously learn event data in an environment where social events continuously occur in time sequence, avoid catastrophic forgetting of past event knowledge, and continuously classify incremental social events. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 It is one of the flow charts of the training method of the time-series-based social event classification model provided by the present invention;

[0045] Figure 2 This is the second flow chart of the training method of the time-series-based social event classification model provided by the present invention;

[0046] Figure 3 It is a flow chart of the time-series-based social event classification method provided by the present invention;

[0047] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] It should be noted that, in the description of the present invention, the term "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0050] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0051] Combine the following Figure 1-Figure 4 The present invention describes the training method of the time-series-based social event classification model.

[0052] It should be noted that the training method of the time-series-based social event classification model and the execution subject of the time-series-based social event classification method provided by the present invention can be a server, a computer device, such as a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc.

[0053] Figure 1 is one of the flow charts of the training method of the time-series-based social event classification model provided by the present invention, such as Figure 1 As shown, the social event classification model includes but is not limited to a linear classifier and a temporal event knowledge network, and the training method includes but is not limited to the following steps:

[0054] Step 101: Input the fusion features of multiple multimodal social event samples into the linear classifier, and output the prediction classification evaluation results corresponding to the multiple multimodal social event samples.

[0055] Multimodal social events refer to social events recorded in multiple modal data such as text, pictures, and videos. For example, a multimodal social event is a news post on a hot social networking platform about the natural disaster "Hurricane Harvey", which includes picture information about the passage of "Hurricane Harvey" and text information about the impact of "Hurricane Harvey" on the economy and society.

[0056] In order to train the social event classification model, it is necessary to collect multiple multimodal social events as multimodal social event samples in advance, and annotate the category corresponding to each multimodal social event sample as the classification evaluation result label, so as to construct a training sample set, that is, the training sample set includes multiple multimodal social event samples and their corresponding classification evaluation result labels. For example, for the multimodal social event of the natural disaster "Hurricane Harvey", its classification evaluation result label is "Hurricane Harvey".

[0057] Specifically, when training a social event classification model including a linear classifier and a temporal event knowledge network, the fusion features of multiple multimodal social event samples are input into the linear classifier, and the prediction classification evaluation results corresponding to the multiple multimodal social event samples are output.

[0058] Among them, the fusion features of multimodal social event samples can be obtained by extracting multimodal features from the multimodal event samples and fusing them by other sub-models in the social event classification model after the multimodal social event samples are input into the social event classification model. At this time, when the model parameters of the social event classification model are updated and optimized, the model parameters of the sub-model for multimodal feature extraction and fusion will also be updated and optimized accordingly; it can also be that before the multimodal social event samples are input into the social event classification model, the multimodal social event samples are first input into other feature extraction models and feature fusion models for processing, so as to obtain the fusion features of the multimodal social event samples.

[0059] Step 102: Based on the occurrence time of the plurality of multimodal social event samples, the prediction classification evaluation results are divided into past spatial event results, current spatial event results and future spatial event results using the temporal event knowledge network.

[0060] Specifically, based on the different occurrence times of multimodal social events, each multimodal social event has corresponding temporal features. After the fusion features of the multimodal social event samples are input into the linear classifier to obtain the prediction classification evaluation results corresponding to multiple multimodal social event samples, the temporal event knowledge network is used to divide the obtained prediction classification evaluation results into past spatial event results, current spatial event results, and future spatial event results.

[0061] Step 103: Calculate the total loss value of the social event classification model using the classification assessment result labels corresponding to the multiple multimodal social event samples, the past spatial event results, the current spatial event results, and the future spatial event results, and optimize the model parameters of the social event classification model with the goal of minimizing the total loss value.

[0062] Among them, the total loss value of the social event classification model is determined based on the current space cross entropy loss, the past-future space co-activation constraint loss, the past event memory storage loss and the future event comparison mixed loss.

[0063] Specifically, after using the temporal event knowledge network to divide the predicted classification evaluation results into past spatial event results, current spatial event results and future spatial event results, the classification evaluation result labels corresponding to the multimodal social event samples are combined to calculate the total loss value determined by four losses: current spatial cross entropy loss, past-future spatial common activation constraint loss, past event memory storage loss and future event comparison mixed loss. With the goal of minimizing the total loss value, the model parameters of the social event classification model including the linear classifier and the temporal event knowledge network are optimized.

[0064] Accordingly, the total loss function of the social event classification model is determined based on the current space cross entropy loss function, the past-future space co-activation constraint loss function, the past event memory storage loss function, and the future event contrast hybrid loss function.

[0065] The current space cross entropy loss and the past-future space jointly activated constraint loss are used as part of the total loss value of the social event classification model. That is, when classifying and learning current events, a current event knowledge learning mechanism is established by introducing past space and future space bias mitigation techniques.

[0066] The memory storage loss of past events is taken as part of the total loss value of the social event classification model. That is, an adaptive classification assessment memory storage body is established to store a small amount of current training samples, and based on the knowledge distillation mechanism, the knowledge distillation loss and cross entropy loss of past events are calculated as the memory storage loss of past events, thereby establishing a knowledge playback mechanism for past events.

[0067] The future event contrast mixing loss is taken as part of the total loss value of the social event classification model. That is, the modal mixing technology is used to mix the different modal data of social events, and the mixed social events are taken as future events to calculate the future event contrast mixing loss, thereby establishing a knowledge preheating mechanism for future events.

[0068] The training method of the social event classification model provided by the present invention performs time series classification on the prediction classification evaluation results obtained by the training model according to the time series characteristics of the events, and introduces current event knowledge learning, past event knowledge playback and future event knowledge warm-up links when constructing the loss function of the social event classification model, so as to continuously learn event data in an environment where social events continuously occur in time series, avoid catastrophic forgetting of past event knowledge, and continuously classify incremental social events.

[0069] Figure 2 FIG. 2 is a flow chart of the training method of the time-series-based social event classification model provided by the present invention. Figure 2As shown, the social event classification model also includes a picture feature extraction model, a text feature extraction model and a feature fusion model;

[0070] Accordingly, the fusion feature is obtained according to the following steps:

[0071] Inputting the multimodal social event sample into the image feature extraction model to obtain image features corresponding to the multimodal social event sample;

[0072] Inputting the multimodal social event sample into the text feature extraction model to obtain text features corresponding to the multimodal social event sample;

[0073] The image features and the text features are fused into fused features using the feature fusion model.

[0074] Specifically, a pre-trained image feature extraction model is used to extract corresponding image features from multimodal social event samples; a pre-trained text feature extraction model is used to extract corresponding text features from the same multimodal social event sample; and a feature fusion model is used to transform and fuse the image features and text features of the same multimodal social event sample into fused features, so as to input the fused features of the multimodal social event sample into the linear classifier of the social event classification model to obtain the corresponding predictive classification evaluation results.

[0075] In one embodiment, the text feature extraction model is implemented based on a BERT (Bidirectional Encoder Representations from Transformers) network, and the image feature extraction model is implemented based on a ResNet (Residual Network) network.

[0076] From the training sample set Extracting multimodal social event samples in, is a sample of multimodal social events Text information, is a sample of multimodal social events 's picture information.

[0077] Furthermore, the multimodal social event samples Input into the text feature extraction model to obtain multimodal social event samples The corresponding text feature formula is as follows:

[0078]

[0079] Multimodal social event samples Input into the image feature extraction model to obtain multimodal social event samples The corresponding image feature formula is as follows:

[0080]

[0081] in, A sample of multimodal social events The corresponding text features, That is, the feature dimension is n p ×768 vector matrix, A sample of multimodal social events The corresponding image features, That is, the feature dimension is n v ×768 vector matrix, W r To unify the training matrix of image feature dimension and text feature dimension, σ(·) is the ReLU activation function.

[0082] In one embodiment, the feature fusion model is implemented based on a Multimodal Mask Transformer Network (MMTN).

[0083] Accordingly, the feature fusion model is used to transform the image features and the text features The formula for fusion into fusion features is as follows:

[0084]

[0085] Among them, f i t is the fusion feature, That is, the output dimension is a vector matrix of D, and θ1 is the training parameter of the feature fusion model implemented based on MMTN.

[0086] Furthermore, the fusion features f of the extracted multimodal social event samples are i t The linear classifier input to the social event classification model outputs the corresponding predicted classification evaluation result as follows:

[0087]

[0088] in, A sample of multimodal social events The corresponding prediction classification evaluation results, That is, the output dimension is a vector matrix C, and θ2 is the training parameter of the linear classifier W.

[0089] Furthermore, the output dimension C is the total number of categories of the predicted classification evaluation results.

[0090] Furthermore, the prediction space is divided into past space, current space and future space using the time series event knowledge network, that is, the prediction classification evaluation result is divided into past space event results, current space event results and future space event results using the time series event knowledge network. The calculation formula is as follows:

[0091]

[0092] Among them, [·,·] is the concatenation operation on the prediction dimension, As a result of past space events, is the result of the current space event, is the result of future space events, θ={W r ,θ1,θ2} are the training parameters of the social event classification model TEKNet.

[0093] Furthermore, the output dimension C is the total number of categories of past spatial event results, current spatial event results, and future spatial event results.

[0094] Based on the above embodiment, as an optional embodiment, the current spatial cross entropy loss is determined according to the following steps, including:

[0095] According to the classification and evaluation result labels of the current spatial event results and their corresponding multimodal social event samples, a normalized exponential function is calculated to obtain the current spatial cross entropy loss.

[0096] The normalized exponential function (Softmax function) refers to the logarithmic normalization of the gradient of a finite discrete probability distribution. It is widely used in various probability-based multi-classification problem methods including multinomial logistic regression, multinomial linear discriminant analysis, naive Bayes classifier and artificial neural network.

[0097] Cross-entropy loss is a commonly used loss function in deep learning. It is usually used for classification problems. It can measure the gap between the model prediction results and the actual results. It is one of the key indicators for optimizing model parameters.

[0098] Specifically, for the multimodal social event samples corresponding to the current spatial event results, that is, the current social events, the normalized exponential function is calculated according to the classification and evaluation result labels of the current spatial event results and their corresponding multimodal social event samples to obtain the current spatial cross entropy loss.

[0099] In one embodiment, for a training sample set The current social event in , according to the classification and evaluation result labels of the current spatial event results and their corresponding multimodal social event samples, calculates the normalized exponential function, and the calculation formula for the current spatial cross entropy loss is as follows:

[0100]

[0101] in, is the cross entropy loss of the current space, CE(·,·) is the cross entropy loss, A sample of multimodal social events The corresponding current spatial event result, A sample of multimodal social events The corresponding classification assessment result label.

[0102] Based on the above embodiment, as an optional embodiment, the past-future spatial co-activation constraint loss is determined according to the following steps, including:

[0103] Converting the classification and evaluation result labels of the multimodal social event samples corresponding to the current spatial event result into a result vector, and determining a threshold value according to the inner product of the result vector and the vector of the current spatial event result;

[0104] Determining a past space classification assessment limit value based on the threshold value and the past space event result;

[0105] Determining a future space classification assessment limit value based on the threshold value and the future space event result;

[0106] The past-future spatial co-activation constraint loss is determined based on the past spatial classification assessment limit and the future spatial classification assessment limit.

[0107] The multimodal social event samples corresponding to the current spatial event results are current social events, that is, positive events. The multimodal social event samples corresponding to the past spatial event results and the future spatial event results are negative events, which correspond to the past social events and future social events, respectively. By limiting the predicted classification assessment of negative events to below the threshold value, the activation restrictions of the past space and future space are set.

[0108] In one embodiment, the predicted classification evaluation of negative events is limited to be lower than a threshold value, that is, the classification evaluation result label of the multimodal social event sample corresponding to the current spatial event result is converted into a result vector, and the calculation formula for determining the threshold value is as follows based on the inner product of the result vector and the vector of the current spatial event result:

[0109]

[0110] in, is the threshold value, one-hot(·) converts the classification and evaluation result label of the current social event into a one-hot vector, and <·,·> is the vector inner product operation.

[0111] Furthermore, based on the past space classification assessment limit and the future space classification assessment limit, the calculation formula for determining the past-future space common activation constraint loss is as follows:

[0112]

[0113] in, is the past-future space joint activation constraint loss, is the threshold value, For the assessment limits of past space classifications determined based on threshold values ​​and the results of past space events, is the future space classification assessment limit determined according to the threshold value and the future space event results, and α is a hyperparameter that controls the relaxation of the constraint.

[0114] Based on the above embodiment, as an optional embodiment, the past event memory storage loss is determined according to the following steps, including but not limited to:

[0115] Inputting the fusion features of multiple past event samples in the memory storage body into the linear classifier to obtain the prediction classification evaluation results corresponding to the multiple past event samples;

[0116] The memory storage body includes a plurality of past event samples, and past prediction classification evaluation results and classification evaluation result labels corresponding to the past event samples; the past event samples are pre-extracted from the multimodal social event samples that have completed training and added to the memory storage body; the past prediction classification evaluation results are prediction classification evaluation results generated by the past event samples when training the social event classification model;

[0117] The past event memory storage loss is determined based on the predicted classification assessment results corresponding to the plurality of past event samples, the past predicted classification assessment results, and the classification assessment result labels.

[0118] Specifically, in one embodiment, before calculating the past event memory storage loss, a memory storage volume storing a small amount of past event samples is first established. Past event samples are pre-extracted from the multimodal social event samples that have completed training and added to the memory storage of.

[0119] Memory Storage Included samples of past events, is the i-th past event sample in the memory storage, is a sample of past events The predicted classification results generated when training the social event classification model, is a sample of past events The classification evaluation result label.

[0120] Optionally, the method of pre-extracting and adding past event samples from the trained multimodal social event samples to the memory storage body adopts a pond sampling algorithm.

[0121] In fact, after each iterative training process, a small number of multimodal social event samples are extracted from the training sample set of the iterative training based on various sampling methods such as the pond sampling algorithm and added to the memory storage as past event samples, thereby constructing the memory storage. Furthermore, the past event memory storage loss calculated based on the memory storage is used as part of the total loss value of the social event classification model, which ensures the replayability of the training samples.

[0122] After constructing the memory storage body, when calculating the memory storage loss of past events, the fused features of multiple past event samples in the memory storage body are input into the linear classifier in the current training process to obtain the prediction classification assessment results corresponding to the multiple past event samples (rather than the past prediction classification assessment results), and the past event memory storage loss is determined based on the prediction classification assessment results corresponding to the multiple past event samples, the past prediction classification assessment results and the classification assessment result labels.

[0123] As an optional embodiment, determining the past event memory storage loss based on the prediction classification evaluation results corresponding to the multiple past event samples, the past prediction classification evaluation results and the classification evaluation result labels includes:

[0124] Calculating the past event knowledge distillation loss according to the Euclidean distance between the prediction classification evaluation result corresponding to the past event sample and the past prediction classification evaluation result;

[0125] Calculating a past event cross entropy loss based on a cross entropy loss between the predicted classification evaluation result corresponding to the past event sample and the classification evaluation result label;

[0126] The past event memory storage loss is determined based on the past event knowledge distillation loss and the past event cross entropy loss of all past event samples in the memory storage body.

[0127] In one embodiment, the calculation formula for determining the past event memory storage loss is as follows:

[0128]

[0129] in, Loss of memory storage for past events, Memory storage Sample past events The past prediction classification evaluation results of For past event samples The predicted classification evaluation results in the current training process, The Euclidean distance operation for calculating the past event knowledge distillation loss, For past event samples The classification evaluation result label of , CE(·,·) is the cross entropy loss of past events.

[0130] By calculating the memory storage loss of past events and replaying the past event samples and classification evaluation results in the memory storage body, the social event classification model including the temporal event knowledge network can be optimized and updated.

[0131] Based on the above embodiment, as an optional embodiment, the future event comparison mixed loss is determined according to the following steps, including:

[0132] Modally mixing the image information of the plurality of multimodal social event samples with the text information of each of the past event samples, respectively, to obtain a first mixed sample set consisting of first mixed samples;

[0133] Modally mixing the text information of the plurality of multimodal social event samples with the image information of each of the past event samples, respectively, to obtain a second mixed sample set consisting of second mixed samples;

[0134] Inputting the fusion features of the first mixed sample into the linear classifier to obtain a corresponding first mixed prediction classification evaluation result;

[0135] Inputting the fusion features of the second mixed sample into the linear classifier to obtain a corresponding second mixed prediction classification evaluation result;

[0136] The future event comparison hybrid loss is determined based on the cosine similarity between the first hybrid prediction classification evaluation result and the second hybrid prediction classification evaluation result.

[0137] In one embodiment, when calculating the future event contrast mixing loss, firstly, a first mixed sample set and a second mixed sample set representing the future event are obtained through modal mixing.

[0138] Specifically, a multimodal social event sample is selected from multiple multimodal social event samples in the training sample set. Also select a past event sample from multiple past event samples in the memory storage body Perform mode mixing to obtain a first mixed sample and a second mixed sample By traversing all multimodal social event samples in the current training sample set and the memory storage, and repeating the above modal mixing operation, the first mixed sample set is obtained. and the second mixed sample set Among them, each first mixed sample in the first mixed sample set consists of picture information of a multimodal social event sample and text information of a past event sample, and each second mixed sample in the second mixed sample set consists of text information of a multimodal social event sample and picture information of a past event sample.

[0139] Further, the feature extraction and feature fusion model is used to obtain each first mixed sample in the first mixed sample set. The corresponding fusion features Get each second mixed sample in the second mixed sample set The corresponding fusion features

[0140] Then, the first mixed sample Fusion features Input to the linear classifier to get the first mixed sample The corresponding first mixed prediction classification evaluation result The second mixed sample Fusion features Input to the linear classifier to get the second mixed sample The corresponding second hybrid prediction classification evaluation result

[0141] Finally, the future event contrast hybrid loss is determined based on the cosine similarity of the first hybrid prediction classification evaluation results of all first hybrid samples in the first hybrid set and the second hybrid prediction classification evaluation results of all second hybrid samples in the second hybrid set.

[0142] In one embodiment, based on the cosine similarity between the first hybrid prediction classification evaluation result and the second hybrid prediction classification evaluation result, a calculation formula for determining the future event comparison hybrid loss is as follows:

[0143]

[0144] in, Compare mixed losses for future events, is the second mixed sample set, is the first mixed sample set, For the second mixed sample The second mixed prediction classification evaluation results, The first mixed sample The first mixed prediction classification evaluation result is , cos(·,·) is the cosine similarity, and τ is the temperature parameter.

[0145] Taking the minimum total loss value as the optimization goal, the model parameters of the social event classification model are updated by continuously optimizing the sum of the current space cross entropy loss, the past-future space co-activation constraint loss, the past event memory storage loss and the future event contrast mixed loss. Finally, a continuous social event classification model based on the temporal event knowledge network is obtained.

[0146] In one embodiment, the total loss function of the social event classification model determined based on the current space cross entropy loss function, the past-future space co-activation constraint loss function, the past event memory storage loss function and the future event comparison mixed loss function is calculated as follows:

[0147]

[0148] in, is the total loss function, δ,∈,ζ are weight coefficients, and θ is the model parameter of the social event classification model including the linear classifier and the temporal event knowledge network.

[0149] Figure 3 is a flow chart of the time-series-based social event classification method provided by the present invention, such as Figure 3 As shown, the social event classification method includes but is not limited to:

[0150] Step 300: Input the multimodal social events to be classified into a social event classification model to obtain a classification assessment prediction result output by the social event classification model; the social event classification model includes a linear classifier and a temporal event knowledge network.

[0151] Among them, the social event classification model including the linear classifier and the time series event knowledge network is trained according to the training method of the time series-based social event classification model described in any of the above embodiments.

[0152] The time-series-based social event classification method provided by the present invention performs time-series classification on the predictive classification evaluation results obtained by the training model according to the time-series characteristics of the events, and introduces current event knowledge learning, past event knowledge playback and future event knowledge warm-up links when constructing the loss function of the social event classification model, so as to continuously learn event data in an environment where social events continuously occur in time series, avoid catastrophic forgetting of past event knowledge, and continuously classify incremental social events.

[0153] In order to better illustrate the training method of the time-series-based social event classification model provided by the present invention and the performance effect of the social event classification method, the following shows the results of the social event classification model provided by the present invention and other different social event classification models on the CrisisMMD dataset by evaluating three common indicators for classification quality: Average Accuracy (AA), Average F1 (AF1) and Final Forgetting (FF).

[0154] The CrisisMMD dataset is an existing dataset widely used in computer vision research for image classification and related tasks. It provides data of different scales and complexities, which can be used to test and verify the performance of image processing and learning algorithms. The CrisisMMD dataset specifically includes news releases on seven natural disasters.

[0155] Table 1

[0156]

[0157] Table 1 is the results of different social event classification models provided by the present invention on CrisisMMD. As shown in Table 1, Joint Training is the upper limit set for this task, and each task is trained using all data; the GEM classifier uses episodic memory to learn the common correlation subset of the data distribution, the iCaRL classifier combines the group buffer with the recent sample mean classifier for classification, the LiDER classifier repeatedly optimizes a small amount of data, the X-DER classifier maintains existing knowledge through continuous updating and considers possible social events, the CLS-ER classifier maintains short-term and long-term semantic memory that interacts with episodic memory, the PCR classifier performs corresponding anchor proxy replacement through sample comparison, the ESMER classifier modulates the error sensitivity of the dual memory system based on the principle mechanism, and the TAMiL classifier includes experience exercises and self-regulating neural network extensions.

[0158] It can be seen that the time-series-based social event classification model TEKNet (Ours) provided by the present invention has the best effect on AA, AF1 and FF indicators, while X-DER has suboptimal effect on AA and AF1, and LiDER has suboptimal effect on FF.

[0159] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4As shown, the electronic device may include: a processor (Processor) 410, a communication interface (Communications Interface) 420, a memory (Memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logic instructions in the memory 430 to execute the training method of the time-series-based social event classification model described in any of the above embodiments, wherein the social event classification model includes a linear classifier and a time-series event knowledge network, and the training method includes but is not limited to: inputting the fusion features of multiple multimodal social event samples into the linear classifier, and outputting the predicted classification evaluation results corresponding to the multiple multimodal social event samples; according to the occurrence time of the multiple multimodal social event samples, using the time-series event knowledge network to divide the predicted classification evaluation results into past spatial event results, current spatial event results and future spatial event results; using the classification evaluation result labels corresponding to the multiple multimodal social event samples, the past spatial event results, the current spatial event results and the future spatial event results, calculate the total loss value of the social event classification model, and optimize the model parameters of the social event classification model with the minimum total loss value as the goal; the total loss value is determined based on the current space cross entropy loss, the past-future space common activation constraint loss, the past event memory storage loss and the future event comparison mixed loss;

[0160] Alternatively, execute the time series-based social event classification method described in any of the above embodiments, which includes but is not limited to: inputting the multimodal social event to be classified into a social event classification model to obtain a classification assessment prediction result output by the social event classification model; the social event classification model includes a linear classifier and a time series event knowledge network; and the social event classification model is trained according to the time series-based social event classification model training method described in any of the above embodiments.

[0161] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0162] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when executed by a processor to execute the training method of the time-series-based social event classification model described in any of the above embodiments, wherein the social event classification model includes a linear classifier and a time-series event knowledge network, and the training method includes but is not limited to: inputting the fusion features of multiple multimodal social event samples into the linear classifier, and outputting the predicted classification evaluation results corresponding to the multiple multimodal social event samples; dividing the predicted classification evaluation results into past spatial event results, current spatial event results, and future spatial event results according to the occurrence time of the multiple multimodal social event samples using the time-series event knowledge network; calculating the total loss value of the social event classification model using the classification evaluation result labels corresponding to the multiple multimodal social event samples, the past spatial event results, the current spatial event results, and the future spatial event results, and optimizing the model parameters of the social event classification model with the goal of minimizing the total loss value; the total loss value is determined based on the current space cross entropy loss, the past-future space co-activation constraint loss, the past event memory storage loss, and the future event comparison mixed loss;

[0163] Alternatively, execute the time series-based social event classification method described in any of the above embodiments, which includes but is not limited to: inputting the multimodal social event to be classified into a social event classification model to obtain a classification assessment prediction result output by the social event classification model; the social event classification model includes a linear classifier and a time series event knowledge network; and the social event classification model is trained according to the time series-based social event classification model training method described in any of the above embodiments.

[0164] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0165] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a time-series-based social event classification model, characterized in that: The social event classification model includes a linear classifier and a temporal event knowledge network, and the training method includes: Inputting the fusion features of multiple multimodal social event samples into the linear classifier, and outputting the prediction classification evaluation results corresponding to the multiple multimodal social event samples; According to the occurrence time of the plurality of multimodal social event samples, the prediction classification evaluation results are divided into past spatial event results, current spatial event results and future spatial event results by using the temporal event knowledge network; Calculate the total loss value of the social event classification model using the classification evaluation result labels corresponding to the multiple multimodal social event samples, the past spatial event results, the current spatial event results, and the future spatial event results, and optimize the model parameters of the social event classification model with the goal of minimizing the total loss value; The total loss value is determined based on the current space cross entropy loss, the past-future space co-activation constraint loss, the past event memory storage loss and the future event contrast mixed loss; The calculation formula of the current spatial cross entropy loss is: in, is the cross entropy loss of the current space, CE(·,·) is the cross entropy loss, A sample of multimodal social events The corresponding current spatial event result, A sample of multimodal social events The corresponding classification evaluation result label; The calculation formula of the past-future space co-activation constraint loss is: in, is the past-future space joint activation constraint loss, is the threshold value, For the assessment limits of past space classifications determined based on threshold values ​​and the results of past space events, is the future space classification assessment limit determined based on the threshold value and the future space event results, and α is a hyperparameter that controls the relaxation of the constraint; The calculation formula for the past event memory storage loss is: in, Loss of memory storage for past events, Memory storage Sample past events The past prediction classification evaluation results of For past event samples The predicted classification evaluation results in the current training process, The Euclidean distance operation for calculating the past event knowledge distillation loss, For past event samples The classification evaluation result label of , CE(·,·) is the cross entropy loss of past events; The calculation formula for the future event comparison mixed loss is: in, Compare mixed losses for future events, is the second mixed sample set, is the first mixed sample set, For the second mixed sample The second mixed prediction classification evaluation results, The first mixed sample The first mixed prediction classification evaluation result is , cos(·,·) is the cosine similarity, and τ is the temperature parameter.

2. The training method of the time-series-based social event classification model according to claim 1 is characterized in that: The social event classification model also includes a picture feature extraction model, a text feature extraction model and a feature fusion model; Accordingly, the fusion feature is obtained according to the following steps: Inputting the multimodal social event sample into the image feature extraction model to obtain image features corresponding to the multimodal social event sample; Inputting the multimodal social event sample into the text feature extraction model to obtain text features corresponding to the multimodal social event sample; The image features and the text features are fused into fused features using the feature fusion model.

3. The training method of the time-series-based social event classification model according to claim 1 is characterized in that: The current spatial cross entropy loss is determined according to the following steps, including: According to the classification and evaluation result labels of the current spatial event results and their corresponding multimodal social event samples, a normalized exponential function is calculated to obtain the current spatial cross entropy loss.

4. The training method of the time-series-based social event classification model according to claim 1 is characterized in that: The past-future spatial co-activation constraint loss is determined according to the following steps, including: Converting the classification and evaluation result labels of the multimodal social event samples corresponding to the current spatial event result into a result vector, and determining a threshold value according to the inner product of the result vector and the vector of the current spatial event result; Determining a past space classification assessment limit value based on the threshold value and the past space event result; Determining a future space classification assessment limit value based on the threshold value and the future space event result; The past-future spatial co-activation constraint loss is determined based on the past spatial classification assessment limit and the future spatial classification assessment limit.

5. The training method of the time-series-based social event classification model according to claim 1 is characterized in that: The past event memory storage loss is determined according to the following steps, comprising: Inputting the fusion features of multiple past event samples in the memory storage body into the linear classifier to obtain the prediction classification evaluation results corresponding to the multiple past event samples; The memory storage body includes a plurality of past event samples, and past prediction classification evaluation results and classification evaluation result labels corresponding to the past event samples; the past event samples are pre-extracted from the multimodal social event samples that have completed training and added to the memory storage body; the past prediction classification evaluation results are prediction classification evaluation results generated by the past event samples when training the social event classification model; The past event memory storage loss is determined based on the predicted classification assessment results corresponding to the plurality of past event samples, the past predicted classification assessment results, and the classification assessment result labels.

6. The training method of the time-series-based social event classification model according to claim 5 is characterized in that: The determining of the past event memory storage loss based on the prediction classification evaluation results corresponding to the plurality of past event samples, the past prediction classification evaluation results and the classification evaluation result labels comprises: Calculating the past event knowledge distillation loss according to the Euclidean distance between the prediction classification evaluation result corresponding to the past event sample and the past prediction classification evaluation result; Calculating a past event cross entropy loss based on a cross entropy loss between the predicted classification evaluation result corresponding to the past event sample and the classification evaluation result label; The past event memory storage loss is determined based on the past event knowledge distillation loss and the past event cross entropy loss of all past event samples in the memory storage body.

7. The training method of the time-series-based social event classification model according to claim 5 or 6, characterized in that: The future event versus mixed loss is determined according to the following steps, including: Modally mixing the image information of the plurality of multimodal social event samples with the text information of each of the past event samples, respectively, to obtain a first mixed sample set consisting of first mixed samples; Modally mixing the text information of the plurality of multimodal social event samples with the image information of each of the past event samples, respectively, to obtain a second mixed sample set consisting of second mixed samples; Inputting the fusion features of the first mixed sample into the linear classifier to obtain a corresponding first mixed prediction classification evaluation result; Inputting the fusion features of the second mixed sample into the linear classifier to obtain a corresponding second mixed prediction classification evaluation result; The future event comparison hybrid loss is determined based on the cosine similarity between the first hybrid prediction classification evaluation result and the second hybrid prediction classification evaluation result.

8. A social event classification method based on time series, characterized in that: include: Inputting the multimodal social event to be classified into the social event classification model to obtain the classification assessment prediction result output by the social event classification model; The social event classification model includes a linear classifier and a temporal event knowledge network; The social event classification model is trained according to the training method for the time-series-based social event classification model according to any one of claims 1-7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the training method of the time-series-based social event classification model as described in any one of claims 1 to 7, or implements the time-series-based social event classification method as described in claim 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the training method of the time-series-based social event classification model as described in any one of claims 1 to 7, or implements the time-series-based social event classification method as described in claim 8.

Images