A short video event detection method and system based on deep dynamic semantic association

Abstract

The application discloses a short video event detection method and system based on deep dynamic semantic correlation, which comprises the following steps: acquiring and strengthening frame importance information of visual features to obtain frame importance scores with distinguishability and short video feature representations weighted by the frame importance scores; according to the attention characteristics of the feature space, the short video feature representations weighted by the frame importance scores are used to guide the learning of attention to obtain frame inter self-attention enhanced feature representations; specific graph representations of short video samples are constructed, hidden attributes of complex events and the correlation therebetween are learned through a hidden attribute correlation learning network under dynamic graph convolution to obtain short video feature representations with potential semantic information perception; and event category scores are obtained according to the short video feature representations to complete the short video event detection task. The application provides a new method for solving the short video event detection problem and effectively enhances the feature representation capability.

Description

Technical Field

[0001] This invention belongs to the field of multimedia and big data analysis technology, and in particular relates to a short video event detection method and system based on deep dynamic semantic association. Background Technology

[0002] With the rapid development of the short video industry, short video content analysis, represented by short video event detection, has received increasing attention. Short video event detection helps solve the challenges of short video regulation and ensures the industry's continued healthy development. However, with the ever-growing number of short videos and the increasing complexity and diversity of the information they contain, how to quickly and efficiently find the short videos that users need using existing short video information has become an urgent problem to be solved.

[0003] Currently, artificial intelligence (AI) technologies, represented by deep learning, are rapidly developing across various fields, including video information processing. Utilizing AI to solve short video event detection problems can not only advance the field of computer vision but also improve user experience, making it valuable for both research and practical applications. Summary of the Invention

[0004] To achieve the above objectives, the present invention provides the following solution: a method and system for short video event detection based on deep dynamic semantic association. Specifically, the method for short video event detection based on deep dynamic semantic association includes:

[0005] Collect short videos and extract their visual features. Based on the visual features of the short videos, obtain and enhance the frame importance information of the visual features to obtain a frame importance score with discriminative power and a short video feature representation after weighting the frame importance scores.

[0006] Based on the attention characteristics of the feature space, the short video feature representation weighted by the frame importance score is used to guide the learning of attention, and the intrinsic correlation between frames and the feature space is coordinated to obtain the feature representation with enhanced self-attention between frames.

[0007] The hidden attributes of short video events are treated as nodes, and the correlation between these hidden attributes is treated as edges, thus constructing a specific graph representation of short video samples. A hidden attribute association learning network under dynamic graph convolution learns the hidden attributes of complex events and their correlations, thereby obtaining a short video feature representation with latent semantic information awareness.

[0008] The event category score is obtained based on the short video feature representation to complete the short video event detection task.

[0009] Preferably, the process of acquiring and enhancing frame importance information of the visual features based on the visual features of the short video, and obtaining a frame importance score with discriminative power and a short video feature representation weighted by the frame importance scores, includes:

[0010] By maximally enhancing frame importance information through a joint structure of variational autoencoder and generative adversarial network, we obtain a discriminative frame importance score and a short video feature representation after weighting the frame importance score.

[0011] The expression for the short video feature representation after weighting the frame importance scores is:

[0012]

[0013] in, The frame importance score is updated after passing through the embedded variational autoencoder and generative adversarial network joint structure. The representation after expanding the dimensions; The extracted original visual features of the short video; The feature representation is weighted by importance scores; T is the number of keyframes in the short video, B is the number of samples in the short video, and D is the number of feature dimensions; ⊙ indicates element-wise multiplication.

[0014] Preferably, the expression for the feature representation of the inter-frame self-attention enhancement is:

[0015] F = W o Concat(G 1 G 2 ,…,G L )

[0016] in, This represents the feature representation for inter-frame self-attention enhancement, where L is the number of heads in the multi-head attention mechanism, and Concat(·) denotes the matrix concatenation operation. The weight parameters to be learned. d represents the l-th head. v =D / L is the scaling factor.

[0017] Preferably, the process of obtaining short video feature representations with latent semantic information awareness includes,

[0018] A hidden attribute activation mapping unit is constructed to capture the hidden attribute response matrix. The hidden attribute response matrix is ​​then input into a dynamic correlation unit. The dynamic correlation unit obtains the correlation characteristics between hidden attributes by constructing static and dynamic graphs, and finally obtains a feature representation with potential semantic information correlation.

[0019] Preferably, the expression for the short video feature representation with latent semantic information awareness is:

[0020] Z = LeakyReLU(A d HW d ),A d =δ(f conv (H'))

[0021] Where LeakyReLU(·) and δ(·) are both activation functions. These are convolutional layers used for dimensionality transformation.

[0022] This represents the static association of hidden attributes.

[0023] and These are the dynamic graph correlation matrix and the dynamic weight update matrix, respectively.

[0024] This represents a dynamic association of hidden attributes. For the global representation of H;

[0025] It is a feature representation that includes potential semantic relationships;

[0026] D represents the number of hidden attributes. r D represents the number of feature dimensions obtained after static part training. k This represents the number of feature dimensions obtained after training the dynamic part.

[0027] Preferably, the process of obtaining an event category score based on the short video feature representation includes,

[0028] The event category score is obtained by passing the short video feature representation with potential semantic information perception through a global average pooling layer and a normalized exponential function;

[0029] The expression for the event category score is:

[0030]

[0031] in, This represents the event category score, GAP represents the global average pooling layer, and Softmax(·) is the normalized exponential function.

[0032] This invention also provides a short video event detection system based on deep dynamic semantic association, comprising,

[0033] The frame importance assessment module is used to collect short videos and extract the visual features of the short videos. Based on the visual features of the short videos, it obtains and enhances the frame importance information of the visual features to obtain a frame importance score with discriminative power and a short video feature representation after weighting the frame importance scores.

[0034] The inter-frame self-attention enhancement module, connected to the frame importance evaluation module, is used to guide attention learning based on the attention characteristics of the feature space and the short video feature representation weighted by the frame importance score, thereby coordinating the intrinsic correlation between inter-frames and the feature space to obtain the feature representation enhanced by inter-frame self-attention.

[0035] The latent semantic information perception module, connected to the inter-frame self-attention enhancement module, is used to treat the hidden attributes of short video events as nodes and the correlation between the hidden attributes as edges, and construct a specific graph representation of short video samples; and learn the hidden attributes of complex events and their correlations through a hidden attribute association learning network under dynamic graph convolution to obtain a short video feature representation with latent semantic information perception.

[0036] The category score calculation module, connected to the latent semantic information perception module, is used to obtain event category scores based on the short video feature representation and complete the short video event detection task.

[0037] Preferably, the frame importance assessment module includes an indicator vector calculation unit, a weight update unit, an indicator, an encoder, a decoder, a discriminator, and a weight allocation unit;

[0038] The indicator vector calculation unit is used to generate the initial importance weights of short video keyframes;

[0039] The weight update unit and the indicator work together to update the importance weights;

[0040] The encoder and decoder together constitute a variational autoencoder, used to mine potential importance information of samples. At the same time, the decoder and discriminator together constitute a generative adversarial network. The feedback value learned by the discriminator is applied to the weight update unit and the indicator to guide the update of importance weights.

[0041] Preferably, the latent semantic information perception module includes a hidden attribute activation mapping unit and a dynamic correlation unit;

[0042] The hidden attribute activation mapping unit is used to capture the hidden attribute response matrix and input the hidden attribute response matrix into the dynamic correlation unit;

[0043] The dynamic association unit is used to obtain the association characteristics between hidden attributes by constructing static and dynamic graphs, and finally obtain a feature representation with potential semantic information association.

[0044] Compared with the prior art, the present invention has the following advantages and technical effects:

[0045] This invention utilizes a joint structure of variational autoencoder and generative adversarial network to maximize the enhancement of importance information and obtain a frame importance score with discriminative power.

[0046] This invention utilizes importance score-weighted feature representations to guide attention learning, and effectively learns inter-frame self-attention-enhanced feature representations by coordinating the intrinsic correlation between inter-frame and feature spaces;

[0047] This invention fully utilizes the visual features of short videos to obtain frame importance scores with discriminative power, and uses dynamic graph structures to obtain the hidden attributes of complex events and their correlations, ultimately obtaining a short video representation with latent semantic information awareness to achieve short video event detection. Compared with the prior art, this invention achieves short video event detection by fully enhancing frame importance information and constructing a frame importance score-guided inter-frame self-attention enhancement and hidden attribute correlation learning network.

[0048] This invention differs from traditional event explicit semantic learning and traditional static graph convolutional networks. It learns the hidden attributes of complex events and their relationships through a hidden attribute association learning network under dynamic graph convolution, thereby obtaining short video feature representations with latent semantic information awareness. This provides a new approach to solving the problem of short video event detection. Attached Figure Description

[0049] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0050] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the system structure according to an embodiment of the present invention. Detailed Implementation

[0052] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0053] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0054] Example 1

[0055] like Figure 1As shown, the short video event detection method based on deep dynamic semantic association provided by this invention specifically includes the following steps:

[0056] 101: Collect short videos and extract their visual features. Based on the visual features of the short videos, obtain and enhance the frame importance information of the visual features. Obtain frame importance scores with discriminative power and short video feature representations after weighting the frame importance scores.

[0057] Further optimization of the scheme, 201: Without loss of generality, assume that the downsampled short video has T keyframes, and the extracted original visual features are... Where B represents the number of short video samples and D represents the number of feature dimensions. This feature is processed by an indicator vector computation unit consisting of a Bidirectional Long-Short Term Memory (Bi-LSTM) network and average pooling operations. The Bi-LSTM is used to capture the temporal dependencies of the frame sequence in the forward and backward directions, thereby obtaining the importance scores of the short video keyframes. Where s t Let S represent the score of frame t. The weight update unit and the indicator jointly update the frame importance score S. The weight update unit contains a fully connected network and a normalized exponential function Softmax(·), and the indicator is a fully connected network. They work together, and the interaction process is repeated N times, where N is the number of important keyframes to be selected. During the i-th interaction, the probability density distribution of the importance score S of frame T in the current state can be obtained from the Softmax(·) of the weight update unit. Then, random sampling is performed on this distribution. Assuming that the k1∈[1,T] frame is selected during random sampling, a weighting factor is applied to the importance score S of the short video keyframe. w i ab Let be the element in the a∈[1,B] row and b∈[1,T] column of this weight factor, defined as follows:

[0058]

[0059] This factor is designed to ensure that the frames selected by the model earlier have higher importance. After N interactions, a weighting factor is applied to the frames that were not selected. Assuming the frame k2∈[1,T] is ultimately not selected, then the values ​​in column k2 of the RF are all (TN) / T, and the values ​​in the remaining columns are all 1. After N selections, a new short video keyframe score is finally generated:

[0060]

[0061] Where s't Let S' represent the current score in frame t. Multiplying S' by the expanded dimension and feature X yields the feature after applying the importance score.

[0062]

[0063] in, The dimension is obtained by expanding S', where ⊙ represents element-wise multiplication. Subsequently, P passes through an encoder-decoder consisting of a Long Short-Term Memory (LSTM) network and linear layers, producing a reconstructed feature structure with the same dimensions as P but containing the potential importance information of keyframes in the short video. Reconstructed features Together with the original feature X, it passes through a discriminator consisting of an LSTM, a linear layer, and a sigmoid activation function, ultimately outputting a feedback value r for weight updates and indicator training. i :

[0064]

[0065] Among them, L recon Used to guarantee reconstructed features Without losing too much information contained in the original feature X, L recon The mean squared error is used to define this, and is specifically expressed as follows:

[0066]

[0067] The above weight updates are trained based on the following loss function:

[0068]

[0069] Where log_prob(·) is a function that takes the logarithm of the probability density distribution, and its value is a scalar; c i Let be the probability density distribution of the importance score S during the i-th interaction; δ = 0.1 is the entropy regularization coefficient; entropy(·) is the entropy function, and its value is a scalar value used to measure the sparsity of the probability distribution. The smaller the entropy, the more concentrated the probability distribution; α i =d i -v i (i = 1, 2, ..., N) represents the advantage of applying a weighting factor to a specific frame selected by random sampling in the i-th interaction compared to applying the same weighting factor to all frames directly, v i d is the evaluation value of the importance score S obtained through the indicator for frame selection based on weight updates; this value is a scalar value. i This represents the cumulative discriminator feedback value from the i-th interaction to the last interaction, defined as:

[0070]

[0071] Where, r i It is the feedback value generated by the discriminator during the i-th interaction, where r is the feedback value generated by the discriminator. i Obtained from equation (3). γ represents the degree of influence of this feedback value. Applying γ to the power of ki (k∈[i,N]) ensures that the feedback value has the greatest impact on the i-th interaction, with the impact of feedback values ​​from subsequent interactions decreasing sequentially. To ensure that the feedback values ​​have a large overall impact, let γ = 0.99. Finally, the indicator is trained based on the following loss function:

[0072]

[0073] In summary, L act and L instru The combined effect of the two makes c i and α i It is continuously updated until the optimal value is reached.

[0074] 102: Considering the attention characteristics of the feature space, the short video feature representation weighted by the frame importance scores obtained in step 101 is used to guide the learning of attention. This process coordinates the intrinsic correlation between frames and the feature space, and finally obtains a feature representation with more attention characteristics.

[0075] Further optimization of the scheme, 202: This process is inspired by the multi-head attention mechanism. Assuming there are L heads in total, then the l-th head... The definition is as follows:

[0076]

[0077] in, This is the expanded matrix after enhancing the inter-frame importance score, which is the feature matrix obtained by slicing the feature P after enhancing the importance score along the first order. and These are the weight parameters to be learned in the query matrix, key matrix, and value matrix, respectively. k =d v =D / L is the scaling factor. The meaning of the above formula is: The feature space weights corresponding to each key are obtained after Softmax(·), and these weights are then summed. Multiplication yields the feature representation after weight allocation in the feature space. This process coordinates the intrinsic correlation between frames and the feature space, ultimately resulting in a feature representation with more attention-oriented properties.

[0078] F = W o Concat(G 1 G 2 ,…,G L (9)

[0079] Where L is the number of heads in the multi-head attention mechanism, and Concat(·) is the matrix concatenation operation. These are the weight parameters to be learned.

[0080] 103: To learn the latent semantics of short video events and their interrelationships, the hidden attributes of events are treated as nodes, and the correlation between hidden attributes is treated as edges. A specific graph representation is constructed for each short video sample. A latent attribute association learning network under dynamic graph convolution learns the latent attributes of complex events and their interrelationships, ultimately obtaining a short video feature representation with latent semantic information awareness.

[0081] To further optimize the scheme, the hidden attribute response matrix is ​​first captured by constructing a hidden attribute activation mapping unit: in Let e ​​represent the number of hidden attributes, and D represent the number of feature dimensions. Each hidden attribute response vector is e. i The calculation formula is as follows:

[0082]

[0083] in, The output of the inter-frame self-attention enhancement module contains feature map representations with more attentional characteristics; F q and These are the feature representation and hidden attribute activation map of the q-th frame for each sample; m i The activation map of hidden attributes captured when using a convolutional filter as a response detector:

[0084] Next, E is input into the dynamic correlation unit. This unit acquires the correlation characteristics between hidden attributes by constructing static and dynamic graphs, ultimately obtaining a feature representation Z with latent semantic information correlation. The static graph is used to learn the overall relationship between the hidden attributes of all samples, and the dynamic graph is used to learn the specific attribute relationships contained in each sample. Specifically, this unit consists of a static part and a dynamic part. The static part is defined as: H = LeakyReLU(A s EW s ), where LeakyReLU(·) is the activation function, This is the hidden attribute response matrix obtained from the hidden attribute activation mapping unit. and These are the static correlation matrix and the static weight update matrix, respectively. Both are randomly initialized and continuously updated during training. D represents the number of hidden attributes, and D represents the number of feature dimensions. r This represents the number of feature dimensions obtained after training the static part. The dynamic part is defined as: Z = LeakyReLU(A d HW d ),A d =δ(f conv (H')), where LeakyReLU(·) and δ(·) are both activation functions. It is a convolutional layer used for dimension transformation. This is a static representation. and D represents the learned dynamic graph correlation matrix and dynamic weight update matrix, respectively. Both are randomly initialized and continuously updated during the learning process. Their synergistic effect continuously transmits dynamic information related to the hidden attributes to H, thereby updating H. k This represents the number of feature dimensions obtained after training the dynamic part. A matrix was constructed to obtain the dynamic characteristics. This matrix consists of H and its global representation. This is achieved through combination. Ultimately, a feature representation containing latent semantic relationships is obtained.

[0085] 104: Use the short video feature representation learned above to obtain event category scores and complete the short video event detection task.

[0086] Further optimization scheme 204: The feature representation obtained after the above learning that contains latent semantic associations The event category score is obtained after passing through a global average pooling layer and a normalized exponential function Softmax(·). Where R represents the number of short video event categories. The classification loss function constructed based on cross-entropy in this part is shown below:

[0087]

[0088] Where σ(·) is the sigmoid function; The value is 1 if the i-th sample belongs to the j-th class, and 0 otherwise; Let be the probability value that the i-th sample is predicted to be of class j.

[0089] In summary, the loss function of this model is ultimately structured as follows:

[0090] L=αL act +βL recon +γL dcm (12)

[0091] Where α, β, and γ are hyperparameters that balance the weights among the various losses.

[0092] The model is trained by setting appropriate parameters, and the final results are evaluated using accuracy, recall, and mean average precision (mAP) as metrics.

[0093] Example 2

[0094] The dataset consisted of 20,231 short videos crawled from Flickr, each no longer than 30 seconds, covering 20 event types. 80% of the short videos were randomly selected as the training set, and the remainder as the test set. Visual features were extracted from these videos and used to verify the effectiveness of the invention for short video event detection.

[0095] Example 3

[0096] like Figure 2 As shown, this invention also provides a short video event detection system based on deep dynamic semantic association, comprising:

[0097] The frame importance assessment module is used to collect short videos and extract their visual features. Based on the visual features of the short videos, it obtains and enhances the frame importance information of the visual features, and obtains a frame importance score with discriminative power and a short video feature representation after weighting the frame importance scores.

[0098] The inter-frame self-attention enhancement module, connected to the frame importance evaluation module, is used to guide attention learning by using the short video feature representation weighted by frame importance scores based on the attention characteristics of the feature space, and to obtain the feature representation enhanced by inter-frame self-attention by coordinating the intrinsic correlation between the inter-frame and feature space.

[0099] The latent semantic information perception module, connected to the inter-frame self-attention enhancement module, is used to treat the hidden attributes of short video events as nodes and the correlation between hidden attributes as edges to construct a specific graph representation of short video samples. The latent attribute association learning network under dynamic graph convolution learns the hidden attributes of complex events and their correlations to obtain a short video feature representation with latent semantic information perception.

[0100] The category score calculation module, connected to the latent semantic information perception module, is used to obtain event category scores based on the feature representation of short videos, thereby completing the short video event detection task.

[0101] Further optimization of the scheme reveals that the frame importance assessment module mainly consists of an indicator vector calculation unit, a weight update unit, an indicator, an encoder, a decoder, a discriminator, and a weight allocation unit. In this module, the indicator vector calculation unit generates the initial importance weights for keyframes in the short video; the weight update unit and the indicator work together to update the importance weights; the encoder and decoder together form a variational autoencoder to mine the potential importance information of the samples. Simultaneously, the decoder also acts as a generator, forming a generative adversarial network with the discriminator, ensuring that the latent feature representation trained by the variational autoencoder does not lose too much information contained in the original features. Finally, the feedback value learned by the discriminator is applied to the weight update and indicator parts to guide the update of the importance weights.

[0102] Further optimization of the scheme: the latent semantic information perception module includes a hidden attribute activation mapping unit and a dynamic correlation unit;

[0103] The hidden attribute activation mapping unit is used to capture the hidden attribute response matrix and input the hidden attribute response matrix into the dynamic correlation unit;

[0104] Dynamic association units are used to obtain the association characteristics between hidden attributes by constructing static and dynamic graphs, and finally obtain feature representations with potential semantic information association.

[0105] Example 4

[0106] The short video event detection device based on deep dynamic semantic association includes a processor and a memory. The memory stores program instructions. The processor calls the program instructions stored in the memory to make the device execute the specific steps of the short video event detection method based on deep dynamic semantic association described above, which will not be repeated here.

[0107] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A short video event detection method based on deep dynamic semantic association, characterized in that, include: Collect short videos and extract their visual features. Based on the visual features of the short videos, obtain and enhance the frame importance information of the visual features to obtain a frame importance score with discriminative power and a short video feature representation after weighting the frame importance scores. Based on the attention characteristics of the feature space, the short video feature representation weighted by the frame importance score is used to guide the learning of attention, and the intrinsic correlation between frames and the feature space is coordinated to obtain the feature representation with enhanced self-attention between frames. The hidden attributes of short video events are treated as nodes, and the correlation between these hidden attributes is treated as edges, thus constructing a specific graph representation of short video samples. A hidden attribute association learning network under dynamic graph convolution learns the hidden attributes of complex events and their correlations, thereby obtaining a short video feature representation with latent semantic information awareness. The event category score is obtained based on the short video feature representation to complete the short video event detection task. The process of acquiring and enhancing frame importance information based on the visual features of the short video, and obtaining a frame importance score with discriminative power and a weighted representation of the short video features based on the frame importance scores, includes the following steps: By maximally enhancing frame importance information through a joint structure of variational autoencoder and generative adversarial network, we obtain a discriminative frame importance score and a short video feature representation after weighting the frame importance score. The expression for the short video feature representation after weighting the frame importance scores is: in, The frame importance score is updated after passing through the embedded variational autoencoder and generative adversarial network joint structure. The representation after expanding the dimensions; The extracted original visual features of the short video; The feature representation weighted by importance scores; The number of keyframes in a short video. The number of short video samples. The number of feature dimensions; This indicates element-wise multiplication; The process of obtaining short video feature representations with latent semantic information awareness includes, A hidden attribute activation mapping unit is constructed to capture the hidden attribute response matrix. The hidden attribute response matrix is ​​then input into a dynamic correlation unit. The dynamic correlation unit obtains the correlation characteristics between hidden attributes by constructing static and dynamic graphs, and finally obtains a feature representation with potential semantic information correlation.

2. The short video event detection method based on deep dynamic semantic association according to claim 1, characterized in that, The expression for the feature representation of the inter-frame self-attention enhancement is: in, This represents the feature representation for inter-frame self-attention enhancement, where L is the number of heads in the multi-head attention mechanism. This represents the matrix concatenation operation. The weight parameters to be learned. Indicates the first Size, This is the scaling factor.

3. The short video event detection method based on deep dynamic semantic association according to claim 1, characterized in that, The expression for short video feature representation with latent semantic information awareness is as follows: in, and All are activation functions. These are convolutional layers used for dimensionality transformation. This represents the static association of hidden attributes. and These are the dynamic graph correlation matrix and the dynamic weight update matrix, respectively. This represents a dynamic association of hidden attributes. for The global representation; It is a feature representation that includes potential semantic relationships; Indicates the number of hidden attributes. This represents the number of feature dimensions obtained after static part training. This represents the number of feature dimensions obtained after training the dynamic part.

4. The short video event detection method based on deep dynamic semantic association according to claim 1, characterized in that, The process of obtaining an event category score based on the short video feature representation includes: The event category score is obtained by passing the short video feature representation with potential semantic information perception through a global average pooling layer and a normalized exponential function; The expression for the event category score is: in, This represents the event category score, and GAP represents the global average pooling layer. It is a normalized exponential function.

5. A short video event detection system based on deep dynamic semantic association, characterized in that, include, The frame importance assessment module is used to collect short videos and extract the visual features of the short videos. Based on the visual features of the short videos, it obtains and enhances the frame importance information of the visual features to obtain a frame importance score with discriminative power and a short video feature representation after weighting the frame importance scores. The inter-frame self-attention enhancement module, connected to the frame importance evaluation module, is used to guide attention learning based on the attention characteristics of the feature space and the short video feature representation weighted by the frame importance score, thereby coordinating the intrinsic correlation between inter-frames and the feature space to obtain the feature representation enhanced by inter-frame self-attention. The latent semantic information perception module, connected to the inter-frame self-attention enhancement module, is used to treat the hidden attributes of short video events as nodes and the correlation between the hidden attributes as edges, and construct a specific graph representation of short video samples; and learn the hidden attributes of complex events and their correlations through a hidden attribute association learning network under dynamic graph convolution to obtain a short video feature representation with latent semantic information perception. The category score calculation module is connected to the latent semantic information perception module and is used to obtain the event category score based on the short video feature representation to complete the short video event detection task. The process of acquiring and enhancing frame importance information based on the visual features of the short video, and obtaining a frame importance score with discriminative power and a short video feature representation weighted by the frame importance score, includes the following steps: By maximally enhancing frame importance information through a joint structure of variational autoencoder and generative adversarial network, we obtain a discriminative frame importance score and a short video feature representation after weighting the frame importance score. The expression for the short video feature representation after weighting the frame importance scores is: in, The frame importance score is updated after passing through the embedded variational autoencoder and generative adversarial network joint structure. The representation after expanding the dimensions; The extracted original visual features of the short video; The feature representation weighted by importance scores; The number of keyframes in a short video. The number of short video samples. The number of feature dimensions; This indicates element-wise multiplication; The process of obtaining short video feature representations with latent semantic information awareness includes, A hidden attribute activation mapping unit is constructed to capture the hidden attribute response matrix. The hidden attribute response matrix is ​​then input into a dynamic correlation unit. The dynamic correlation unit obtains the correlation characteristics between hidden attributes by constructing static and dynamic graphs, and finally obtains a feature representation with potential semantic information correlation.

6. The short video event detection system based on deep dynamic semantic association according to claim 5, characterized in that, The frame importance assessment module includes an indicator vector calculation unit, a weight update unit, an indicator, an encoder, a decoder, a discriminator, and a weight allocation unit. The indicator vector calculation unit is used to generate the initial importance weights of short video keyframes; The weight update unit and the indicator work together to update the importance weights; The encoder and decoder together constitute a variational autoencoder, used to mine potential importance information of samples. At the same time, the decoder and discriminator together constitute a generative adversarial network. The feedback value learned by the discriminator is applied to the weight update unit and the indicator to guide the update of importance weights.

7. The short video event detection system based on deep dynamic semantic association according to claim 5, characterized in that, The latent semantic information perception module includes a hidden attribute activation mapping unit and a dynamic correlation unit; The hidden attribute activation mapping unit is used to capture the hidden attribute response matrix and input the hidden attribute response matrix into the dynamic correlation unit; The dynamic association unit is used to obtain the association characteristics between hidden attributes by constructing static and dynamic graphs, and finally obtain a feature representation with potential semantic information association.

Images