Long-tail video detection method based on cross-modality

Abstract

The invention provides a long-tail video detection method based on cross-modality, and solves the problem of low tail category recognition rate caused by the fact that a traditional video detection model is easily dominated by a head category. Taking video and audio data as input, and extracting audio features, visual features and visual motion features; the features are recombined through a data enhancement strategy, and long-tail distribution is balanced; guiding a visual attention mechanism by using audio features, and enhancing visual feature representation related to the audio; capturing time sequence information through time sequence modeling, and screening positive sample aggregation features based on correlation analysis; dynamically constructing a high-confidence branch and a low-confidence branch, and performing multi-branch feature fusion in combination with motion features; classification representation is optimized through a time sequence feature encoder, prediction results of all branches are superposed in proportion, and the classification performance of tail categories is improved. According to the method, through cross-modal feature collaborative optimization and a dynamic branch weighting strategy, the recognition accuracy of the tail category in the long-tail video data is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of video data processing and computer vision, and particularly relates to a cross-modal based long-tail video detection method. Background Art

[0002] Currently, with the wide application of artificial intelligence technology in the field of video analysis, video detection, as one of the core tasks, has become a research focus. Traditional video detection methods mainly rely on the extraction of visual features and temporal modeling (such as 3D-CNN, LSTM), and achieve high accuracy on balanced datasets. However, video data in real scenarios generally exhibits long-tail distribution characteristics, that is, a small number of common categories occupy a large number of samples, while the samples of most rare categories are extremely scarce, resulting in traditional models being severely biased towards head categories.

[0003] Although existing long-tail video detection methods alleviate the problem of class imbalance through strategies such as resampling and reweighting, there are still significant deficiencies in the aspects of multi-modal feature fusion and dynamic temporal modeling. Traditional methods usually rely on a single visual modality, ignoring the guiding role of audio information for key actions, resulting in being sensitive to background noise and difficult to locate sparse key frames of tail classes. In addition, existing models do not consider that video data usually only provides video-level labels, while real actions may only exist in a few key frames, resulting in irrelevant features interfering with the classification accuracy. Summary of the Invention

[0004] To address the above problems, the present invention proposes a cross-modal based long-tail video detection method, which systematically solves the above challenges through an audio-guided cross-modal attention and adaptive branch fusion mechanism. The cross-attention and correlation analysis between audio features and visual features are introduced to suppress background noise and enhance the expression of strongly relevant key frames. Dual-branch confidence separation is adopted, combined with dynamic frame-level weight allocation, to accurately distinguish high / low confidence visual segments and avoid the suppression of tail classes by head class features. The present invention significantly improves the classification accuracy of tail classes while ensuring the accuracy of head classes.

[0005] To achieve the above invention objective, the present invention proposes a cross-modal based long-tail video detection method, including the following steps:

[0006] S1) Use a shooting device to obtain video data and corresponding audio data related to the task categories, and perform category annotation on them to obtain category label y;

[0007] S2) Perform frame extraction on the obtained video data to extract video frame files, use a VGG-like network model to extract audio feature A from the audio data, use a ResNet50 network model to extract visual feature V from the video frame files, and extract visual motion feature VM;

[0008] S3) Perform data augmentation on the extracted audio features A, visual features V, visual motion features VM, and class label y to obtain the data-augmented audio features A′, visual features V′, visual motion features VM′, and class label y′;

[0009] S4) Calculate the frame attention weights using the visual features V′, and perform cross-attention mechanism processing on the audio features A′ with the visual features V′ and visual motion features VM′ respectively, using the audio to enhance the visual part, to obtain the visual features V C , visual motion features VM C ;

[0010] S5) Respectively pass the audio features A′, visual features V C , visual motion features VM C through temporal modeling to capture time series information, to obtain the audio features A L , visual features V L , visual motion features VM L ;

[0011] S6) Respectively perform correlation analysis on the audio features A L with the visual features V L , visual motion features VM L , filter out the parts with low correlation between the audio features and the visual features, and then further fuse the audio features and the visual features, to obtain the visual features V R , visual motion features VM R ;

[0012] S7) Calculate the high-confidence branch features VH R and low-confidence branch features VL R for the visual features V R using the frame attention weights in S4), and reassign the frame weight coefficients for each frame feature of the high-confidence branch and the low-confidence branch;

[0013] S8) Respectively pass the high-confidence branch features VH R , low-confidence branch features VL R , visual motion features VM R through temporal feature encoding to convert them into task-related high-confidence branch temporal features VH S , low-confidence branch temporal features VL S , visual motion temporal features VM S to enhance the adaptability to the long-tail video recognition task;

[0014] S9) Combine the low-confidence branch temporal features VL S with the visual motion temporal features VM SPerform splicing and fusion to obtain the fused feature VF S And the high-confidence branch temporal feature VH S Are respectively converted into the prediction results for each category, and the prediction results for each category are superimposed proportionally to obtain the final prediction results for each category of this video;

[0015] S10) To improve the accuracy of the detection results, calculate the gradient information of the loss function with respect to each trainable parameter through the backpropagation algorithm, and use the gradient descent algorithm to update the model parameters; Iterate N times to optimize the model, and save the model when the average value mAP of the average precision for each category is the highest. Among them, the loss function Among them, L cls Represents the classification loss, C is the total number of categories for classification.

[0016] Furthermore, the above-mentioned video data, audio data and their annotation methods in S1) include the following steps:

[0017] S1.1) Design diverse dimensions for time, space, and objects, use a 4K camera, a directional microphone, and combine a timecode generator to set a unified timestamp for all devices, and perform standardized process shooting;

[0018] S1.2) Manually verify whether each video content is related to the initial shooting label, and use majority voting to decide whether to retain the video;

[0019] S1.3) Save the audio file name, video file name, and category name to ensure data traceability.

[0020] Furthermore, the above-mentioned S2) includes the following steps:

[0021] S2.1) Use the VGG-like network model pre-trained on AudioSet to extract the features of each second of audio segments from the audio data obtained in S1 to obtain audio features Where T represents the duration of the video data segment in seconds, and d a Represents the dimension of the audio encoding feature, which is the number of output feature maps of the VGG-like network model;

[0022] S2.2) Use the ResNet50 network model pre-trained on ImageNet to extract the feature maps of each second of sampled frames from the video frame files to obtain visual features Where k represents the spatial dimension size of the feature maps extracted from each frame, and d v Represents the dimension of the visual encoding feature, which is the number of output feature maps of the ResNet50 network model;

[0023] S2.3) Extract the motion features of the visual feature V. By calculating the feature differences between adjacent frames, capture the temporal motion information and concatenate the global average features. Generate visual motion features where v t represents the feature at the t-th time step, and concat() represents the concatenation operation.

[0024] Furthermore, the data augmentation strategy in the above S3) includes the following steps:

[0025] S3.1) Randomly extract two video features V i and V j from the video feature file in S2.2), and generate a new mixed feature according to the following steps. Calculate the dynamic sampling ratio for the rAP corresponding to the updated video features V i and V j of the corresponding categories. The formula is as follows:

[0026]

[0027] where rAP i and rAP j are the running average precisions of the i-th and j-th categories to which the video features V i and V j belong, with an initial value of 0. β is the dynamic sampling ratio, and its value range is [0,1], reflecting the relative recognition ability of the network for categories i and j, with an initial value of 0.5;

[0028] S3.2) Determine the sampling frames L i and L j of the two video features V i and V j based on β. The formula is as follows:

[0029]

[0030] where L is the preset total number of frames, is the floor symbol, and L i and L j are the number of frames sampled from the video features V i and V j respectively;

[0031] S3.3) Uniformly sample L i and L j frames from the audio feature A, visual feature V, and visual motion feature VM, and concatenate the sampled features along the spatial dimension T to generate audio, visual, and visual motion mixed features respectively. The formula is as follows:

[0032]

[0033] wherein is the video V i a frame sequence of audio, visual, and visual motion features evenly sampled for L i frames, is the video V j a frame sequence of audio, visual, and visual motion features evenly sampled for L j frames, are respectively the generated audio, visual, and visual motion mixed features;

[0034] S3.4) Based on β, determine the mixed feature label y′ of the two video features V i , V j as follows:

[0035] y′ = (1 - β) × y i + β × y j

[0036] where y i , y j are respectively the label vectors of categories i and j, and y′ is the mixed label vector.

[0037] Furthermore, the steps of calculating the frame attention weight and the cross - attention mechanism between audio and visual features in the above step S4) include the following steps:

[0038] S4.1) Take the average value of the features at each position of the visual feature in the K×K spatial dimension, so as to aggregate the features of each spatial block of K×K in each time period into a d v - dimensional vector; perform global pooling operation based on the d v spatial dimension, compress the input features along the channel dimension into a global feature vector; then use two - layer fully - connected layers, combined with ReLU and Sigmoid activation functions, to generate the frame attention weight; finally, expand the frame attention weight to d l dimensions, denoted as

[0039] S4.2) Project the audio feature A′ and the visual feature V′ into the same latent space through linear transformation respectively; then calculate the association strength between the audio feature and each visual spatial position in the form of matrix addition; perform Softmax normalization on the similarity scores to generate the cross - attention weight in the spatial dimension, representing the correlation between different visual regions and the current audio;

[0040] S4.3) Based on the generated cross - attention weight, perform dynamic spatial weighting on the visual feature V′: For each time step of K 2The spatial position features are weighted and summed according to each spatial region weight coefficient to output the audio-guided visual features Visual motion features

[0041] Further, the correlation analysis in the above step S6) includes the following steps:

[0042] S6.1) Map the audio feature A L through two linear transformations to map out two audio branch features A L ′ and A L ″, and map the visual feature V L through two linear transformations to map out two visual branch features V L ′ and V L ″. Use the audio branch feature A L ′ and the visual branch feature V L ′ to calculate the similarity matrix β va of vision and audio as well as the similarity matrix β av of audio and vision. The formula is as follows:

[0043]

[0044] β av =(β va ) T

[0045] where: is the linear transformation weight matrix, d l is the dimension of the audio and visual features,

[0046] S6.2) Introduce an adjustable threshold τ for hard binary processing, only retain the strongly positive correlated audio-visual pairs with a similarity higher than τ, and perform Softmax normalization on the filtered matrix to convert it into a probability distribution. The formula is as follows:

[0047]

[0048] γ va′ =Softmax(γ va ), γ av′ =Softmax(γ av )

[0049] where γ va′ is the strongly correlated matrix of vision and audio, γ av′ is the strongly correlated matrix of audio and vision, is the indicator function, which outputs 1 when the input is greater than or equal to 0, otherwise outputs 0, and τ is a hyperparameter,

[0050] S6.3) Strong correlation matrix γ va′ , γ av′ are respectively weighted and summed with the audio branch feature A L ″ and the visual branch feature V L ″ to obtain the audio aggregated visual feature and the visual aggregated audio feature; a residual connection structure is adopted, and the feature after temporal modeling is added to the cross-modal aggregated feature to obtain the visually enhanced audio feature A G and the audio enhanced visual feature V G . Finally, the two parts of features are added and further fused in a certain proportion to obtain the enhanced visual feature The formula is as follows:

[0051] A G =γ av ′V L ″ + A L , V G =γ va ′A L ″ + V L

[0052] V R =ηV G + (1 - η)A G

[0053] where η is a hyperparameter, representing the proportion of the audio enhanced visual feature V R in the enhanced visual feature V G ;

[0054] S6.4) The audio feature A L and the visual motion feature VM L are calculated according to the steps of S6.1)-S6.3) to obtain the enhanced visual motion feature

[0055] Furthermore, the dynamic construction of the double-branch representation in the above step S7) includes the following steps:

[0056] S7.1) Based on the frame attention weights calculated in S4.1), calculate the high-confidence branch feature VH R =V R ⊙H, where ⊙ represents element-wise multiplication;

[0057] S7.2) Calculate the low-confidence branch feature where β is a hyperparameter to prevent the denominator from being zero, represents element-wise division, forcing it to focus on the frames ignored in 7.1).

[0058] Furthermore, the temporal feature encoder in the above step S8) includes the following steps:

[0059] S8.1) For the high-confidence branch features Low-confidence branch features Enhanced visual motion features Use the MLP layer to perform non-linear transformation respectively to enhance the model's ability to express complex temporal patterns in long-tail videos; then perform 3-kernel one-dimensional convolution on its spatial dimension T respectively to extract the correlation features VH R ′, VL R ′, VM R ′, and perform residual connections with the high-confidence branch features VH R , low-confidence branch features VL R , and enhanced visual motion features VM R respectively. The formula is as follows:

[0060] VH R ″ = VH R + VH R ′

[0061] VL R ″ = VL R + VL R ′

[0062] VM R ″ = VM R + VM R ′

[0063] Where VH R ″ is the high-confidence branch position enhanced feature, VL R ″ is the low-confidence branch position enhanced feature, VM R ″ is the visual motion position enhanced feature.

[0064] S8.2) Normalize VH R ″ which is the high-confidence branch position enhanced feature, VL R ″ which is the low-confidence branch position enhanced feature, and VM R ″ which is the visual motion position enhanced feature respectively, and take the average value of the spatial dimension T to obtain the high-confidence branch embedding feature VH S , low-confidence branch embedding feature VL S , and visual motion embedding feature VM S .

[0065] Furthermore, the calculation of the final result in the above step S9) includes the following steps:

[0066] S9.1) Pass the high-confidence branch embedding feature VH S through the fully connected layer and sigmoid activation function respectively to convert it into the prediction result of each category where N represents the number of categories, and among them

[0067] S9.2) Embed the low-confidence branch into the feature VL S and the visual motion embedding feature VM S Concatenate them in the spatial dimension T, reduce the dimension through a fully connected layer, take the average value in the spatial dimension T, and then transform it into the prediction result for each category through a fully connected layer and a sigmoid activation function

[0068] S9.3) The prediction result and the prediction result are superimposed proportionally to obtain the final prediction result of each category for this video The formula is as follows:

[0069]

[0070] where κ is a hyperparameter, representing the prediction result of the high-confidence branch accounting for the proportion of the final prediction result.

[0071] S9.4) Sort the values of the prediction results of each video from high to low, count the cumulative number of correct categories in the sorted order, and calculate the precision Precision i at each position, take the average value of the precisions corresponding to the positions of the correct categories to obtain the running average precision rAP i of this category. The formula is as follows:

[0072]

[0073] where Precision i is the accuracy rate calculated for each category of this video; Precision T is the Precision of the correctly labeled category by the category label y i ; N T represents the number of correctly labeled categories by the category label y of this video, and calculate the running average precision

[0074] for all categories in turn. Further, in the above step S10), the gradient descent algorithm is used to update the model parameters, including the following parameters: the running average precision rAP of each category; the fully connected layer parameters, frame attention weights, cross-attention weights; temporal modeling parameters; linear transformation weight matrix MLP layer parameters, one-dimensional convolutional layer parameters, normalization layer parameters.

[0075] The beneficial technical effects of the present invention are as follows:

[0076] 1. Dynamically adjust the hybrid sampling ratio β according to the real-time tracked class average precision (rAP), so that the model adaptively focuses on the difficult-to-recognize tail classes during training, alleviating the class imbalance problem caused by the long-tail distribution.

[0077] 2. Through the cross-attention calculation of audio features and visual features, dynamically screen the visual regions strongly related to audio, use the hard threshold of the cross-modal similarity matrix to screen the visual segments strongly related to audio, and aggregate the positive sample features by combining residual connections, effectively filtering out the interference of irrelevant temporal segments.

[0078] 3. Separate high-confidence and low-confidence features based on frame attention weights, splice and fuse the low-confidence branch with motion features, and use the motion temporal information to supplement the discriminative ability of the low-confidence branch, reducing misclassification caused by missed detection of key frames. Description of the Drawings

[0079] Figure 1 Flowchart of a cross-modal based long-tail video detection method. Detailed Embodiments

[0080] The present invention will be further described below in conjunction with the drawings and specific embodiments. It should be noted that only one optimal technical solution is used here to elaborate in detail the technical solution and design principle of the present invention, but the protection scope of the present invention is not limited to this.

[0081] The described embodiments are the preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Without departing from the essential content of the present invention, any obvious improvements, substitutions or modifications made by those skilled in the art shall fall within the protection scope of the present invention.

[0082] As Figure 1 shown, the present invention is a cross-modal based long-tail video detection method, including the following steps:

[0083] S1) Use a shooting device to obtain video data and corresponding audio data related to the task categories, and perform category annotation on them to obtain category labels y.

[0084] As a preferred embodiment of the present invention, the method for obtaining and annotating the video data and audio data includes the following steps:

[0085] S1.1) Design diverse dimensions for time, space, and objects, use a 4K camera, a directional microphone and a timecode generator to set a unified timestamp for all devices, and perform standardized process shooting;

[0086] S1.2) Manually verify whether each video content is related to the initial shooting tags, and use majority voting to determine whether the video is retained;

[0087] S1.3) Save the audio file name, video file name, and category name to ensure data traceability.

[0088] S2) Perform frame extraction on the obtained video data to extract video frame files. Use a VGG-like network model to extract audio feature A from the audio data, and use a ResNet50 network model to extract visual feature V from the video frame files, and extract visual motion feature VM.

[0089] As a preferred embodiment of the present invention, the extraction of audio, visual, and visual motion features includes the following steps:

[0090] S2.1) Use a VGG-like network model pre-trained on AudioSet to extract the features of each second of audio segments from the audio data obtained in S1 to obtain audio feature where T represents the duration of the video data segment in seconds, and d a represents the dimension of the audio encoding feature, which is the number of output feature maps of the VGG-like network model;

[0091] S2.2) Use a ResNet50 network model pre-trained on ImageNet to extract the feature maps of each second of sampled frames from the video frame files to obtain visual feature where k represents the spatial dimension size of the feature maps extracted from each frame, and d v represents the dimension of the visual encoding feature, which is the number of output feature maps of the ResNet50 network model;

[0092] S2.3) Perform motion feature extraction on the visual feature V. By calculating the feature differences between adjacent frames, temporal motion information is captured, and the global average feature is concatenated to generate visual motion feature where v t represents the feature at the t-th time step, and concat() represents the concatenation operation.

[0093] S3) Perform data augmentation on the extracted audio feature A, visual feature V, visual motion feature VM, and class label y to obtain the data-augmented audio feature A′, visual feature V′, visual motion feature VM′, and class label y′.

[0094] As a preferred embodiment of the present invention, the data augmentation strategy includes the following steps:

[0095] S3.1) Randomly extract two video features V from the video feature files in S2.2) i, V j , generate a new hybrid feature according to the following steps. Update the video feature V according to S10) i , V j Calculate the dynamic sampling ratio for the corresponding categories of rAP, and the formula is as follows:

[0096]

[0097] where rAP i and rAP j are the running average precisions of the i-th and j-th categories to which the video features V i , V j belong, and the initial value is 0. β is the dynamic sampling ratio, and its value range is [0, 1], which reflects the relative recognition ability of the network for categories i and j, and the initial value is 0.5;

[0098] S3.2) Determine the sampling frame numbers L i , V j of the two video features V i , L j , and the formula is as follows:

[0099]

[0100] L i = L - L j

[0101] where L is the preset total number of frames, is the floor symbol, and L i , L j are the frame numbers sampled from the video features V i and V j respectively;

[0102] S3.3) Uniformly sample L i and L j frames from the audio feature A, visual feature V, and visual motion feature VM, and splice the sampled features along the spatial dimension T to generate audio, visual, and visual motion hybrid features respectively. The formula is as follows:

[0103]

[0104] where is the frame sequence of the audio, visual, and visual motion features of the video V i uniformly sampled with L i frames, is the frame sequence of the audio, visual, and visual motion features of the video V j uniformly sampled with L j frames, The generated audio, visual, and visual-motor hybrid features respectively;

[0105] S3.4) Determine the hybrid feature label y' of the two video features V i and V j based on β, and the formula is as follows:

[0106] y' = (1 - β) × y i + β × y j

[0107] where y i and y j are the label vectors of categories i and j respectively, and y' is the hybrid label vector.

[0108] S4) Calculate the frame attention weight using the visual feature V', and perform cross-attention mechanism processing on the audio feature A' with the visual feature V' and the visual-motor feature VM' respectively, and use the audio to enhance the visual part to obtain the visual feature V C and the visual-motor feature VM C .

[0109] As a preferred embodiment of the present invention, the calculation of the frame attention weight and the cross-attention mechanism between the audio and visual features include the following steps:

[0110] S4.1) Take the average value of the features at each position of the visual feature in the K×K spatial dimension, so as to aggregate the features of each spatial block of K×K in each time period into a d v -dimensional vector. Perform global pooling operation on it based on the d v spatial dimension, and compress the input feature along the channel dimension into a global feature vector. Then use two fully connected layers, cooperate with the ReLU and Sigmoid activation functions, to generate the frame attention weight. Finally, expand the frame attention weight to d l dimensions, denoted as

[0111] S4.2) Project the audio feature A' and the visual feature V' into the same latent space through linear transformation respectively. Subsequently, calculate the association strength between the audio feature and each visual spatial position in the form of matrix addition. Perform Softmax normalization on the similarity score to generate the cross-attention weight in the spatial dimension, representing the correlation between different visual regions and the current audio;

[0112] S4.3) Based on the generated cross-attention weight, perform dynamic spatial weighting on the visual feature V': Weight the features of K 2 spatial positions (K×K grid) at each time step by the weight coefficient of each spatial region and sum them up, and output the audio-guided visual feature Visual-motor feature

[0113] S5) Subject the audio feature A′ and the visual feature V C , and the visual motion feature VM C to temporal modeling respectively to capture time series information, obtaining the audio feature A L , the visual feature V L , and the visual motion feature VM L ;

[0114] S6) Correlate the audio feature A L with the visual feature V L and the visual motion feature VM L respectively through correlation analysis, filter out the parts with low correlation between the audio feature and the visual feature, and then further fuse the audio feature and the visual feature to obtain the visual feature V R , and the visual motion feature VM R .

[0115] As a preferred embodiment of the present invention, the correlation analysis module includes the following steps:

[0116] S6.1) Map the audio feature A L through two linear transformations to map out two audio branch features A L ′ and A L ″, map the visual feature V L through two linear transformations to map out two visual branch features V L ′ and V L ″, and use the audio branch feature A L ′ and the visual branch feature V L ′ to calculate the visual-audio similarity matrix β va and the audio-visual similarity matrix β av , and the formula is as follows:

[0117]

[0118] β av =(β va ) T

[0119] where: is the linear transformation weight matrix, d l is the dimension of the audio and visual features,

[0120] S6.2) Introduce an adjustable threshold τ for hard binary processing, only retain the strongly positively correlated audio-visual pairs with a similarity higher than τ, and perform Softmax normalization on the filtered matrix to convert it into a probability distribution, and the formula is as follows:

[0121]

[0122] γ va′ = Softmax(γ va ), γ av′ = Softmax(γ av )

[0123] where γ va′ is the strongly correlated matrix of vision and audio, and γ av′ is the strongly correlated matrix of audio and vision. is the indicator function, which outputs 1 when the input is greater than or equal to 0, otherwise outputs 0, and τ is the hyperparameter.

[0124] S6.3) The strongly correlated matrices γ va′ , γ av′ are respectively weighted and summed with the audio branch feature A L ″ and the vision branch feature V L ″ to obtain the audio aggregated vision feature and the vision aggregated audio feature; the residual connection structure is adopted, and the features after temporal modeling are added to the cross-modal aggregated features to obtain the vision enhanced audio feature A G and the audio enhanced vision feature V G . Finally, the two parts of features are added and further fused in a certain proportion to obtain the enhanced vision feature The formula is as follows:

[0125] A G = γ av′ V L ″ + A L , V G = γ va′ A L ″ + V L

[0126] V R = ηV G + (1 - η)A G

[0127] where η is the hyperparameter, representing the proportion of the audio enhanced vision feature V R in the enhanced vision feature V G ;

[0128] S6.4) The audio feature A L and the vision motion feature VM L are calculated according to the steps in 6.1)-6.3) to obtain the enhanced vision motion feature

[0129] S7) Use the frame attention weights in S4) for the vision feature V RCalculate the high-confidence branch feature VH R and the low-confidence branch feature VL R , and reassign the frame weight coefficients to the features of each frame of the high-confidence branch and the low-confidence branch.

[0130] As a preferred embodiment of the present invention, the dynamic construction of the dual-branch representation includes the following steps:

[0131] S7.1) Based on the frame attention weights calculated in S4.1), calculate the high-confidence branch feature VH R =V R ⊙H, where ⊙ represents element-wise multiplication;

[0132] S7.2) Calculate the low-confidence branch feature where β is a hyperparameter to prevent the denominator from being zero, represents element-wise division, forcing it to focus on the frames ignored in 7.1).

[0133] S8) Pass the high-confidence branch feature VH R , the low-confidence branch feature VL R , and the visual motion feature VM R through the temporal feature encoding respectively to convert them into the task-related high-confidence branch temporal feature VH S , the low-confidence branch temporal feature VL S , and the visual motion temporal feature VM S to enhance the adaptability to the long-tail video recognition task.

[0134] As a preferred embodiment of the present invention, the temporal feature encoder includes the following steps:

[0135] S8.1) For the high-confidence branch feature the low-confidence branch feature the enhanced visual motion feature perform non-linear transformation using the MLP layer respectively to enhance the model's ability to express complex temporal patterns in long-tail videos. Then perform 3-kernel one-dimensional convolution on their spatial dimension T respectively to extract the correlation features VH R ′, VL R ′, VM R ′ of adjacent positions, and perform residual connections with the high-confidence branch feature VH R , the low-confidence branch feature VL R , and the enhanced visual motion feature VM R respectively. The formula is as follows:

[0136] VH R ″ = VH R +VH R′

[0137] VL R ″ = VL R +VL R ′

[0138] VM R ″ = VM R +VM R ′

[0139] where VH R ″ is the high-confidence branch position enhancement feature, VL R ″ is the low-confidence branch position enhancement feature, VM R ″ is the visual motion position enhancement feature.

[0140] S8.2) Normalize VH R ″ which is the high-confidence branch position enhancement feature, VL R ″ which is the low-confidence branch position enhancement feature, and VM R ″ which is the visual motion position enhancement feature respectively, and take the average value for the spatial dimension T to obtain the high-confidence branch embedding feature VH S 、the low-confidence branch embedding feature VL S 、and the visual motion embedding feature VM S .

[0141] S9) Concatenate and fuse the low-confidence branch temporal feature VL S and the visual motion temporal feature VM S , convert the fused feature VF S and the high-confidence branch temporal feature VH S into the prediction results for each category respectively, and superimpose the prediction results for each category proportionally to obtain the final prediction result for each category of this video.

[0142] As a preferred embodiment of the present invention, the calculation of the final probability includes the following steps:

[0143] S9.1) Pass the high-confidence branch embedding feature VH S through the fully connected layer and the sigmoid activation function respectively to convert it into the prediction results for each category where N represents the number of categories, where

[0144] S9.2) Concatenate the low-confidence branch embedding feature VL S and the visual motion embedding feature VM S in the spatial dimension T, reduce the dimension through a fully connected layer, take the average value for its spatial dimension T, and then pass through the fully connected layer and the sigmoid activation function to convert it into the prediction results for each category

[0145] S9.3) Superimpose the prediction results and the prediction results proportionally to obtain the final prediction result of the video for each category The formula is as follows:

[0146]

[0147] where κ is a hyperparameter representing the proportion of the prediction result of the high-confidence branch in the final prediction result .

[0148] S9.4) Sort the values of the prediction results of each video from high to low, count the cumulative number of correct categories in the sorted order, and calculate the precision Precision at each position. Take the average of the precisions corresponding to the positions of the correct categories to obtain the running average precision rAP of this category , and the formula is as follows: i , and the formula is as follows: i The formula is as follows:

[0149]

[0150] where Precision i is the accuracy rate calculated by the video for each category; Precision T is the Precision of the correctly labeled category by the category label y i ; N T represents the number of correctly labeled categories by the category label y of this video, and calculate the running average precision of all categories in turn

[0151] S10) To improve the accuracy of the detection results, use the loss function, adopt the gradient descent method to update the parameters, optimize the model, iterate N times, and save the model when the average value of the average precision mAP of each category is the highest. Among them, the loss function where L cls represents the classification loss, C is the total number of classification categories

[0152] As a preferred embodiment of the present invention, the updated model parameters include the following parameters: the running average precision rAP of each category; the fully connected layer parameters, frame attention weights, cross-attention weights; temporal modeling parameters; linear transformation weight matrix MLP layer parameters, one-dimensional convolutional layer parameters, normalization layer parameters

Claims

1. A cross-modal based long-tail video detection method, characterized in that It includes the following steps: S1) Use a shooting device to obtain video data and corresponding audio data related to the task category, and perform category annotation on them to obtain the category label y; S2) Perform frame extraction on the obtained video data to extract video frame files, use a VGG-like network model to extract audio features A from the audio data, use a ResNet50 network model to extract visual features V from the video frame files, and extract visual motion features VM; S3) Perform data augmentation on the extracted audio feature A, visual feature V, visual motion feature VM, and class label y to obtain the data-augmented audio feature A ′ , visual feature V ′ , visual motion feature VM ′ , class label y ′ ; S4) Utilize the visual feature V ′ Calculate the frame attention weights, and perform cross-attention mechanism processing on the audio feature A ′ respectively with the visual feature V ′ and the visual motion feature VM ′ to enhance the visual part with the audio and obtain the visual feature V C and the visual motion feature VM C ; S5) The audio feature A ′ , the visual feature V C , and the visual motion feature VM C are respectively subjected to temporal modeling to capture time series information, obtaining the audio feature A L , the visual feature V L , and the visual motion feature VM L ; S6) The audio feature A L is respectively associated with the visual feature V L , the visual motion feature VM L Through correlation analysis, the parts with low correlation between the audio feature and the visual feature are filtered out, and then the audio feature and the visual feature are further fused to obtain the visual feature V R , the visual motion feature VM R ; S7) Use the frame attention weights in S4) for the visual feature V R to calculate the high-confidence branch feature VH R and the low-confidence branch feature VL R , and reassign the frame weight coefficients to the features of each frame in the high-confidence branch and the low-confidence branch; S8) Convert the high-confidence branch feature VH R , the low-confidence branch feature VL R , and the visual motion feature VM R into the task-related high-confidence branch temporal feature VH S , the low-confidence branch temporal feature VL S , and the visual motion temporal feature VM S respectively through temporal feature encoding to enhance the adaptability to the long-tail video recognition task; S9) Concatenate and fuse the low-confidence branch temporal feature VL S with the visual motion temporal feature VM S to obtain the fused feature VF S and the high-confidence branch temporal feature VH S respectively convert them into the prediction results for each category, and superimpose the prediction results for each category proportionally to obtain the final prediction results for each category of this video; S10) To improve the accuracy of the detection results, calculate the gradient information of the loss function with respect to each trainable parameter through the backpropagation algorithm, and update the model parameters using the gradient descent algorithm; iterate N times to optimize the model, and save the model when the average value mAP of the average precision of each category is the highest; where the loss function where L cls represents the classification loss, C is the total number of categories for classification.

2. The cross-modal based long-tail video detection method according to claim 1, wherein The video data, audio data and their annotation methods in S1) include the following steps: S1.1) Design diverse dimensions for time, space, and objects, use a 4K camera, a directional microphone, and combine a timecode generator to set a unified timestamp for all devices, and perform standardized process shooting; S1.2) Manually verify whether each video content is related to the initial shooting label, and use majority voting to decide whether to retain the video; S1.3) Save the audio file name, video file name, and category name to ensure data traceability.

3. The cross-modal based long-tail video detection method according to claim 1, wherein S2) includes the following steps: S2.1) Extract the features of each second of audio segments from the audio data obtained in S1 using the pre-trained VGG-like network model on AudioSet to obtain audio features where T represents the duration of the video data segment in seconds, and d a represents the dimension of the audio encoding feature, which is the number of output feature maps of the VGG-like network model; S2.2) Extract the feature maps of the sampled frames per second from the video frame file using the ResNet50 network model pre-trained on ImageNet to obtain visual features where k represents the spatial dimension size of the feature maps extracted for each frame, and d v represents the dimension of the visual encoding features, which is the number of output feature maps of the ResNet50 network model; S2.3) Extract the motion features from the visual feature V, capture the temporal motion information by calculating the feature differences between adjacent frames, and concatenate the global average features Generate visual motion features where v t represents the feature at the t-th time step, and concat() represents the concatenation operation.

4. The cross-modal based long-tail video detection method according to claim 1, wherein The data augmentation strategy in S3) includes the following steps: S3.1) Randomly extract two video features V i and V j from the video feature file in S2.2), and generate a new hybrid feature according to the following steps; calculate the dynamic sampling ratio of rAP corresponding to the categories of the updated video features V i and V j in S10) as follows: where rAP i and rAP j are the running average precisions of the video features V i and V j belonging to the i-th and j-th categories respectively, with an initial value of 0; β is the dynamic sampling ratio, with a value range of [0,1], reflecting the relative recognition ability of the network for categories i and j, and the initial value is 0.5; S3.2) Determine the sampling frame numbers L of two video features V i and V j as follows: i and L j The formula is as follows: L i = L - L j where L is the preset total number of frames, is the floor symbol, and L i , L j are the number of frames sampled from the video features V i and V j respectively; S3.3) Uniformly sample L i and L j frames from the audio feature A, the visual feature V, and the visual motion feature VM, and splice the sampled features along the spatial dimension T to generate audio, visual, and visual motion hybrid features respectively. The formula is as follows: Among them is the video V i is the frame sequence of the audio, visual, and visual motion features sampled uniformly at L i frames, is the video V j is the frame sequence of the audio, visual, and visual motion features sampled uniformly at L j frames, are the generated audio, visual, and visual motion mixed features respectively; S3.4) Determine the hybrid feature label y of two video features V i and V j based on β, and the formula is as follows: ′ ​ y ′ = (1 - β) × y i + β × y j where y i and y j are the label vectors of categories i and j respectively, and y ′ is the mixed label vector.

5. The cross-modal based long-tail video detection method according to claim 1, wherein The steps for calculating the frame attention weight and the cross-attention mechanism between audio and visual features in step S4) include the following steps: S4.1) Take the visual features Average the features at each position in the K×K spatial dimension, so as to aggregate the features of the spatial blocks of K×K in each time period into a d v -dimensional vector; Based on the d v Spatial dimension, perform global pooling operation to compress the input features into a global feature vector along the channel dimension; Then use two fully connected layers, combined with ReLU and Sigmoid activation functions, to generate frame attention weights; Finally, expand the frame attention weights to d l dimensions, denoted as S4.2) Project the audio feature A ′ and the visual feature V ′ onto the same latent space through linear transformation respectively; Subsequently, calculate the association strength between the audio feature and each visual spatial position in the form of matrix addition; perform Softmax normalization on the similarity scores to generate the cross-attention weight in the spatial dimension, indicating the correlation between different visual regions and the current audio; S4.3) Based on the generated cross-attention weights, perform dynamic spatial weighting on the visual feature V ′ : Dynamically weight the K 2 spatial location features at each time step by the weight coefficient of each spatial region, and output the audio-guided visual feature Visual motion feature 6. The cross-modal based long-tail video detection method according to claim 1, wherein The association analysis in step S6) includes the following steps: S6.1) Map the audio feature A L to two audio branch features A L ′ and A L ″ through two linear transformations, and map the visual feature V L to two visual branch features V L ′ and V L ″ through two linear transformations. Use the audio branch features A L ′ and the visual branch features V L ′ to calculate the similarity matrix β between vision and audio va and the similarity matrix β between audio and vision av . The formula is as follows: Wherein: is the linear transformation weight matrix, d l is the dimension of the audio and visual features, S6.2) Introduce an adjustable threshold τ for hard binary processing, only retain the strongly positive correlated audio-visual pairs with a similarity higher than τ, perform Softmax normalization on the filtered matrix, and convert it into a probability distribution. The formula is as follows: γ va′ = Softmax(γ va ), γ av′ = Softmax(γ av ) Among them, γ va′ is a strong visual-audio correlation matrix, and γ av′ is a strong audio-visual correlation matrix. is an indicator function that outputs 1 when the input is greater than or equal to 0 and outputs 0 otherwise. τ is a hyperparameter. S6.3) Strong correlation matrix γ va′ and γ av′ are respectively weighted and summed with the audio branch feature A L ″ and the visual branch feature V L ″ to obtain the audio aggregated visual feature and the visual aggregated audio feature; a residual connection structure is adopted to add the feature after temporal modeling and the cross-modal aggregated feature to obtain the visually enhanced audio feature A G and the audio enhanced visual feature V G . Finally, the two parts of features are added and further fused in a certain proportion to obtain the enhanced visual feature The formula is as follows: A G = γ av′ V L ″ + A L , V G = γ va′ A L ″ + V L V R = ηV G + (1 - η)A G where η is a hyperparameter representing the enhanced visual feature V R in the audio-enhanced visual feature V G the proportion occupied; S6.4) The audio feature A L is combined with the visual motion feature VM L and calculated according to the steps of S6.1)-S6.3) to obtain the enhanced visual motion feature 7. The cross-modal based long-tail video detection method according to claim 1, characterized in that The steps for dynamically constructing a two-branch representation in step S7) include the following steps: S7.1) Calculate the high-confidence branch feature VH based on the frame attention weights calculated in S4.1) R = V R ⊙ H, where ⊙ represents element-wise multiplication; S7.2) Calculate the low-confidence branch feature where β is a hyperparameter to prevent the denominator from being zero, denoted as element-wise division, forcing it to focus on the frames ignored in 7.1).

8. The cross-modal based long-tail video detection method according to claim 1, wherein, The steps for the temporal feature encoder in step S8) include the following steps: S8.1) For high-confidence branch features Low-confidence branch features Enhanced visual motion features Respectively use the MLP layer for non-linear transformation to enhance the model's ability to express complex temporal patterns in long-tail videos; then perform 3-kernel one-dimensional convolution on their spatial dimension T respectively to extract the associated features VH R ′ 、VL R ′ 、VM R ′ ,And respectively perform residual connections with the high-confidence branch feature VH R 、Low-confidence branch feature VL R 、Enhanced visual motion feature VM R The formula is as follows: VH R ″ = VH R + VH R ′ VL R ″ = VL R + VL R ′ VM R " = VM R + VM R ′ Among them, VH R ″ is the high-confidence branch position enhancement feature, VL R ″ is the low-confidence branch position enhancement feature, VM R ″ is the visual motion position enhancement feature; S8.2) Normalize VH R ″ as the enhanced feature of the high-confidence branch position, VL R ″ as the enhanced feature of the low-confidence branch position, VM R ″ as the enhanced feature of the visual motion position, respectively, and take the average value of the spatial dimension T to obtain the high-confidence branch embedding feature VH S 、 the low-confidence branch embedding feature VL S 、 the visual motion embedding feature VM S 。 9. The cross-modal based long-tail video detection method according to claim 1, wherein The steps for calculating the final result in step S9) include the following steps: S9.1) Embed the high-confidence branch into the feature VH S Convert them into prediction results for each category through a fully connected layer and a sigmoid activation function respectively where N represents the number of categories, and among them S9.2) Embed the low-confidence branch into the feature VL S and the visual-motion embedding feature VM S Concatenate them in the spatial dimension T, reduce the dimension through a fully connected layer, take the average value in the spatial dimension T, and then convert it into the prediction results for each category through a fully connected layer and a sigmoid activation function S9.3) Stack the prediction results and the prediction results proportionally to obtain the final prediction result of the video for each category The formula is as follows: where κ is a hyperparameter representing the prediction result of the high-confidence branch accounting for the final prediction result in proportion; S9.4) According to the prediction results of each video Sort the values from high to low, count the cumulative number of correct categories in the sorted order, and calculate the precision Precision at each position i , take the average of the precisions corresponding to the positions of the correct categories to obtain the running average precision rAP of this category i , the formula is as follows: Among them Precision i is the accuracy calculated for each category of this video; Precision T is the Precision of the correctly labeled category by the category label y i ; N T represents the number of categories correctly labeled by the category label y of this video, and the running average precision of all categories is calculated in turn 10. The cross-modal based long-tail video detection method according to claim 1, wherein, The steps for updating the model parameters using the gradient descent algorithm in step S10) include the following parameters: the running average precision rAP for each category; the fully connected layer parameters, frame attention weight, cross-attention weight; Temporal modeling parameters; Linear transformation weight matrix MLP layer parameters, one-dimensional convolutional layer parameters, normalization layer parameters.

Images