System for classifying video materials and training method of video tag identification model

Abstract

The embodiment of the invention provides a system for classifying video materials and a training method of a video tag identification model. The system comprises the following steps: acquiring a plurality of historical delivery materials to construct a historical delivery material library; vectorizing each historical delivery material in the historical delivery material library to obtain a corresponding first material vector so as to construct a first material vector library; obtaining a plurality of hot delivery materials to construct a hot delivery material library; vectorizing each hot delivery material in the hot delivery material library to obtain a corresponding second material vector so as to construct a second material vector library; performing traversal matching on the first material vector in the first material vector library and the second material vector in the second material vector library to calculate the similarity between the corresponding historical put materials and the hot put materials; and based on the similarity, screening historical putting materials from the historical putting material library to serve as salvaged putting materials.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing, and in particular to a system for classifying video materials, a training method for a video tag recognition model, and a video material delivery system. Background Art

[0002] In today's digital age, the field of mobile multimedia applications is booming. The display and dissemination of multimedia content, as an important way to promote related products and information, is becoming increasingly important. With the intensification of competition in this field, the improvement of content display effect has become a core issue of concern to content creators and communication practitioners. Efficient multimedia content classification plays a vital role in this. Accurate classification can enable the displayed content to reach the target audience more accurately, thereby significantly improving the effect of content display.

[0003] At present, in the field of multimedia content classification, existing methods have many limitations. Most classification methods mainly rely on manual annotation or simple rule matching. Although manual annotation can guarantee the accuracy of classification to a certain extent, this method is inefficient, consumes a lot of manpower and time costs, and is difficult to meet the needs of rapid processing of large-scale multimedia content. Moreover, with the continuous increase in the amount of multimedia content and the increasing complexity of the content, the limitations of manual annotation are becoming more and more obvious. Simple rule matching methods are too mechanical and lack the ability to deeply mine the rich information in multimedia content. Especially when faced with complex image and video content, these traditional methods are difficult to make full use of the various information therein, such as picture details, color matching, dynamic changes, and emotional expressions in audio, resulting in inaccurate and incomplete classification results that cannot meet the needs of practical applications.

[0004] In multimedia content management, the labeling of video materials is an important basis for efficient video indexing and display effect analysis. Accurate labels can help content management practitioners quickly locate and screen the required video materials, and also provide strong support for the evaluation of display effects. However, existing AI-based video analysis methods have certain limitations. These methods mainly focus on relatively single fields such as detecting character actions and scene classification. For videos composed of multiple complex pictures, especially some videos with rich plots, they cannot provide comprehensive and effective analysis results. Such videos often contain rich and diverse picture composition forms, such as fantasy scenes, unique images, complex plot pictures, etc., and also involve various topics, such as story development, theme characteristics, etc. Existing video analysis methods are difficult to accurately describe these complex picture composition forms, nor can they comprehensively summarize the topics involved in the video, so they cannot provide accurate labels for such videos, which seriously affects the indexing and display effect analysis of the video. Summary of the invention

[0005] In view of this, embodiments of the present invention provide a system for classifying video materials, a method for training a video tag recognition model, and a video material delivery system to at least partially solve the above problems.

[0006] According to a first aspect of an embodiment of the present invention, a system for classifying video material is provided, characterized in that it includes: a front-end device and a back-end server, wherein a video segmentation module is deployed on the front-end device, a video classification model is deployed on the back-end server, the video tag recognition model includes a visual encoding module, an audio encoding module, a fusion module, a video classification module, and a sorting module, wherein the video segmentation module is used to segment the video to be classified to generate a plurality of video segments, the visual encoding module is used to encode the visual content in each video segment to obtain a visual aggregation feature, the audio encoding module is used to encode the audio content in each video segment to obtain an audio aggregation feature, the fusion module is used to fuse the visual aggregation feature and the audio aggregation feature to obtain a content fusion feature, the classification module is used to predict the probability of each video segment being classified into different video tags based on the content fusion feature; the sorting module is used to sort the probability of each video segment being classified into different video tags to determine a predicted video tag for each video segment and classify each video segment accordingly.

[0007] Optionally, the video tag recognition model is trained according to the following steps:

[0008] The video segmentation module segments each constructed video sample to generate a plurality of video segment samples, a portion of which are annotated with real video labels as annotated video segment samples;

[0009] The visual encoding module encodes the visual content in each labeled video segment sample to obtain a visual aggregation feature;

[0010] The audio encoding module encodes the audio content in each labeled video segment sample to obtain an audio aggregation feature;

[0011] The fusion module fuses the visual aggregation feature and the audio aggregation feature to obtain a content fusion feature;

[0012] The classification module predicts the probability of each labeled video segment sample being classified into different video labels based on the content fusion feature;

[0013] The sorting module sorts the probability of each labeled video segment sample being classified into different video labels to determine a predicted video label for each labeled video segment sample and calculates a loss value in combination with the corresponding real video label, and the loss value is used to adjust the module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module.

[0014] Optionally, for a portion of the plurality of video segment samples that are not labeled with video labels, the video segment samples are used as unlabeled video segment samples; the video label recognition model further comprises training according to the following steps:

[0015] The visual encoding module encodes the visual content in each video segment sample to obtain a visual aggregation feature, wherein each video segment sample is the labeled video segment sample or the unlabeled video segment sample;

[0016] The audio encoding module encodes the audio content in each unlabeled video segment sample to obtain an audio aggregation feature;

[0017] The fusion module fuses the visual aggregation feature and the audio aggregation feature to obtain a content fusion feature;

[0018] The classification module predicts the probability of each unlabeled video segment sample being classified into different video labels based on the content fusion feature;

[0019] The sorting module sorts the probability of each unlabeled video segment sample being classified into different video labels to determine a pseudo video label for each unlabeled video segment sample, and calculates a loss value for the pseudo video label of each unlabeled video segment sample. The loss value is used to adjust module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module.

[0020] Optionally, when adjusting the module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module, all module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the module parameters of other modules whose module parameters are not fixed, or, some module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the unfixed module parameters.

[0021] Optionally, by calling a set first linear probe, all module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the module parameters of other modules whose module parameters are not fixed, or, by calling a set second linear probe, some module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the unfixed module parameters.

[0022] According to a second aspect of an embodiment of the present invention, a training method for a video tag recognition model is provided, wherein the video tag recognition model includes a visual encoding module, an audio encoding module, a fusion module, a video classification module, and a sorting module, and the training method includes:

[0023] The video segmentation module segments each constructed video sample to generate a plurality of video segment samples, a portion of which are annotated with real video labels as annotated video segment samples;

[0024] The visual encoding module encodes the visual content in each labeled video segment sample to obtain a visual aggregation feature;

[0025] The audio encoding module encodes the audio content in each labeled video segment sample to obtain an audio aggregation feature;

[0026] The fusion module fuses the visual aggregation feature and the audio aggregation feature to obtain a content fusion feature;

[0027] The classification module predicts the probability of each labeled video segment sample being classified into different video labels based on the content fusion feature;

[0028] The sorting module sorts the probability of each labeled video segment sample being classified into different video labels to determine a predicted video label for each labeled video segment sample and calculates a loss value in combination with the corresponding real video label, and the loss value is used to adjust the module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module.

[0029] Optionally, for a portion of the plurality of video segment samples that are not labeled with video labels, the video segment samples are used as unlabeled video segment samples; the video label recognition model further comprises training according to the following steps:

[0030] The visual encoding module encodes the visual content in each video segment sample to obtain a visual aggregation feature, wherein each video segment sample is the labeled video segment sample or the unlabeled video segment sample;

[0031] The audio encoding module encodes the audio content in each unlabeled video segment sample to obtain an audio aggregation feature;

[0032] The fusion module fuses the visual aggregation feature and the audio aggregation feature to obtain a content fusion feature;

[0033] The classification module predicts the probability of each unlabeled video segment sample being classified into different video labels based on the content fusion feature;

[0034] The sorting module sorts the probability of each unlabeled video segment sample being classified into different video labels to determine a pseudo video label for each unlabeled video segment sample, and calculates a loss value for the pseudo video label of each unlabeled video segment sample. The loss value is used to adjust module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module.

[0035] Optionally, when adjusting the module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module, all module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the module parameters of other modules whose module parameters are not fixed, or, some module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the unfixed module parameters.

[0036] Optionally, by calling a set first linear probe, all module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the module parameters of other modules whose module parameters are not fixed, or, by calling a set second linear probe, some module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the unfixed module parameters.

[0037] According to a third aspect of an embodiment of the present invention, a video material delivery system comprises: a back-end server and a front-end device, wherein a library of video materials to be delivered is stored on the back-end server, the library of video materials to be delivered comprises a plurality of video materials to be delivered, each of the video materials to be delivered has a video tag predicted according to any one of the methods of the embodiments of the present application, and the front-end device is used to run an application that can deliver the video materials to be delivered, so as to initiate a material delivery request to the back-end server in a manner matching the application, so that the back-end server selects matching video materials to be delivered from the library of video materials to be delivered, delivers them to the application and presents them on the front-end device.

[0038] In the solution of the embodiment of the present invention, the system for classifying video materials has the following technical advantages:

[0039] 1. Existing multimedia content classification mostly relies on manual labeling or simple rule matching, which is inefficient and cannot fully mine rich information. This solution adopts an automated system process. The video segmentation module of the front-end device divides the video to be classified into several video segments, and the video classification model of the back-end server performs classification based on multi-module collaboration. This avoids the large amount of manpower and time consumption of manual labeling, and can quickly process large-scale multimedia content. At the same time, through the visual coding module, audio coding module, fusion module, etc., the visual and audio information of the video segment is deeply mined and fused, overcoming the mechanical nature of simple rule matching, and making full use of rich information such as picture details, color matching, dynamic changes, and audio emotional expression, significantly improving the accuracy and comprehensiveness of the classification results, and meeting the needs of practical applications for fast and accurate classification of large-scale and complex multimedia content.

[0040] 2. Existing AI video analysis methods focus on single fields such as character action and scene classification, and have difficulty processing videos with complex images and multiple topics. The video classification model of this solution analyzes video segments from multiple dimensions. The visual encoding module and the audio encoding module extract visual and audio features respectively, and the fusion module fuses the two, allowing the model to comprehensively consider the multimodal information of the video. The classification module predicts the probability of classifying the video segment into different labels based on the fused features, and the sorting module determines the final label. This process can comprehensively analyze videos with rich and diverse picture composition forms (such as fantasy scenes, unique images, complex plot pictures) and multiple topics (story development, theme characteristics), effectively making up for the limitations of existing methods and providing more accurate classification results for complex videos.

[0041] 3. In multimedia content management, accurate video material labeling is crucial. This solution can generate accurate predicted video labels for each video segment by performing detailed feature extraction, fusion and classification of video segments. This helps content management practitioners locate and filter required video materials more quickly and accurately, improves the efficiency and accuracy of video indexing, and provides strong support for efficient management of multimedia content.

[0042] 4. Efficient multimedia content classification is the key to improving display effects. The accurate classification results of this solution can make the display content reach the target audience more accurately. By reasonably classifying and labeling video materials, content creators and communication practitioners can select and display multimedia content in a targeted manner according to the needs of different audiences, thereby significantly improving the effect of content display and facilitating more effective promotion of related products and information in the field of mobile multimedia applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings.

[0044] Figure 1 A system for classifying video materials is provided in an embodiment of the present application.

[0045] Figure 2 A method for training a video tag recognition model is provided in an embodiment of the present application.

[0046] Figure 3 A video material delivery system is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be described clearly and in detail below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in the field based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0048] It should be understood that the terms "first", "second", and "third" in the claims, specifications, and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specifications and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their collections.

[0049] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all combinations, including these combinations.

[0050] Figure 1 A system for classifying video materials is provided in an embodiment of the present application. Figure 1 As shown, it includes: a front-end device 101 and a back-end server 102, wherein a video segmentation module 111 is deployed on the front-end device 101, and a video classification model 112 is deployed on the back-end server 102. The video tag recognition model includes a visual encoding module, an audio encoding module, a fusion module, a video classification module, and a sorting module. The video segmentation module is used to segment the video to be classified to generate a plurality of video segments, the visual encoding module is used to encode the visual content in each video segment to obtain a visual aggregation feature, the audio encoding module is used to encode the audio content in each video segment to obtain an audio aggregation feature, the fusion module is used to fuse the visual aggregation feature and the audio aggregation feature to obtain a content fusion feature, the classification module is used to predict the probability of each video segment being classified into different video tags based on the content fusion feature; the sorting module is used to sort the probability of each video segment being classified into different video tags to determine the predicted video tag for each video segment and classify each video segment accordingly.

[0051] Optionally, the video tag recognition model is trained according to the following steps:

[0052] The video segmentation module segments each constructed video sample to generate a plurality of video segment samples, a portion of which are annotated with real video labels as annotated video segment samples;

[0053] The visual encoding module encodes the visual content in each labeled video segment sample to obtain a visual aggregation feature;

[0054] The audio encoding module encodes the audio content in each labeled video segment sample to obtain an audio aggregation feature;

[0055] The fusion module fuses the visual aggregation feature and the audio aggregation feature to obtain a content fusion feature;

[0056] The classification module predicts the probability of each labeled video segment sample being classified into different video labels based on the content fusion feature;

[0057] The sorting module sorts the probability of each labeled video segment sample being classified into different video labels to determine a predicted video label for each labeled video segment sample and calculates a loss value in combination with the corresponding real video label, and the loss value is used to adjust the module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module.

[0058] Specifically, a detailed description of the technical implementation details of the video tag recognition model training steps:

[0059] 1. Segmentation and Labeling

[0060] 1. Video sample segmentation

[0061] The video segmentation module segments each constructed video sample to generate a number of video segment samples.

[0062] Decompose the complete video sample into more processable video segments to facilitate subsequent feature extraction and analysis. For example, in multimedia content scenarios, a long video contains multiple themes or plot segments, and segmentation can more accurately classify each segment.

[0063] Some of the video segment samples are annotated with real video labels, which are called labeled video segment samples. Real video labels are obtained through manual labeling or other reliable labeling methods, representing the accurate category of the video segment. For example, in multimedia content management, they are the theme type, plot type, etc. of the video.

[0064] 2. Feature Extraction

[0065] 1. Visual feature extraction

[0066] The visual encoding module encodes the visual content in each labeled video segment sample to obtain visual aggregation features.

[0067] The visual coding module is based on the convolutional neural network (CNN) architecture. The convolution kernel size of a layer in the visual coding module is k×k, the step size is s, and the size of a frame image of the input video segment sample is I in ×I in , after the convolution operation, the size of the output feature map is I out ×I out , and its calculation formula is: Where padding is the number of pixels to fill the edges.

[0068] After multiple layers of convolution and pooling operations, the obtained features are processed by the fully connected layer to obtain visual aggregate features. The input feature vector dimension of the fully connected layer is n in , the weight matrix is ​​W v (The dimension is n out ×n in ), the bias vector is b v (The dimension is n out ), then the calculation formula of the visual aggregation feature V is: V = W v ×X v +b v , where X v is the feature vector after convolution and pooling operations. in and n out Depends on the network structure design, in multimedia content classification, n in Determined by the pixel information of the video frame and the previous feature processing, n out It is the designed visual aggregation feature dimension used for subsequent classification operations.

[0069] 2. Audio feature extraction

[0070] The audio encoding module encodes the audio content in each labeled video segment sample to obtain audio aggregation features.

[0071] For audio content, short-time Fourier transform (STFT) is used to extract the time-frequency features of the audio. The audio signal is x(t). When performing the STFT operation, the audio signal is divided into frames, each frame length is N, and the frame shift is M. The calculation formula of STFT is: Where k represents the frame index and f represents the frequency index.

[0072] After STFT and other operations, the obtained audio time-frequency features are further processed, for example, by performing feature transformation through a multi-layer perceptron (MLP). The audio feature input vector is X a (The dimension is m in ), the weight matrix of MLP is W a (The dimension is m out ×m in), the bias vector is b a (The dimension is m out ), then the calculation formula of the audio aggregation feature A is: A = W a ×X a +b a Here m in and m out Depending on the design of the audio feature processing network, in multimedia content classification, m in Determined by the audio sampling rate, duration, etc., m out It is the designed audio aggregation feature dimension.

[0073] 3. Feature Fusion

[0074] The fusion module fuses the visual aggregation features and the audio aggregation features to obtain the content fusion features.

[0075] The dimension of the visual aggregate feature vector V is d v , the dimension of the audio aggregate feature vector A is d a , a simple fusion method is to concatenate the two into a new vector. The calculation formula of content fusion feature F is: Its dimension is d v +d a This fusion method combines the visual and audio information of the video at the feature level. In the classification of multimedia content, the multimodal information of the video can be comprehensively considered. For example, the fantasy scene (visual) and the corresponding fantasy background music (audio) in a video can be better recognized and classified by the model through this fusion.

[0076] IV. Classification and Sorting

[0077] The classification module predicts the probability of each labeled video segment sample being classified into different video labels based on content fusion features.

[0078] The classification module is a multi-layer perceptron (MLP), with the input being the content fusion feature F (dimension d v +d a ), after the first fully connected layer (weight matrix is ​​W c1 , the dimension is p×(d v +d a ), the bias vector is b c1 , dimension is p), the activation function is g (such as ReLU function: g(x) = max(0,x)), then the output of the first layer is: H1 = g(W c1 ×F+b c1 ).

[0079] After being processed by multiple layers of fully connected layers (here we take two layers as an example), the output of the last fully connected layer is processed by the Softmax function to obtain the classification probability. The weight matrix of the last fully connected layer is W c2 (dimension is q×p), the bias vector is b c2 (dimension is q), where q is the number of categories of video labels, then the calculation formula of the predicted classification probability vector P is: P = Softmax (W c2 ×H1+b c2 ), where the calculation formula of the Softmax function is: x i is the i-th element of the input vector. In multimedia content classification, q represents the number of different video categories, such as different plot types, theme types, etc.

[0080] 2. Sorting operation

[0081] The sorting module sorts the probability of each labeled video segment sample being classified into different video labels, determines the predicted video label and calculates the loss value in conjunction with the corresponding real video label.

[0082] The sorting operation sorts the video labels according to the predicted probability values. For example, the element P in the predicted probability vector P i It represents the probability that the video segment belongs to the i-th video label. By sorting the elements in P in descending order, the most likely video label category is determined.

[0083] Loss calculation: The commonly used loss function is the cross entropy loss function. The real video label is represented by one-hot encoding as Y (dimension is q), then the calculation formula of the cross entropy loss function L is: This loss value is used to measure the difference between the predicted result and the true label. In multimedia content classification, the model parameters are optimized by minimizing this loss value, so that the model can classify video segments more accurately.

[0084] 5. Parameter Adjustment

[0085] The loss value is used to adjust the module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module.

[0086] The back propagation algorithm is used to calculate the gradient of the loss function to each module parameter. For example, for the weight matrix W in the classification module c2 , according to the chain rule, the loss function L is W c2 Gradient The calculation is done by following the steps below: First calculate Then, according to the derivative of the Softmax function and the operation rules of the fully connected layer, we can gradually calculate

[0087] Use an optimization algorithm (such as stochastic gradient descent (SGD) or its variants such as Adam) to update the parameters. Taking stochastic gradient descent as an example, for the parameter θ (which is the weight or bias in each module), the update formula is: Where α is the learning rate, which controls the step size of parameter update. Through continuous iteration of this process, the parameters of the model are gradually optimized, improving the accuracy of multimedia video content classification.

[0088] To this end, the technical benefits of the video tag recognition model training step are explained as follows:

[0089] 1. Visual feature extraction convolution operation

[0090] This formula is used to calculate the size of the feature map after the convolution operation. in is the size of the input image, k is the convolution kernel size, $padding$ is the number of pixels padded at the edge, and s is the step size.

[0091] In the multimedia content classification scenario, video frames are used as input. This formula ensures that when performing convolution operations, a standardized feature map size can be obtained regardless of the original size of the video frame. This helps subsequent feature processing operations to be performed in a unified manner, for example, when processing videos of different resolutions, it can ensure the consistency of feature extraction. This is crucial for capturing visual features in videos because it enables the model to stably learn local features such as texture and shape in video frames, thereby improving the ability to represent visual features of different video content.

[0092] 2. Visual feature extraction fully connected layer operation

[0093] V=W v ×X v +b v :This formula describes the calculation process of the fully connected layer, where X v is the feature vector after convolution and pooling operations, W v is the weight matrix of the fully connected layer, b v is the bias vector, and V is the obtained visual aggregation feature.

[0094] To this end, in multimedia content classification, the fully connected layer integrates the local visual features extracted by convolution and pooling operations. v and the bias vector b vThe local features are integrated into a more abstract and representative global visual feature V. This global feature can better reflect the overall visual content of the video segment. For example, for multimedia videos containing rich scenes, it can integrate the local visual elements in the scene to form an effective feature representation of the entire scene, which helps to improve the accuracy of video visual content classification.

[0095] 3. Audio feature extraction Short-time Fourier transform (STFT)

[0096] This formula is the calculation formula of short-time Fourier transform, which is used to convert audio signals from time domain to time-frequency domain, where x(t) is the audio signal, X(k,f) is the feature of time-frequency domain, N is the length of each frame, k is the frame index, and f is the frequency index.

[0097] Therefore, in the classification of multimedia content, audio often contains important semantic information. Through STFT, audio signals can be converted into time-frequency features, effectively capturing the frequency and time characteristics of audio. For example, different types of music, speech or sound effects in a video have unique time-frequency features. After extracting these features through STFT, the model can use audio information to assist in the classification of video content. For example, for videos with a specific musical style, audio features can be used to better identify and classify them.

[0098] 4. Audio feature extraction multi-layer perceptron (MLP) operation

[0099] A=W a ×X a +b a :This formula describes the process of MLP processing audio features, where X a is the audio feature input vector after STFT and other operations, W a is the weight matrix of MLP, b a is the bias vector, and A is the obtained audio aggregation feature.

[0100] Therefore, in the multimedia content classification scenario, the audio time-frequency features extracted by STFT are further processed by MLP to obtain audio aggregation features. MLP uses the weight matrix W a and the bias vector b a Nonlinear transformation of audio features can better meet the needs of classification tasks. This helps to convert audio time-frequency features into feature representations that are more suitable for video classification, allowing the model to make full use of audio information to assist in the classification of video content.

[0101] 5. Feature fusion and splicing operation

[0102] This formula indicates that the visual aggregation feature V and the audio aggregation feature A are concatenated to obtain the content fusion feature F.

[0103] Therefore, in multimedia content classification, the visual and audio information of a video are interrelated and complementary. Through this simple splicing operation, the multimodal information of the video can be integrated at the feature level. For example, for a video with a specific scene and corresponding background music, the spliced ​​features can contain both visual scene features and audio music features, allowing the model to consider both types of information for classification at the same time, avoiding the limitation of relying on only a single modal information for classification, thereby improving the accuracy of classification.

[0104] 6. Classification Operation Multilayer Perceptron (MLP)

[0105] First layer: H1=g(W c1 ×F+b c1 )

[0106] The last layer: P = Softmax (W c2 ×H1+b c2 )

[0107] These formulas describe the calculation process of MLP in the classification module. F is the content fusion feature, W c1 、b c1 is the weight matrix and bias vector of the first fully connected layer, g is the activation function (such as ReLU), H1 is the output of the first layer, and W c2 、b c2 is the weight matrix and bias vector of the last fully connected layer, P is the predicted classification probability vector, and the Softmax function is used to convert the output of the last layer into probability.

[0108] For this reason, in multimedia content classification scenarios, the categories of video tags often have complex semantic relationships. Through the calculations in these formulas, the multi-layer structure of MLP can gradually perform nonlinear transformations on the fused features, thereby learning more complex feature representations. The use of the Softmax function enables the model to output the probability that a video segment belongs to different video tags, which not only provides classification results but also gives the confidence level of each classification. For example, when classifying videos by topic, the model can accurately determine the topic category that the video is most likely to belong to based on these probability values, which helps content managers better screen and classify videos.

[0109] 7. Sorting Operation Cross Entropy Loss Function

[0110] This formula is the calculation formula of the cross entropy loss function, where Y is the one-hot encoding of the real video label, P is the predicted classification probability vector, q is the number of categories of the video label, and L is the loss value.

[0111] Therefore, in multi-category classification tasks such as multimedia content classification, the cross entropy loss function can well measure the difference between the predicted probability and the true label. By minimizing this loss value, the model can continuously adjust parameters during the training process to make the predicted results closer to the actual situation. This helps guide the model to learn the correct classification mode. For example, when classifying videos containing different plot types, the model can accurately learn the feature patterns corresponding to different plot types, thereby improving the accuracy of classification.

[0112] 8. Parameter Adjustment Stochastic Gradient Descent (SGD)

[0113] This formula is the calculation formula used by the stochastic gradient descent algorithm to update parameters, where θ is the model parameter (such as the weight or bias in each module), Δ is the learning rate, is the gradient of the loss function with respect to the parameters.

[0114] Therefore, in the complex neural network structure of multimedia content classification model, the gradient calculated by the back propagation algorithm is used to update the parameters according to this formula using the stochastic gradient descent algorithm. This parameter update method can find the appropriate parameter value during the training process, so that the model converges to a better performance state. For example, when processing a large number of different types of multimedia videos, it can prevent the model from falling into the local optimal solution in the complex parameter space, thereby improving the accuracy and generalization ability of the model in multimedia video content classification.

[0115] Optionally, for a portion of the plurality of video segment samples that are not labeled with video labels, the video segment samples are used as unlabeled video segment samples; the video label recognition model further comprises training according to the following steps:

[0116] The visual encoding module encodes the visual content in each unlabeled video segment sample to obtain a visual aggregation feature, wherein each video segment sample is the unlabeled video segment sample;

[0117] The audio encoding module encodes the audio content in each unlabeled video segment sample to obtain an audio aggregation feature;

[0118] The fusion module fuses the visual aggregation feature and the audio aggregation feature to obtain a content fusion feature;

[0119] The classification module predicts the probability of each unlabeled video segment sample being classified into different video labels based on the content fusion feature;

[0120] The sorting module sorts the probability of each unlabeled video segment sample being classified into different video labels to determine a pseudo video label for each unlabeled video segment sample, and calculates a loss value for the pseudo video label of each unlabeled video segment sample. The loss value is used to adjust module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module.

[0121] Specifically, for the case of considering both unlabeled and labeled data, an attention mechanism is introduced in a scenario. The specific model training technology processing details are as follows (a different structure from the above case of considering only labeled data is used):

[0122] 1. Separation and labeling

[0123] Before training the video label recognition model, we need to process the input data. The video segmentation module divides each constructed video sample into several video segment samples. These video segment samples are divided into two categories: one part is labeled with the real video label, called labeled video segment samples; the other part is not labeled, called unlabeled video segment samples.

[0124] 2. Processing of labeled video segment samples

[0125] The visual encoding module is responsible for encoding the visual content in each labeled video segment sample.

[0126] Assume that the input labeled video segment sample consists of a series of video frames, which can be represented as a four-dimensional tensor in:

[0127] T represents the number of frames in a video segment, which reflects the time length of the video segment. Different videos may have different T values, which will affect the subsequent extraction of time dimension features of the video content.

[0128] H is the height of the video frame, and W is the width of the video frame. They together determine the spatial size of the video frame. Video frames of different resolutions have different H and W values.

[0129] C is the number of channels. For common RGB images, C=3, representing the three color channels of red, green, and blue respectively.

[0130] In the visual encoding module, we adopt an improved method based on convolutional neural network (CNN) combined with attention mechanism to extract features.

[0131] First, the convolution operation is performed, and the convolution kernel is in:

[0132] k is the size of the convolution kernel, usually an odd number such as 3×3 or 5×5, which determines the size of the local area focused on in each convolution operation.

[0133] C out is the number of output channels, by adjusting C out It can control the richness of the output features of the convolutional layer. Different C out The value will affect the model's ability to extract visual features.

[0134] After the convolution operation, the output feature map The calculation formula is:

[0135] in:

[0136] p is the padding size, which is used to fill the edges of the input image during the convolution operation to maintain a certain correspondence between the size of the output feature map and the input image to avoid information loss.

[0137] s is the step size, which determines the step size of the convolution kernel moving on the input image. The choice of step size affects the resolution and computational efficiency of the feature map.

[0138] Next, we introduce the attention mechanism to enhance the model’s attention to important visual information. conv , we reshape it into a two-dimensional matrix Where N = T' × H' × W', D = C out .

[0139] Compute the query, key, and value matrices:

[0140] Q=V reshaped ·W Q K=V reshaped ·W K

[0141] V=V reshaped ·W V Where W Q , W K , W V is a learnable weight matrix with dimensions Usually k =d v .

[0142] Calculate the attention score matrix A:

[0143]

[0144] Calculate the self-attention output V att :

[0145] V att =A·V

[0146] V att Reshape back to a 3D tensor

[0147] Finally, V is pooled by global average pooling. attreshaped Converted into a one-dimensional vector, and then passed through the fully connected layer to obtain the visual aggregation feature V. Assume that the weight matrix of the fully connected layer is The bias vector is but:

[0148] V=W v ·GAP(V attreshaped )+b v

[0149] The audio encoding module encodes the audio content in each labeled video segment sample.

[0150] First, the audio signal is preprocessed, including pre-emphasis, framing and windowing. The length of each frame is N audio sampling points, the frame shift is M sampling points. For each frame of audio signal x(n), n=0,1,…,N audio 1. Calculate its spectrum X(k).

[0151] Audio features are extracted by calculating Mel-frequency cepstral coefficients (MFCC). The mth MFCC coefficient c m The calculation formula is:

[0152]

[0153] K is the number of discrete Fourier transform (DFT) points, which determines the resolution of the spectrum analysis.

[0154] M MFCC It is the number of MFCC coefficients, which determines the dimension of the MFCC feature vector and reflects the richness of the audio features.

[0155] Combine the MFCC feature vectors of all audio frames into a sequence Where L is the number of audio frames.

[0156] This sequence is then input into an improved Long Short-Term Memory (LSTM) network that introduces an attention mechanism to better capture the time series characteristics of the audio.

[0157] For the improved LSTM unit, in addition to the conventional input weight matrix W i , forget weight matrix Wf , output weight matrix W o , input gate bias vector b i , forget gate bias vector b f , output gate bias vector b o In addition, the attention weight matrix W is introduced att .

[0158] Calculate the attention weight α t :

[0159] α t =Softmax(W att ·X a,t )

[0160] The calculation formula of LSTM unit is as follows:

[0161] i t =Sigmoid(W i ·X a,t +b i )f t =Sigmoid(W f ·X a,t +b f )

[0162] c t =f t ⊙c t1 +i t ⊙Tanh(W c ·(α t ⊙X a,t )+b c ) t =Sigmoid(W o ·X a,t +b o )

[0163] h t =o t ⊙Tanh(c t )in:

[0164] i t 、f t , o t are the activation values ​​of the input gate, forget gate, and output gate, respectively.

[0165] c t is the cell state, h t is a hidden state.

[0166] The hidden state of the last time step of the LSTM network is taken as the audio aggregation feature A.

[0167] The fusion module fuses the visual aggregate feature V and the audio aggregate feature A. We adopt a fusion method based on tensor decomposition to combine V and A into a tensor Where h is the dimension of the audio aggregate feature A.

[0168] Then, Tucker decomposition is performed on the tensor T to decompose it into core tensors and the factor matrix where r v and r h is the decomposed dimension.

[0169]

[0170] Finally, the core tensor G is flattened into a one-dimensional vector to obtain the content fusion feature F.

[0171] The classification module predicts the probability of each labeled video segment sample being classified into different video labels based on the content fusion feature F. Assume that the classification module is a deep residual network (ResNet).

[0172] The content fusion feature F is input into ResNet and processed by a series of residual blocks. Each residual block contains two convolutional layers and a skip connection.

[0173] For the i-th residual block, the input feature map is F i1 , after the first convolution layer (convolution kernel K i1 , bias b i1 ) to get Z i1 :

[0174] Z i1 =ReLU(K i1 ·F i1 +b i1 ) After the second convolution layer (convolution kernel K i2 , bias b i2 ) to get Z i2 :

[0175] Z i2 =K i2 ·Z i1 +b i2 Then the input is connected to Z through a skip connection i2 Add together to get the output feature map F i :

[0176] F i =F i1 +Z i2

[0177] After passing through multiple residual blocks, the final feature map is converted into a one-dimensional vector through global average pooling, and then passes through a fully connected layer and a Softmax function to obtain the predicted classification probability vector P.

[0178] The sorting module sorts the predicted probability vector P to determine the predicted video label for each labeled video segment sample. The cross entropy loss function is used to calculate the loss value. Assuming that the true video label is (One-hot encoding form), the calculation formula of the loss value L is:

[0179]

[0180] 3. Processing of unlabeled video segment samples

[0181] #Visual feature extraction

[0182] The visual feature extraction process of unlabeled video segment samples is exactly the same as that of labeled video segment samples. The video frame of the input unlabeled video segment sample is represented as After the same convolution, attention mechanism, global average pooling and fully connected layer operations, the visual aggregation feature V is obtained.

[0183] The audio feature extraction process of unlabeled video segment samples is also the same as that of labeled video segment samples. The audio signal is preprocessed, the MFCC features are calculated, and the feature sequence is input into the improved LSTM network to obtain the audio aggregate feature A.

[0184] The same fusion method based on tensor decomposition is used to fuse the visual aggregation feature V and the audio aggregation feature A to obtain the content fusion feature F.

[0185] The classification module predicts the probability of each unlabeled video segment sample being classified into different video labels based on the content fusion feature F. The process is the same as the classification process of the labeled video segment samples, and the predicted probability vector P is obtained.

[0186] The sorting module sorts the probability vector P and takes the category with the highest probability as the pseudo video label. Assume that the pseudo label is (One-hot encoding form), the cross entropy loss function is used to calculate the loss value:

[0187]

[0188] 4. Parameter adjustment

[0189] The back propagation algorithm is used to calculate the loss value L for each module parameter (such as W v 、b v , W i 、b i , K i1、b i1 etc.) gradient.

[0190] Taking the Adagrad optimization algorithm as an example, for the parameter θ, its update formula is:

[0191] in:

[0192] α is the learning rate, which controls the step size of each parameter update. A suitable α value can make the model converge to a better performance state faster during training.

[0193] g t is the gradient at the current moment, calculated by the back-propagation algorithm.

[0194] ∈ is a small positive number to prevent the denominator from being zero.

[0195] To this end, the benefits of each processing link in implementing the video tag recognition model training are described as follows:

[0196] 1. Visual feature extraction

[0197] These formulas are used to calculate the size of the feature map after the convolution operation. T, H, W represent the time dimension, height, and width of the input video frame sequence, respectively, k is the convolution kernel size, p is the padding size, and s is the step size. Through these formulas, the size of the output feature map can be accurately determined based on the input and convolution kernel parameters.

[0198] To this end, in multimedia content classification scenarios, different videos have different durations (T) and resolutions (H, W). This formula ensures that regardless of the input, feature maps can be generated with unified rules, providing standardized input for subsequent feature processing. For example, for videos of different durations and resolutions, the feature maps output by the convolutional layer can be adjusted to have consistent dimensions, which facilitates processing and feature fusion in subsequent network layers and improves the model's adaptability to various videos.

[0199] Q=V reshaped ·W Q , K=V reshaped ·W K , V = V reshaped ·W V , V att =A·V: These formulas are used to calculate the query, key, value matrix, attention score and output in the attention mechanism. Through the learnable weight matrix W Q , W K , W VTransform the feature map to get different representations, then calculate the attention score matrix A, and weight the value matrix according to the score to get the attention output V att .

[0200] Therefore, in video classification, not all visual information is equally important for classification. The attention mechanism enables the model to automatically focus on key visual information, such as the main object and key actions in the video. For example, when identifying animal videos, the model can use the attention mechanism to focus on key features such as the appearance and actions of the animal, ignore irrelevant information such as the background, improve the pertinence and effectiveness of feature extraction, and thus improve classification accuracy.

[0201] V=W v ·GAP(V attreshaped )+b v :Global average pooling (GAP) converts the three-dimensional feature map into a one-dimensional vector, reducing the data dimension while retaining the global information of the feature. The fully connected layer uses the weight matrix W v and the bias vector b v Perform a linear transformation on the GAP output to obtain the final visual aggregation feature V.

[0202] To this end, the GAP operation reduces the number of model parameters and computational complexity while retaining key information and preventing overfitting. The fully connected layer can further transform and combine the features of the GAP output, enabling the model to learn more abstract and representative visual features, providing strong support for subsequent classification tasks.

[0203] 2. Audio feature extraction

[0204] This formula is used to calculate the MFCC coefficients, which are obtained by weighted summation of the audio spectrum X(k). K is the number of DFT points, m is the number of MFCC coefficients, and M is the MFCC The MFCC coefficients constitute the feature vector of the audio frame.

[0205] To this end, MFCC features can effectively capture the spectral envelope information of audio, reflecting the timbre, pitch and other characteristics of audio. In video classification, audio contains rich semantic information, such as dialogue, background music, etc. MFCC feature extraction can convert audio signals into feature vectors suitable for model processing, helping the model learn the association between audio and video content and improve classification accuracy. For example, by analyzing the background music style in a video, it can help determine the theme or emotional tendency of the video.

[0206] α t =Softmax(W att ·X a,t ),i t =Sigmoid(Wi ·X a,t +b i ), f t =Sigmoid(W f ·X a,t +b f ), c t =f t ⊙c t1 +i t ⊙Tanh(W c ·(α t ⊙X a,t )+b c ), o t =Sigmoid(W o ·X a,t +b o ), h t =o t ⊙Tanh(c t ): These formulas describe the computational process of improving the LSTM unit. α t is the attention weight, which is used to enhance the attention to the important audio features at the current moment. t 、f t , o t are the activation values ​​of the input gate, forget gate, and output gate, respectively, controlling the inflow, retention, and output of information. t is the cell state, h t is the hidden state. Through the update of these gates and states, LSTM can effectively process the time series information of audio.

[0207] Therefore, time series information is crucial in audio processing. The improved LSTM introduces an attention mechanism that can better capture the key features of audio at different moments. For example, in a continuous voice conversation, it can focus on important voice content and ignore noise or irrelevant voice fragments. In this way, the model can more accurately learn the temporal dynamic features of audio, improve the ability to understand and utilize audio information, and further improve the accuracy of video classification.

[0208] 3. Feature Fusion

[0209] Combine the visual aggregate feature V and the audio aggregate feature A into a tensor T, and then use Tucker decomposition to get the core tensor G and factor matrix U v , U a This decomposition method can explore the potential relationship between visual and audio features and reduce and reconstruct high-dimensional features.

[0210] Therefore, in video classification, visual and audio information are interrelated, and the tensor decomposition and fusion method can dig deeper into this relationship. Through decomposition and reconstruction, it is possible to reduce the feature dimension while retaining key fusion information, reduce the amount of calculation, and improve the training efficiency and generalization ability of the model. Compared with simple splicing and fusion, tensor decomposition can better capture the intrinsic connection between the two modal features and provide more valuable feature representation for classification.

[0211] 4. Classification and loss calculation

[0212] Z i1 =ReLU(K i1 ·F i1 +b i1 ), Z i2 =K i2 ·Z i1 +b i2 , F i =F i1 +Z i2 :These formulas describe the calculation process of the residual block. Each residual block transforms the input feature map through two convolutional layers to obtain Z i1 and Z i2 , and then connect the input to Z through a skip connection i2 Add together to get the output feature map F i .

[0213] Therefore, in deep neural networks, as the number of network layers increases, gradient vanishing or gradient exploding problems are prone to occur, making the model difficult to train. The design of residual blocks uses jump connections to make it easier for the model to learn identity mapping and alleviate the gradient problem. This allows the model to build a deeper network structure, thereby learning more complex feature representations and improving classification performance. In video classification, complex video content requires a deep network to learn its features, and the application of residual blocks enables the model to effectively process this complex information.

[0214] or The cross entropy loss function is used to measure the difference between the predicted probability P and the true label Y (or pseudo label Y u ). It obtains the loss value by taking the logarithm of the predicted probability of each category label and weighted summing them. The smaller the loss value, the closer the prediction result is to the true label.

[0215] Therefore, in the video classification task, the cross entropy loss function is an effective metric. It can clearly reflect the deviation between the model prediction results and the actual situation, and provide a clear optimization direction for model training. During the training process, by minimizing the cross entropy loss, the model continuously adjusts the parameters so that the predicted probability is closer to the actual label, thereby improving the accuracy of classification.

[0216] 5. Parameter adjustment

[0217] This formula is used to update the model parameters θ. α is the learning rate, which controls the step size of parameter update; g t is the gradient at the current moment, indicating the direction of parameter update; is the cumulative sum of squared gradients, used to adjust the adaptive changes in the learning rate; ∈ is a small constant to prevent the denominator from being zero.

[0218] To this end, the Adagrad optimization algorithm can automatically adjust the learning rate according to the gradient history of the parameters. For frequently updated parameters, the learning rate will gradually decrease to avoid excessive parameter updates; for parameters that are not frequently updated, the learning rate will be relatively large, prompting them to converge faster. In the training of the video tag recognition model, the importance and update frequency of different parameters are different. The Adagrad algorithm can adaptively adjust the learning rate of each parameter, accelerate the model convergence speed, improve training efficiency, and help the model find a better combination of parameters to improve the accuracy and stability of classification.

[0219] Optionally, when adjusting the module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module, all module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the module parameters of other modules whose module parameters are not fixed, or, some module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the unfixed module parameters.

[0220] Specifically, the technical implementation process of the above-mentioned fixed part module parameters is as follows:

[0221] Identify the modules for which all parameters are to be fixed, for example, determine that all parameters of the visual encoding module, audio encoding module, and fusion module are not to be adjusted; at the same time, determine the modules for which parameters need to be adjusted, such as the video classification module and the sorting module. For each module for which parameters are to be fixed, traverse all its parameters and set the `requires_grad` attribute of each parameter to `False`. This means that these parameters will not be updated when backpropagation is used to calculate the gradient. For example, for the visual encoding module, the code is implemented as:

[0222] for paramin visual_encoder.parameters():

[0223] param.requires_grad = False

[0224] The audio encoding module and the fusion module are operated in the same way.

[0225] Collect all parameters of the modules that need to be adjusted. In the code, the parameters of the video classification module and the sorting module can be extracted separately and then merged into a list. For example:

[0226] modules_to_update=list(classification_module.parameters())+list(sorting_module.parameters())

[0227] Create an optimizer object using the above optimized Adagrad, and pass the parameter list of the module to be adjusted as input to the optimizer. In this way, the optimizer will only update these parameters. For example:

[0228] optimizer=optim.Adam(modules_to_update,lr=0.001)

[0229] In the training loop, training is performed according to the normal forward propagation and back propagation process. During forward propagation, the data passes through the module with fixed parameters and the module with parameters to be adjusted in turn to obtain the prediction result; the loss value between the prediction result and the true label is calculated; then back propagation is performed. At this time, due to the parameter `requires_grad=False` of the fixed module, only the parameters of the module to be adjusted will calculate the gradient and be updated through the optimizer.

[0230] In addition, when fixing some parameters of certain modules and adjusting the parameters of other modules, it is necessary to clearly identify the modules whose parameters are to be fixed, such as the visual encoding module. Then further determine the specific parameters to be fixed, such as the weight parameters of the first layer of convolution in the visual encoding module. At the same time, determine the modules whose parameters need to be adjusted, which can be all modules except those that have fixed parameters, or some of them. For modules whose parameters are to be fixed, traverse all their parameters and determine whether they are the parameters to be fixed based on the name or other identifiers of the parameters. If so, set its `requires_grad` attribute to `False`. For example, for the visual encoding module, determine whether it is the weight parameter of the first layer of convolution by name:

[0231] for name,paramin visual_encoder.named_parameters():

[0232] if 'conv1.weight' in name:

[0233] param.requires_grad = False

[0234] Traverse all modules (including modules with fixed parameters and other modules), collect all parameters with `requires_grad=True`, and form these parameters into a list. For example:

[0235] modules_to_update = []

[0236] for module in[visual_encoder,audio_encoder,fusion_module,classification_module,sorting_module]:

[0237] for paramin module.parameters():

[0238] if param.requires_grad:

[0239] modules_to_update.append(param)

[0240] Similar to the above method, create an optimizer object using the appropriate optimization algorithm and pass the list of parameters to be adjusted as input to the optimizer. For example:

[0241] optimizer=optim.Adam(modules_to_update,lr=0.001)

[0242] The same training process of fixing all module parameters to adjust other module parameters is the same as that of the training loop, where forward propagation, loss calculation, backpropagation, and optimizer parameter updates are performed. During the backpropagation process, only those parameters with `requires_grad=True` will calculate gradients and be updated by the optimizer.

[0243] Optionally, by calling a set first linear probe, all module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the module parameters of other modules whose module parameters are not fixed, or, by calling a set second linear probe, some module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the unfixed module parameters.

[0244] Specifically, in the training process of the video tag recognition model, in order to better optimize the model performance, we can choose to fix the parameters of some modules to adjust the parameters of other modules. This method helps to accelerate model convergence or tap the potential of different parts of the model in certain situations. Here, the concepts of first linear probe and second linear probe are introduced to implement the fixed operation of module parameters.

[0245] In the case of calling the first linear probe to fix all parameters of the module, the first linear probe is a pre-set function, which is used to fix all parameters of the specified module so that they do not change during the training process.

[0246] First, we need to identify the set of modules whose parameters are to be fixed. Suppose we want to fix all parameters of the visual encoding module and the audio encoding module, and the modules to be adjusted are the fusion module, the video classification module, and the sorting module.

[0247] The code for calling the first linear probe with fixed parameters is implemented as follows:

[0248]

[0249] In the above code, the `first_linear_probe` function accepts a list of modules as input, iterates over each module in the list, and then sets the `requires_grad` attribute of all parameters in the module to `False`. This means that in the subsequent training process, these parameters will not participate in the gradient calculation and thus remain fixed.

[0250] Code to obtain the parameters of the module to be adjusted:

[0251] modules_to_update = list(fusion_module.parameters()) +list(video_classification_module.parameters()) +list(sorting_module.parameters())

[0252] Here all parameters of the modules to be adjusted are collected into a list `modules_to_update`.

[0253] The Adam optimization algorithm is used to update the parameters of the module to be adjusted.

[0254] optimizer=torch.optim.Adam(modules_to_update,lr=0.001)

[0255] Here, `lr` is the learning rate, which controls the step size of each parameter update. In practical applications, the learning rate needs to be adjusted according to the training situation of the model.

[0256] In the training cycle, forward propagation, loss calculation and back propagation operations are performed. Since the parameters of the visual encoding module and the audio encoding module are fixed, only the parameters of the module to be adjusted will be updated according to the gradient during back propagation.

[0257]

[0258] Specifically, in the case of calling the second linear probe to fix some parameters of the module, the second linear probe is also a pre-set tool or function, which is used to fix some parameters of the specified module, rather than all parameters.

[0259] For example, the weight parameters of the first convolutional layer in the visual encoding module and the bias parameters of the first two fully connected layers in the audio encoding module need to be fixed, while other parameters and all parameters of the fusion module, video classification module and sorting module are kept as parameters to be adjusted.

[0260] The code for calling the second linear probe to fix some parameters is as follows:

[0261]

[0262]

[0263] In the above code, the `second_linear_probe` function accepts a dictionary as input, the key of the dictionary is the module whose parameters are to be fixed, and the value is a list of parameter names to be fixed. The function iterates over all the parameters of the module and determines whether they need to be fixed based on the parameter name.

[0264] The code for obtaining the parameters of the module to be adjusted is as follows:

[0265]

[0266] Here we traverse all modules and collect the parameters whose `requires_grad` attribute is `True` into the `modules_to_update` list.

[0267] Similar to the case where all parameters are fixed, define the optimizer and do a training loop.

[0268]

[0269] The above operations of fixing some modules or fixing some parameters are closely related to gradient calculation and model optimization. From a mathematical point of view, when certain parameters are fixed, the gradients corresponding to these parameters are zero during the back propagation process, which will not affect the parameter update. It can flexibly adjust the model training method according to the performance and needs of the model at different training stages. For example, in the early stage of model training, fixing the parameters of some underlying feature extraction modules (such as visual encoding modules and audio encoding modules) can allow the model to learn high-level semantic features (such as classification modules) faster; or in the model fine-tuning stage, fixing some of the trained parameters to prevent these parameters from being destroyed during the fine-tuning process, so as to better utilize the advantages of the pre-trained model. At the same time, in this way, the number of parameter updates in the training process can be reduced, the computational complexity can be reduced, and the training efficiency can be improved, which is especially suitable for large-scale data sets and complex model structures.

[0270] Figure 2 A training method for a video tag recognition model is provided in an embodiment of the present application. Figure 2 As shown, the video tag recognition model includes a visual encoding module, an audio encoding module, a fusion module, a video classification module, and a sorting module. The training method includes:

[0271] The video segmentation module segments each constructed video sample to generate a plurality of video segment samples, a portion of which are annotated with real video labels as annotated video segment samples;

[0272] The visual encoding module encodes the visual content in each labeled video segment sample to obtain a visual aggregation feature;

[0273] The audio encoding module encodes the audio content in each labeled video segment sample to obtain an audio aggregation feature;

[0274] The fusion module fuses the visual aggregation feature and the audio aggregation feature to obtain a content fusion feature;

[0275] The classification module predicts the probability of each labeled video segment sample being classified into different video labels based on the content fusion feature;

[0276] The sorting module sorts the probability of each labeled video segment sample being classified into different video labels to determine a predicted video label for each labeled video segment sample and calculates a loss value in combination with the corresponding real video label, and the loss value is used to adjust the module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module.

[0277] Optionally, for a portion of the plurality of video segment samples that are not labeled with video labels, the video segment samples are used as unlabeled video segment samples; the video label recognition model further comprises training according to the following steps:

[0278] The visual encoding module encodes the visual content in each video segment sample to obtain a visual aggregation feature, wherein each video segment sample is the labeled video segment sample or the unlabeled video segment sample;

[0279] The audio encoding module encodes the audio content in each unlabeled video segment sample to obtain an audio aggregation feature;

[0280] The fusion module fuses the visual aggregation feature and the audio aggregation feature to obtain a content fusion feature;

[0281] The classification module predicts the probability of each unlabeled video segment sample being classified into different video labels based on the content fusion feature;

[0282] The sorting module sorts the probability of each unlabeled video segment sample being classified into different video labels to determine a pseudo video label for each unlabeled video segment sample, and calculates a loss value for the pseudo video label of each unlabeled video segment sample. The loss value is used to adjust module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module.

[0283] Optionally, when adjusting the module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module, all module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the module parameters of other modules whose module parameters are not fixed, or, some module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the unfixed module parameters.

[0284] Optionally, by calling a set first linear probe, all module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the module parameters of other modules whose module parameters are not fixed, or, by calling a set second linear probe, some module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the unfixed module parameters.

[0285] Figure 3 A video material delivery system is provided in an embodiment of the present application. Figure 3 As shown, it includes: a back-end material server 101 and a front-end material device 102, wherein a video material library to be delivered is stored on the back-end material server, wherein the video material library to be delivered includes multiple video materials to be delivered, each of which has a video tag predicted according to any method of the embodiments of the present application, and the front-end material device is used to run an application that can deliver the video material to be delivered, so as to initiate a material delivery request to the back-end material server in a manner matching the application, so that the back-end material server selects matching video materials to be delivered from the video material library to be delivered, delivers them to the application, and presents them on the front-end material device.

[0286] Above Figure 2 and Figure 3 For an exemplary explanation of the steps in the embodiment, see the above Figure 1 .

[0287] The above implementation methods are only used to illustrate the embodiments of the present invention, but not to limit the embodiments of the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present invention, and the patent protection scope of the embodiments of the present invention should be defined by the claims. The systems, devices, modules or units described in the above embodiments are specifically implemented by computer chips or entities, or by products with certain functions.

[0288] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CDROM, optical storage, etc.) containing computer-usable program codes.

Claims

1. A system for classifying video material, characterized in that: include: A front-end device and a back-end server, wherein a video segmentation module is deployed on the front-end device, and a video classification model is deployed on the back-end server. The video label recognition model includes a visual encoding module, an audio encoding module, a fusion module, a video classification module, and a sorting module. The video segmentation module is used to segment the video to be classified to generate a plurality of video segments. The visual encoding module is used to encode the visual content in each video segment to obtain a visual aggregation feature. The audio encoding module is used to encode the audio content in each video segment to obtain an audio aggregation feature. The fusion module is used to fuse the visual aggregation feature and the audio aggregation feature to obtain a content fusion feature. The classification module is used to predict the probability of each video segment being classified into different video labels based on the content fusion feature. The sorting module is used to sort the probability of each video segment being classified into different video labels to determine the predicted video label for each video segment and classify each video segment accordingly.

2. The system for classifying video materials according to claim 1, characterized in that: The video tag recognition model is trained according to the following steps: The video segmentation module segments each constructed video sample to generate a plurality of video segment samples, a portion of which are annotated with real video labels as annotated video segment samples; The visual encoding module encodes the visual content in each labeled video segment sample to obtain a visual aggregation feature; The audio encoding module encodes the audio content in each labeled video segment sample to obtain an audio aggregation feature; The fusion module fuses the visual aggregation feature and the audio aggregation feature to obtain a content fusion feature; The classification module predicts the probability of each labeled video segment sample being classified into different video labels based on the content fusion feature; The sorting module sorts the probability of each labeled video segment sample being classified into different video labels to determine a predicted video label for each labeled video segment sample and calculates a loss value in combination with the corresponding real video label, and the loss value is used to adjust the module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module.

3. The system for classifying video materials according to claim 2, characterized in that: For a portion of the plurality of video segment samples that are not labeled with video labels, the samples are used as unlabeled video segment samples; the video label recognition model further includes training according to the following steps: The visual encoding module encodes the visual content in each video segment sample to obtain a visual aggregation feature, wherein each video segment sample is the labeled video segment sample or the unlabeled video segment sample; The audio encoding module encodes the audio content in each unlabeled video segment sample to obtain an audio aggregation feature; The fusion module fuses the visual aggregation feature and the audio aggregation feature to obtain a content fusion feature; The classification module predicts the probability of each unlabeled video segment sample being classified into different video labels based on the content fusion feature; The sorting module sorts the probability of each unlabeled video segment sample being classified into different video labels to determine a pseudo video label for each unlabeled video segment sample, and calculates a loss value for the pseudo video label of each unlabeled video segment sample. The loss value is used to adjust module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module.

4. The system for classifying video materials according to claim 3, characterized in that: When adjusting the module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module, all module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the module parameters of other modules whose module parameters are not fixed, or, some module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the unfixed module parameters.

5. The system for classifying video materials according to claim 4, characterized in that: By calling the set first linear probe, all module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the module parameters of other modules whose module parameters are not fixed, or, by calling the set second linear probe, some module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the unfixed module parameters.

6. A method for training a video tag recognition model, characterized in that: The video tag recognition model includes a visual encoding module, an audio encoding module, a fusion module, a video classification module, and a sorting module. The training method includes: The video segmentation module segments each constructed video sample to generate a plurality of video segment samples, a portion of which are annotated with real video labels as annotated video segment samples; The visual encoding module encodes the visual content in each labeled video segment sample to obtain a visual aggregation feature; The audio encoding module encodes the audio content in each labeled video segment sample to obtain an audio aggregation feature; The fusion module fuses the visual aggregation feature and the audio aggregation feature to obtain a content fusion feature; The classification module predicts the probability of each labeled video segment sample being classified into different video labels based on the content fusion feature; The sorting module sorts the probability of each labeled video segment sample being classified into different video labels to determine a predicted video label for each labeled video segment sample and calculates a loss value in combination with the corresponding real video label, and the loss value is used to adjust the module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module.

7. The system for classifying video materials according to claim 6, characterized in that: For a portion of the plurality of video segment samples that are not labeled with video labels, the samples are used as unlabeled video segment samples; the video label recognition model further includes training according to the following steps: The visual encoding module encodes the visual content in each video segment sample to obtain a visual aggregation feature, wherein each video segment sample is the labeled video segment sample or the unlabeled video segment sample; The audio encoding module encodes the audio content in each unlabeled video segment sample to obtain an audio aggregation feature; The fusion module fuses the visual aggregation feature and the audio aggregation feature to obtain a content fusion feature; The classification module predicts the probability of each unlabeled video segment sample being classified into different video labels based on the content fusion feature; The sorting module sorts the probability of each unlabeled video segment sample being classified into different video labels to determine a pseudo video label for each unlabeled video segment sample, and calculates a loss value for the pseudo video label of each unlabeled video segment sample. The loss value is used to adjust module parameters of at least one of the visual encoding module, the audio encoding module, the fusion module, the video classification module, and the sorting module.

8. The system for classifying video materials according to claim 7, characterized in that: When adjusting the module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module, all module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the module parameters of other modules whose module parameters are not fixed, or, some module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the unfixed module parameters.

9. The system for classifying video materials according to claim 8, characterized in that: By calling the set first linear probe, all module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the module parameters of other modules whose module parameters are not fixed, or, by calling the set second linear probe, some module parameters of at least one of the visual coding module, the audio coding module, the fusion module, the video classification module, and the sorting module are fixed to adjust the unfixed module parameters.

10. A video material delivery system, characterized in that: include: A back-end server and the front-end device, wherein the back-end server stores a library of video materials to be delivered, the library of video materials to be delivered includes multiple video materials to be delivered, each video material to be delivered has a video tag predicted according to any method of claim 16, and the front-end device is used to run an application that can deliver the video materials to be delivered, so as to initiate a material delivery request to the back-end server in a manner matching the application, so that the back-end server selects matching video materials to be delivered from the library of video materials to be delivered, delivers them to the application, and presents them on the front-end device.

Images