Video classification model, training method, classification method, device and storage medium

Abstract

Embodiments of the present application provide a video classification model, a training method, a classification method, equipment and a storage medium. The video classification model comprises a visual coding module, a text coding module, a compression module connected with the visual coding module and the text coding module respectively, and a feature fusion module connected with the compression module. The compression module comprises a plurality of network parameters. The network parameters have a learnable characteristic in the training process. The parameter values of the network parameters contained in the compression module in the trained video classification model are learned in the model training process. The technical solution provided by the embodiments of the present application not only reduces the video memory consumption in feature fusion, but also reduces the important information loss caused by compression through the learnable compression mode.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video classification model, training method, classification method, device and storage medium. Background Technology

[0002] In recent years, video has become a major means of information dissemination. Current video search, video recommendation, and platform operations typically utilize video tagging technology to assign corresponding tags to videos, then recommend videos based on these tags, build indexes based on tags to enable video searches, or conduct user interest analysis and operations based on tags.

[0003] Video tagging methods are essentially multi-label classification of videos, belonging to the video-to-text classification method. Existing video-to-text classification methods face the problem of high memory consumption during runtime, which ordinary computers cannot handle. Summary of the Invention

[0004] In view of the above problems, this application is made to provide a video classification model, training method, classification method, device and storage medium for solving the above problems.

[0005] Therefore, in one embodiment of this application, a video classification model is provided. The model includes: a visual encoding module, a text encoding module, a compression module connected to both the visual encoding module and the text encoding module, and a feature fusion module connected to the compression module.

[0006] The compression module contains multiple network parameters; these network parameters are learnable during training, and the parameter values ​​of the network parameters in the compression module of the trained video classification model are learned during model training.

[0007] In another embodiment of this application, a method for training a video classification model is provided. The method includes:

[0008] Obtain training samples, which include: the video frame sequence, the text word sequence corresponding to the sample video, and the true category to which the sample video belongs;

[0009] With the objective of minimizing the loss function, a video classification model is trained based on the training samples. The loss function is determined based on the difference between the classification result output by the video classification model and the true category of the sample video. The video classification model includes: a visual encoding module, a text encoding module, a compression module connected to both the visual encoding module and the text encoding module, and a feature fusion module connected to the compression module. The compression module contains multiple network parameters, which are learnable during training.

[0010] In yet another embodiment of this application, a video classification method is provided. The method includes:

[0011] The process involves obtaining the video frame sequence and text word sequence corresponding to the video to be classified, as well as a trained video classification model. The video classification model includes: a visual encoding module, a text encoding module, a compression module connected to both the visual encoding module and the text encoding module, and a feature fusion module connected to the compression module. The compression module contains multiple network parameters. These network parameters are learnable during training, and the parameter values ​​of the network parameters in the compression module within the trained video classification model are learned during model training.

[0012] Using the visual encoding module and text encoding module in the trained video classification model, feature encoding is performed on the video frame sequence and the text word sequence respectively to obtain visual feature sequence and text feature sequence;

[0013] The visual feature sequence and the text feature sequence are compressed using the compression module in the trained video classification model to obtain compressed visual feature sequence and compressed text feature sequence, respectively.

[0014] The compressed visual feature sequence and the compressed text feature sequence are fused using the feature fusion module in the trained video classification model to classify the video to be classified.

[0015] In another embodiment of this application, an electronic device is provided. The electronic device includes: a memory and a processor, wherein,

[0016] The memory is used to store programs;

[0017] The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method described in any of the preceding embodiments.

[0018] In another embodiment of this application, a computer-readable storage medium storing a computer program is provided, which, when executed by a computer, can implement the methods described in any of the above-described embodiments.

[0019] In the technical solution provided in this application embodiment, the compression module includes multiple network parameters. These network parameters possess learnable characteristics during training. Thus, the parameter values ​​of the network parameters included in the compression module in the trained video classification model are learned during model training. In other words, by performing length compression processing on the visual feature sequences corresponding to video frame sequences and the text feature sequences corresponding to text word sequences in a learnable manner, the amount of visual feature data and text feature data is compressed. This not only reduces the GPU memory consumption of visual and text features during feature fusion but also minimizes the loss of important information caused by compression through a learnable approach. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the internal structure of a video classification model provided in an embodiment of this application;

[0022] Figure 2 A flowchart illustrating a video classification method provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a feature compression process provided in an embodiment of this application;

[0024] Figure 4 A flowchart illustrating a model training method provided in an embodiment of this application;

[0025] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] Existing video-text classification methods face two challenges:

[0027] (1) As video content becomes richer, the number of video frames required and the length of text become longer. When processing long videos and long texts, there is a huge consumption of video memory, which ordinary machines cannot handle.

[0028] (2) A large number of video samples are often required for pre-training to achieve good results, which consumes a lot of time and computing resources.

[0029] In today's digitally driven world, the old model of large-scale pre-training based on video samples and then fine-tuning for downstream tasks appears cumbersome and inefficient.

[0030] Addressing these two challenges is a common hurdle for both academia and industry. Rapidly developing video labeling tasks is a key pain point in the video labeling field.

[0031] To truly address the aforementioned pain points, this application proposes a novel video classification method: it performs length compression on the visual feature sequences corresponding to video frame sequences and the text feature sequences corresponding to text word sequences in a learnable manner. That is, by compressing the amount of visual and text feature data in a learnable way, not only can the GPU memory consumption during feature fusion be reduced, but the degree of loss of important information due to compression can also be minimized through a learnable approach. Furthermore, feature compression allows the model to receive more sparse frames and longer text inputs, further improving performance.

[0032] Furthermore, the model can be pre-trained using a contrastive language-image pre-training method before being transferred to the video-text domain, thus avoiding expensive pre-training on large-scale video datasets. After pre-training, some neural network layers can be frozen during fine-tuning to reduce computational burden and GPU memory consumption.

[0033] The improvements mentioned above enable the rapid development of video labeling tasks.

[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0035] Furthermore, some processes described in the specification, claims, and accompanying drawings of this application include multiple operations that appear in a specific order. These operations may be performed out of order or in parallel. Operation numbers such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be performed sequentially or in parallel. It should be noted that the terms "first," "second," etc., used herein are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0036] Figure 1 A schematic diagram of the internal structure of a video classification model provided in an embodiment of this application is shown. Figure 1 As shown, there is a visual encoding module 11, a text encoding module 12, a compression module 13 connected to the visual encoding module 11 and the text encoding module 12 respectively, and a feature fusion module 14 connected to the compression module 13;

[0037] The compression module 13 contains multiple network parameters; these network parameters are learnable during training, and the parameter values ​​of the network parameters contained in the compression module 13 in the trained video classification model are learned during model training.

[0038] In the technical solution provided in this application embodiment, the compression module includes multiple network parameters. These network parameters possess learnable characteristics during training. Thus, the parameter values ​​of the network parameters included in the compression module in the trained video classification model are learned during model training. In other words, by performing length compression processing on the visual feature sequences corresponding to video frame sequences and the text feature sequences corresponding to text word sequences in a learnable manner, the amount of visual feature data and text feature data is compressed. This not only reduces the GPU memory consumption of visual and text features during feature fusion but also minimizes the loss of important information caused by compression through a learnable approach.

[0039] In one example, the compression module 13 described above has a self-attention mechanism. The specific implementation process of the attention mechanism will be described in detail in the following embodiments.

[0040] The training method for the above visual classification model will also be described in detail in the following embodiments.

[0041] Figure 2This illustration shows a flowchart of a video classification method provided in an embodiment of this application. The execution entity of this method can be a client or a server. The client can be hardware with embedded programs integrated on a terminal, application software installed on the terminal, or utility software embedded in the terminal's operating system, etc., and this embodiment does not limit its scope. The terminal can be any terminal device, including mobile phones and computers. The server can be a conventional server, a cloud server, or a virtual server, etc., and this embodiment does not specifically limit its scope. Figure 2 As shown, the method includes:

[0042] 201. Obtain the video frame sequence and text word sequence corresponding to the video to be classified, as well as the trained video classification model.

[0043] The video classification model includes: a visual encoding module, a text encoding module, a compression module connected to the visual encoding module and the text encoding module respectively, and a feature fusion module connected to the compression module; wherein the compression module contains multiple network parameters; the network parameters have learnable characteristics during training, and the parameter values ​​of the network parameters contained in the compression module in the trained video classification model are learned during model training.

[0044] 202. Using the visual encoding module and text encoding module in the trained video classification model, feature encoding is performed on the video frame sequence and the text word sequence respectively to obtain visual feature sequence and text feature sequence.

[0045] 203. The compression module in the trained video classification model is used to perform length compression processing on the visual feature sequence and the text feature sequence respectively to obtain the compressed visual feature sequence and the compressed text feature sequence.

[0046] The network parameters involved in the compression module are learned during the model training process.

[0047] 204. Using the feature fusion module in the trained video classification model, perform feature fusion on the compressed visual feature sequence and the compressed text feature sequence to classify the video to be classified.

[0048] In step 201 above, a video frame sequence is determined based on the video to be classified. In one example, the video to be classified consists of multiple video frames arranged in sequence; these multiple video frames arranged in sequence can be determined as a video frame sequence.

[0049] In practical applications, due to the high frame rate and large number of video frames, in order to reduce the processing pressure later, in another example, the video to be classified can be subjected to frame extraction to obtain a video frame sequence. The specific frame extraction strategy can be set according to actual needs, and this application embodiment does not impose specific limitations on it.

[0050] It should be added that after obtaining the above video frame sequence, each video frame in the video frame sequence can be cropped and / or scaled to adapt to the input size requirements of the model.

[0051] The text word sequence corresponding to the video to be classified consists of multiple text words arranged in order. The text word sequence corresponding to the video to be classified can be determined using one of the following methods:

[0052] Method 1: Generate the text word sequence based on the descriptive text information of the video to be classified.

[0053] In practical applications, some videos contain descriptive text information added by users. Therefore, the descriptive text information of the video to be classified can be obtained; based on the descriptive text information of the video to be classified, a text word sequence can be generated.

[0054] Method 2: Perform optical character recognition on the video to be classified to obtain the text word sequence.

[0055] Note: The process of generating a text word sequence usually involves word segmentation. The word segmentation strategy can be set according to actual needs, and this application embodiment does not impose specific limitations on it. For example, when the text is Chinese, each Chinese character can be treated as a text word; when the text is English, each English word can be treated as a text word.

[0056] Specifically, optical character recognition (OCR) can be performed on video frames in the video to be classified to obtain a text word sequence.

[0057] When a video frame sequence is obtained by extracting frames, OCR recognition can be performed on each video frame in the extracted video frame sequence to obtain a text word sequence.

[0058] Method 3: Perform audio recognition on the audio in the video to be classified to obtain the text word sequence.

[0059] Audio can be extracted from the video to be classified; audio recognition is performed on the audio to obtain a text word sequence.

[0060] When a video frame sequence is obtained by frame extraction, audio recognition can be performed on the audio frames corresponding to each video frame in the extracted video frame sequence to obtain a text word sequence.

[0061] In section 202 above, the video classification model can be any machine learning model, and this application embodiment does not specifically limit it. The video classification model is trained based on the training samples with the objective of minimizing the loss function. The training samples include: the video frame sequence corresponding to the sample video, the text word sequence, and the true category to which the sample video belongs. The loss function is determined based on the difference between the classification result output by the video classification model and the true category to which the sample video belongs.

[0062] In one example, the aforementioned visual classification model can be a deep learning model based on neural networks. Neural networks have the following advantages: strong robustness and fault tolerance because information is distributed and stored in neurons within the network; parallel processing methods, making computation fast; self-learning, self-organizing, and adaptive capabilities, enabling the network to handle uncertain or unknown systems; the ability to fully approximate arbitrarily complex nonlinear relationships; strong information synthesis capabilities, able to process both quantitative and qualitative information simultaneously, and able to coordinate the relationships between multiple input information well, making them suitable for multi-information fusion and multimedia technologies.

[0063] like Figure 1 As shown, the video classification model above includes: a visual encoding module, a text encoding module, a compression module, a feature fusion module, and a classification module.

[0064] The system includes a visual encoding module for acquiring visual feature sequences, a text encoding module for acquiring text feature sequences, a compression module for compressing the visual feature sequences output by the visual encoding module and the text visual feature sequences output by the text encoding module, a feature fusion module for fusing the compressed visual feature sequences and the compressed text visual feature sequences output by the compression module, and a classification module for classifying based on the fused features output by the feature fusion module.

[0065] The length of a visual feature sequence is determined by the number of its elements; the more elements, the longer the sequence. Similarly, the length of a text feature sequence is also determined by the number of its elements; more elements result in a longer sequence. The number of elements in a visual feature sequence is positively correlated with the number of video frames in the video frame sequence; that is, the more video frames, the more elements in the visual feature sequence. Likewise, the number of elements in a text feature sequence is positively correlated with the number of text words in the text word sequence; that is, the more text words, the more elements in the text feature sequence. Visual feature sequences are used to describe video frame sequences; text feature sequences are used to describe text word sequences.

[0066] In step 203 above, the compression module in the video classification model is used to perform length compression processing on the visual feature sequence and the text feature sequence respectively, to obtain the compressed visual feature sequence and the compressed text feature sequence.

[0067] In one example, the compression module may include: at least one fully connected layer; and the at least one fully connected layer is used to perform length compression processing on the visual feature sequence and the text feature sequence respectively, to obtain compressed visual feature sequences and compressed text feature sequences. The network parameters involved in the at least one fully connected layer are learned during model training.

[0068] To facilitate subsequent fusion, the number of elements in the compressed visual feature sequence and the compressed text feature sequence can be equal.

[0069] In step 204 above, the feature fusion module in the video classification model is used to perform feature fusion on the compressed visual feature sequence and the compressed text feature sequence in order to classify the video to be classified.

[0070] Feature fusion can be achieved through vector concatenation or vector averaging.

[0071] In one example, the compressed visual feature sequence and the compressed text feature sequence can be concatenated to obtain the concatenated feature sequence; based on the concatenated feature sequence, the fused feature sequence can be determined.

[0072] For example, if the number of elements in the compressed visual feature sequence is M, then the number of elements in the concatenated feature sequence is 2M.

[0073] Optionally, the concatenated feature vector can be directly used as the fused feature vector.

[0074] In another instance, a weighted average operation can be performed on the compressed visual feature sequence and the compressed text feature sequence to obtain the fused feature sequence.

[0075] The weights corresponding to the compressed visual feature sequence and the compressed text feature sequence can be set according to actual needs, and this application does not impose specific limitations on this. In a specific example, the weights of both can be set to 0.5, that is, both are equally important to the fused feature sequence.

[0076] The video classification model utilizes its classification module to classify the video based on the fused features. Specifically, the classification in this embodiment can be single-label classification or multi-label classification.

[0077] In the technical solution provided in this application embodiment, the visual feature sequence corresponding to the video frame sequence and the text feature sequence corresponding to the text word sequence are compressed in a learnable manner. That is, the amount of visual feature data and text feature data are compressed in a learnable manner. This not only reduces the memory consumption of visual features and text features during feature fusion, but also reduces the degree of loss of important information caused by compression in a learnable manner.

[0078] Optionally, the step 202 above, "using the visual coding module in the trained video classification model to perform feature encoding on the video frame sequence to obtain a visual feature sequence," can be implemented using the following steps:

[0079] 2021. Using the visual encoding module in the trained video classification model, feature encoding is performed on each video frame in the video frame sequence to obtain the frame feature sequence corresponding to each video frame.

[0080] 2022. By splicing the frame feature sequences corresponding to multiple video frames in the video frame sequence, a visual feature sequence is obtained.

[0081] In the aforementioned 2021, the frame feature sequence includes multiple elements, each of which is a feature, which can be a multidimensional feature. When the feature is multidimensional, the data form of the frame feature sequence can be represented as a matrix [n1, d1], where n1 represents the number of features or elements in the frame feature sequence; and d1 represents the dimension of the feature.

[0082] In the above 2022, the number of elements in the visual feature sequence is the sum of the number of elements in the frame feature sequences corresponding to each of the multiple video frames in the video frame sequence.

[0083] Specifically, a visual feature sequence can be obtained by splicing the frame feature sequences corresponding to each of the multiple video frames in the video frame sequence according to the sorting information of the multiple video frames in the video frame sequence.

[0084] Optionally, the step 202 above, "using the text encoding module of the trained video classification model to perform feature encoding on the text word sequence to obtain a text feature sequence," can be implemented using the following steps:

[0085] 2023. Using the text encoding module of the trained video classification model, feature encoding is performed on each text word in the text word sequence to obtain the text word features of each text word.

[0086] 2024. Concatenate the text word characteristics of multiple text words in the text word sequence to obtain a text characteristic sequence.

[0087] Each element in the text feature sequence represents a text word feature. Each text word feature can be a multi-dimensional feature.

[0088] In this context, the number of elements in the text feature sequence is equal to the number of text words in the text word sequence.

[0089] Specifically, the text feature sequence can be obtained by concatenating the text word features of multiple text words in the text word sequence according to the sorting information of multiple text words in the text word sequence.

[0090] To further reduce the loss of important information during dimensionality reduction, a learnable self-attention mechanism can be used. The compression module can be designed to possess a self-attention mechanism. Specifically, step 103 above, "using the compression module in the video classification model to perform length compression processing on the visual feature sequence and the text feature sequence respectively, to obtain compressed visual feature sequences and compressed text feature sequences," may include:

[0091] 2031. Using the self-attention mechanism of the compression module, the visual feature sequence and the text feature sequence are compressed to obtain the compressed visual feature sequence and the compressed text feature sequence.

[0092] The basic principle of self-attention mechanisms is to use a set of learnable network parameters to measure the importance of input information and map it to output information so that it can automatically learn and focus on the most important information.

[0093] In one feasible solution, the phrase "utilizing the self-attention mechanism of the compression module to perform length compression processing on the visual feature sequence and the text feature sequence respectively, to obtain compressed visual feature sequences and compressed text feature sequences" in section 2031 above includes:

[0094] S11. Determine the query sequence based on the network parameters involved in the compression module.

[0095] The length of the query sequence is less than the length of the visual feature sequence.

[0096] S12. For the element at the first position in the compressed visual feature sequence, determine the importance of each element in the visual feature sequence relative to the element at the first position in the compressed visual feature sequence based on the similarity between the element at the first position in the query sequence and each element in the visual feature sequence.

[0097] Wherein, the first sequence is any position in the compressed visual feature sequence.

[0098] S13. Determine the compressed visual feature sequence based on the visual feature sequence and the importance of each element in the visual feature sequence relative to the element in the first position of the compressed visual feature sequence.

[0099] It should be noted that the number of elements in the final compressed visual feature sequence is equal to the number of elements in the query sequence. Therefore, in order to achieve compression, the number of elements in the query sequence must be less than the number of elements in the visual feature sequence.

[0100] In S11 above, the query sequence can be composed of multiple network parameters included in the compression module. Each element in the query sequence can also be a multi-dimensional element. When the element is a multi-dimensional feature, the data form of the query sequence can be represented as a matrix [n2, d2], where n2 represents the number of elements in the query sequence; d2 represents the dimension of the element. The positions of the multiple network parameters involved in the compression module in the query sequence can be configured before model training.

[0101] The above steps S12 and S13 mainly involve: performing self-attention query on the visual feature sequence according to the query sequence to obtain the compressed visual feature sequence.

[0102] In S12 above, in practical applications, each element in the query sequence and the visual feature sequence is a multi-dimensional element, and the dimensions of each element in the query sequence and the visual feature sequence are the same. Therefore, each element can actually be regarded as a one-dimensional vector; the similarity between any two elements can be determined by the dot product of the one-dimensional vectors of the two elements.

[0103] In the self-attention mechanism, the query matrix corresponding to the query sequence can be determined; the first key matrix and the first value matrix can be determined based on the visual feature sequence; the first attention score matrix can be determined based on the query matrix and the first key matrix; the first attention score matrix includes: the importance (or contribution) of each element in the visual feature sequence relative to the element in the first position in the compressed visual feature sequence; the first position is any position in the compressed visual feature sequence.

[0104] The first attention score matrix is ​​normalized to obtain the normalized first attention score matrix; the normalized first attention score matrix is ​​then multiplied by the first value matrix to obtain the first target matrix corresponding to the compressed visual feature sequence.

[0105] For example:

[0106] The mathematical representation of the query sequence is [n, d], where n is the number of elements in the query sequence and d is the dimension of each element; [n, d] is used as the query matrix in the self-attention mechanism.

[0107] The mathematical representation of the visual feature sequence is [N, d], where N is the number of elements in the visual feature sequence and d is the dimension of each element. The [N, d] corresponding to the visual feature sequence is used as the first key matrix and the first value matrix in the self-attention mechanism. n is less than N.

[0108] Multiply the query matrix [n, d] with the transpose of the first key matrix [N, d] to obtain the first attention score matrix [n, N]. The first attention score matrix [n, N] records the importance (or contribution) of each element in the visual feature sequence relative to the element in the first position in the compressed visual feature sequence; the first position is any position in the compressed visual feature sequence.

[0109] The first attention score matrix [n, N] is normalized to obtain the normalized first attention score matrix [n, N].

[0110] The normalized first attention score matrix [n, N] is multiplied by the first value matrix [N, d] to obtain the first target matrix [n, d] corresponding to the compressed visual feature sequence. In the first matrix [n, d], n represents the number of elements in the compressed visual feature sequence, and d represents the dimension of each element in the compressed visual feature sequence.

[0111] As can be seen from the examples above, the number of elements n in the compressed visual feature sequence is less than the number of elements N in the uncompressed visual feature sequence.

[0112] Optionally, the length of the query sequence is also less than the length of the text feature sequence. The step 1031 above, "using the self-attention mechanism of the compression module to perform length compression processing on the visual feature sequence and the text feature sequence respectively, to obtain compressed visual feature sequences and compressed text feature sequences," may further include the following steps:

[0113] S14. For the element at the second position in the compressed text feature sequence, determine the importance of each element in the text feature sequence relative to the element at the second position in the compressed text feature sequence based on the similarity between the element at the second position in the query sequence and each element in the text feature sequence.

[0114] Wherein, the second position is any position in the compressed text feature sequence. S15. Determine the compressed text feature sequence based on the importance of each element in the text feature sequence and the text characteristic sequence relative to the element at the second position in the compressed text feature sequence.

[0115] The above steps S14 and S15 mainly involve: performing self-attention query on the text feature sequence according to the query sequence to obtain the compressed text feature sequence.

[0116] In practical applications, each element in the query sequence and the text feature sequence is a multi-dimensional element, and the dimensions of each element in the query sequence and the text feature sequence are the same. Therefore, each element can actually be regarded as a one-dimensional vector; the similarity between any two elements can be determined by the dot product of the one-dimensional vectors of the two elements.

[0117] In the self-attention mechanism, the query matrix corresponding to the query sequence can be determined; the second key matrix and the second value matrix can be determined based on the text feature sequence; the second attention score matrix can be determined based on the query matrix and the second key matrix; the second attention score matrix includes: the importance (or contribution) of each element in the text feature sequence relative to the element in the second position in the compressed text feature sequence; the second position is any position in the compressed text feature sequence.

[0118] The second attention score matrix is ​​normalized to obtain the normalized second attention score matrix; the normalized second attention score matrix is ​​then multiplied with the second value matrix to obtain the second target matrix corresponding to the compressed visual feature sequence.

[0119] For example:

[0120] The mathematical representation of the query sequence is [n, d], where n is the number of elements in the query sequence and d is the dimension of each element; [n, d] is used as the query matrix in the self-attention mechanism.

[0121] The mathematical representation of the text feature sequence is: [M, d], where M is the number of elements in the text feature sequence and d is the dimension of each element; [M, d] corresponding to the text feature sequence is used as the second key matrix and the second value matrix in the self-attention mechanism. n is less than M.

[0122] Multiply the query matrix [n, d] with the transpose of the second key matrix [M, d] to obtain the second attention score matrix [n, N]. The second attention score matrix [n, M] records the importance (or contribution) of each element in the text feature sequence relative to the element in the second position of the compressed text feature sequence; the second position is any position in the compressed text feature sequence.

[0123] The second attention score matrix [n, M] is normalized to obtain the normalized second attention score matrix [n, M].

[0124] The normalized second attention score matrix [n, M] is multiplied by the second value matrix [M, d] to obtain the second target matrix [n, d] corresponding to the compressed text feature sequence. In the second target matrix [n, d], n represents the number of elements in the compressed text feature sequence, and d represents the dimension of each element in the compressed text feature sequence.

[0125] As can be seen from the above examples, the number of elements n in the compressed text feature sequence is less than the number of elements M in the uncompressed text feature sequence.

[0126] like Figure 3 As shown, a short feature sequence (i.e., the query sequence mentioned above) is determined based on the learnable network parameters. The visual feature sequence or text feature sequence is then used as the long feature sequence. Self-attention query is performed on the long feature sequence based on the short feature sequence to obtain the compressed feature sequence. The length of the compressed feature sequence is the same as the length of the short feature sequence.

[0127] To improve the model's adaptability to different inputs, a compression module is used to compress video and text feature sequences. The compressed feature sequences are significantly shorter, resulting in a substantial reduction in the model's computational cost for feature fusion. Compared to other compression methods, such as max pooling and average pooling, this compression module-based approach utilizes a learnable neural network for compression. It can reduce the length of sequence features while preserving features to the greatest extent possible, thereby lowering the model's computational cost.

[0128] In practical applications, since the compression module compresses the feature sequences generated by long videos and long texts through a learnable self-attention mechanism, the model can receive more sparse frames and longer text inputs, further improving performance.

[0129] The following section introduces a model training method for the aforementioned video classification model. This method includes:

[0130] 205. Obtain training samples.

[0131] The training samples include: the sample video frame sequence corresponding to the sample video, the sample text word sequence, and the true category to which the sample video belongs.

[0132] 206. With the goal of minimizing the loss function, train the video classification model based on the training samples.

[0133] The loss function is determined based on the difference between the classification result output by the video classification model and the true category to which the sample video belongs.

[0134] In step 205 above, the process of determining the sample video frame sequence and sample text word sequence corresponding to the sample video can refer to the process of determining the video frame sequence and text word sequence in the above embodiments, and will not be described in detail here.

[0135] The sample video can belong to multiple real categories, or real labels.

[0136] In step 206 above, the sample video frame sequence and sample text word sequence are input into the video classification model. The internal processing flow of the video classification model can be referred to the corresponding content in the above embodiments, and will not be detailed here. The video classification model will finally output the classification result for the sample video.

[0137] The loss function is determined based on the difference between the classification result output by the video classification model and the true category to which the sample video belongs; the network parameters in the video classification model are optimized with the goal of minimizing the loss function.

[0138] Optionally, before training the video classification model based on the training samples, the above method further includes:

[0139] 207. Initialize the network parameters contained in the visual encoding module using the contrastive language-image pre-training method.

[0140] Contrastive Language-Image Pre-Training (CLIP) is a pre-training method based on text and images. First, text and images are natural paired samples that are ubiquitous in life, such as images and text in news reports, which can reduce the cost of acquiring training data. Second, pre-training is based on text and images, rather than text and video, which can reduce the consumption of computational resources during the training process and shorten the pre-training time, which is conducive to the rapid development of video labeling tasks for downstream tasks.

[0141] Compared to image-based pre-training methods, CLIP-based pre-training is more suitable for multimodal feature fusion. The visual encoding module, pre-trained using CLIP, learns prior knowledge of the text, enabling the extraction of more effective visual features for fusion with textual features.

[0142] Optionally, the text encoding module can also be pre-trained. Specifically, a Bidirectional Encoder Representation from Transformers (BERT) model can be pre-trained, and then the text encoding module can be initialized based on the pre-trained BERT model. BERT model pre-training not only yields good pre-training results but also has low pre-training costs.

[0143] After pre-training, the video classification model can be fine-tuned based on the training samples mentioned above.

[0144] To reduce the pressure of fine-tuning the model for downstream tasks, a partial neural network layer freeze can be applied to the visual encoding module and / or text encoding module: freeze the bottom layer of the encoding module and fine-tune the top layer. Under this freezing strategy, most network parameters do not require backpropagation of gradients, reducing memory usage during training and lowering the training pressure. Since the features extracted from the bottom layer of the encoding module are more abstract and general, they can adapt to downstream tasks without adjustment. Only the top layer, which is strongly correlated with the downstream task, is adjusted.

[0145] Existing multimodal text-to-video methods often require fine-tuning the entire model, which not only consumes a large amount of GPU memory but also disrupts the model's original pre-trained weights. The strategy of partially freezing the encoder not only improves performance but also significantly reduces computational burden and GPU memory consumption.

[0146] Optionally, the visual encoding module involves multiple layers of a neural network. The phrase "training the video classification model based on the training samples with the objective of minimizing the loss function" in section 206 above includes:

[0147] 2061. Freeze the first part of the neural network layer in the multi-layer neural network involved in the visual encoding module.

[0148] 2062. With the goal of minimizing the loss function, the network parameters of the second part of the neural network layer in the multi-layer neural network involved in the visual encoding module are fine-tuned based on the training samples.

[0149] In the above 2061, during the training of the model based on the above training samples, the network parameters of the frozen neural network layers will not be adjusted.

[0150] In the above 2062, the first part of the neural network layer has a higher level than the second part of the neural network layer. The number of the first part of the neural network layer and the number of the second part of the neural network layer can be set according to actual needs, and this application embodiment does not specifically limit this.

[0151] Optionally, the text encoding module involves multiple layers of a neural network. The phrase "training the video classification model based on the training samples with the objective of minimizing the loss function" in section 106 above includes:

[0152] 2063. Freeze the third part of the neural network layer in the multi-layer neural network involved in the text encoding module.

[0153] 2064. With the goal of minimizing the loss function, the network parameters of the fourth part of the neural network layer in the multi-layer neural network involved in the text encoding module are fine-tuned based on the training samples.

[0154] In the above 2063, during the training of the model based on the above training samples, the network parameters of the frozen neural network layers will not be adjusted.

[0155] In the above 2064, the third part of the neural network layer has a higher level than the fourth part of the neural network layer. The number of the third part of the neural network layer and the number of the fourth part of the neural network layer can be set according to actual needs, and this application embodiment does not specifically limit this.

[0156] Figure 4 This illustration shows a flowchart of a model training method provided in another embodiment of this application. The execution entity of this method can be a client or a server. The client can be hardware with embedded programs integrated on a terminal, application software installed on the terminal, or utility software embedded in the terminal's operating system, etc., and this embodiment does not limit its scope. The terminal can be any terminal device, including mobile phones and computers. The server can be a conventional server, a cloud server, or a virtual server, etc., and this embodiment does not specifically limit its scope. Figure 4 As shown, the method includes:

[0157] 401. Obtain training samples.

[0158] The training samples include: the video frame sequence corresponding to the sample video, the text word sequence, and the true category to which the sample video belongs.

[0159] 402. With the goal of minimizing the loss function, train the video classification model to be trained based on the training samples.

[0160] The loss function is determined based on the difference between the classification result output by the video classification model and the true category to which the sample video belongs. The video classification model includes a visual encoding module, a text encoding module, a compression module connected to the visual encoding module and the text encoding module respectively, and a feature fusion module connected to the compression module. The compression module contains multiple network parameters, which are learnable during training.

[0161] The visual encoding module is used to perform feature encoding on the input video frame sequence to obtain a visual feature sequence; the text encoding module is used to perform feature encoding on the input text word sequence to obtain a text feature sequence; the compression module is used to perform length compression processing on the visual feature sequence and the text feature sequence respectively to obtain a compressed visual feature sequence and a compressed text feature sequence; the compression module involves learnable network parameters; the feature fusion module is used to perform feature fusion on the compressed visual feature sequence and the compressed text feature sequence for classification.

[0162] The specific implementation process of steps 401 and 402 above can be found in the corresponding contents of the above embodiments, and will not be repeated here.

[0163] It should be noted that any steps in the method provided in this application that are not described in detail can be found in the corresponding content of the above embodiments, and will not be repeated here. Furthermore, the method provided in this application may include other parts or all of the steps in the above embodiments in addition to the steps described above; for details, please refer to the corresponding content of the above embodiments, and will not be repeated here.

[0164] Figure 5 A schematic diagram of the structure of an electronic device according to an embodiment of this application is shown. Figure 5As shown, the electronic device includes a memory 1101 and a processor 1102. The memory 1101 can be configured to store various other data to support operation on the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device. The memory 1101 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Electrically Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0165] The memory 1101 is used to store programs;

[0166] The processor 1102 is coupled to the memory 1101 and is used to execute the program stored in the memory 1101 to implement the methods provided in the above-described method embodiments.

[0167] Furthermore, such as Figure 5 As shown, the electronic device also includes: communication component 1103, display 1104, power supply component 1105, audio component 1106, and other components. Figure 5 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 5 The components shown.

[0168] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the steps or functions of the methods provided in the above-described method embodiments.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM (Read Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A video classification model product, characterized in that, include: A visual encoding module, a text encoding module, a compression module connected to the visual encoding module and the text encoding module respectively, and a feature fusion module connected to the compression module; The visual encoding module encodes the video frame sequence corresponding to the video to be classified to obtain a visual feature sequence; the text encoding module encodes the text word sequence corresponding to the video to be classified to obtain a text feature sequence; the compression module has an attention mechanism and contains multiple network parameters; the network parameters have learnable characteristics during training, and the parameter values ​​of the network parameters contained in the compression module in the trained video classification model are learned during model training; The compression module is used to perform self-attention query on the visual feature sequence or text feature sequence, which is a long feature sequence, based on the query sequence determined as a short feature sequence according to the network parameters, to obtain a compressed feature sequence. The length of the compressed feature sequence is the same as the length of the short feature sequence. The feature fusion module is used to fuse the compressed visual feature sequence and the compressed text feature sequence to classify the video to be classified.

2. A training method for a video classification model, characterized in that, include: Obtain training samples, which include: the video frame sequence, the text word sequence corresponding to the sample video, and the true category to which the sample video belongs; With the objective of minimizing the loss function, a video classification model is trained based on the training samples. The loss function is determined based on the difference between the classification result output by the video classification model and the true category of the sample video. The video classification model includes: a visual encoding module, a text encoding module, a compression module connected to both the visual encoding module and the text encoding module, and a feature fusion module connected to the compression module. The visual encoding module encodes the video frame sequence corresponding to the video to be classified, obtaining a visual feature sequence. The text encoding module encodes the text word sequence corresponding to the video to be classified, obtaining a text feature sequence. The compression module has an attention mechanism and contains multiple network parameters. These network parameters are learnable during training. The compression module performs self-attention queries on either the visual feature sequence or the text feature sequence (which is a long feature sequence) based on a query sequence determined by the network parameters (which is a short feature sequence), obtaining a compressed feature sequence with the same length as the short feature sequence. The feature fusion module fuses the compressed visual feature sequence and the compressed text feature sequence to classify the video to be classified.

3. A video classification method, characterized in that, include: Obtain the video frame sequence and text word sequence corresponding to the video to be classified, as well as the trained video classification model; The video classification model includes: a visual encoding module, a text encoding module, a compression module connected to the visual encoding module and the text encoding module respectively, and a feature fusion module connected to the compression module; wherein, the compression module has a self-attention mechanism and contains multiple network parameters; the network parameters have learnable characteristics during training, and the parameter values ​​of the network parameters contained in the compression module in the trained video classification model are learned during model training; Using the visual encoding module and text encoding module in the trained video classification model, feature encoding is performed on the video frame sequence and the text word sequence respectively to obtain visual feature sequence and text feature sequence; Using the compression module in the trained video classification model, a self-attention query is performed on the visual feature sequence (a long feature sequence) based on the query sequence determined by the network parameters to obtain a compressed visual feature sequence. Similarly, a self-attention query is performed on the text feature sequence (a long feature sequence) based on the query sequence determined by the network parameters, based on the query sequence determined by the network parameters to obtain a compressed text feature sequence. The lengths of the compressed visual feature sequence and the compressed text feature sequence are the same as the length of the short feature sequence. The compressed visual feature sequence and the compressed text feature sequence are fused using the feature fusion module in the trained video classification model to classify the video to be classified.

4. The method according to claim 3, characterized in that, Using a compression module, a self-attention query is performed on the visual feature sequence, which is a long feature sequence, based on the query sequence determined by the network parameters, to obtain a compressed visual feature sequence, including: The query sequence is determined based on the network parameters involved in the compression module; the length of the query sequence is less than the length of the visual feature sequence. For the element at the first position in the compressed visual feature sequence, the importance of each element in the visual feature sequence relative to the element at the first position in the compressed visual feature sequence is determined based on the similarity between the element at the first position in the query sequence and each element in the visual feature sequence; the first position can be any position in the compressed visual feature sequence. The compressed visual feature sequence is determined based on the visual feature sequence and the importance of each element in the visual feature sequence relative to the element in the first position of the compressed visual feature sequence.

5. The method according to claim 3, characterized in that, Using a compression module, a self-attention query is performed on the text feature sequence, which is a long feature sequence, based on the query sequence determined by the network parameters, to obtain a compressed text feature sequence. Using the self-attention mechanism of the compression module, the visual feature sequence and the text feature sequence are compressed to obtain compressed visual feature sequences and compressed text feature sequences, including: For the element at the second position in the compressed text feature sequence, the importance of each element in the text feature sequence relative to the element at the second position in the compressed text feature sequence is determined based on the similarity between the element at the second position in the query sequence and each element in the text feature sequence; the second position can be any position in the compressed text feature sequence. The compressed text feature sequence is determined based on the text feature sequence and the importance of each element in the text feature sequence relative to the element in the second position of the compressed text feature sequence.

6. The method according to any one of claims 2 to 5, characterized in that, The above text word sequence can be determined using one of the following methods: Generate the text word sequence based on the descriptive text information of the video to be classified; The video to be classified is subjected to optical character recognition to obtain the text word sequence; Audio recognition is performed on the audio in the video to be classified to obtain the text word sequence.

7. The method according to any one of claims 2 to 5, characterized in that, Also includes: Obtain training samples, which include: the sample video frame sequence corresponding to the sample video, the sample text word sequence, and the true category to which the sample video belongs; The video classification model is trained based on the training samples with the objective of minimizing the loss function; wherein the loss function is determined based on the difference between the classification result output by the video classification model and the true category to which the sample video belongs.

8. The method according to claim 7, characterized in that, Before training the video classification model based on the training samples, the method further includes: The network parameters contained in the visual encoding module are initialized using a contrastive language-image pre-training method.

9. The method according to claim 8, characterized in that, The visual encoding module involves multiple neural network layers; The video classification model is trained based on the training samples with the objective of minimizing the loss function, including: Freeze the first part of the neural network layer in the multi-layer neural network involved in the visual encoding module; With the goal of minimizing the loss function, the network parameters of the second part of the neural network layer in the multi-layer neural network involved in the visual encoding module are fine-tuned based on the training samples; The first part of the neural network layer has a higher layer level than the second part of the neural network layer.

10. An electronic device, characterized in that, include: Memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method of any one of claims 2 to 9.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a computer, it can implement the method of any one of claims 2 to 9.

Images