A short video multi-label classification method based on deep matrix decomposition

Abstract

The application discloses a short video multi-label classification method based on deep matrix decomposition, comprising the following steps: dividing a short video dataset into a training set, a validation set and a test set, extracting video features, audio features and optical flow features of the input short video through feature extraction; using an index matrix, a sample relationship matrix, a label coding matrix and a label relationship matrix to encode the potential correlation information of samples and labels, and performing deep matrix decomposition on the video features, the audio features and the optical flow features to obtain a general coding matrix; using the general coding matrix, individual decoding matrices of the video features, the audio features and the optical flow features are obtained through deep decomposition; and after splicing the individual decoding matrices of the video features, the audio features and the optical flow features, the spliced individual decoding matrices are input into a multi-label classifier to obtain the final multi-label classification result. The application effectively enhances the fusion ability between features, guarantees the completeness and uniqueness of the features, and improves the multi-label classification accuracy of the short video.

Description

Technical Field

[0001] This invention relates to the field of multi-label classification of short videos, and more particularly to a multi-label classification method for short videos based on deep matrix factorization. Background Technology

[0002] With the advent of the information age, short videos have gradually replaced traditional images and text, becoming an increasingly popular form of expression for users on social media. As a key research direction in machine learning, multi-label classification of short videos is still in its early stages. Its main challenges lie in the significant semantic gap between the various low-level features of short videos and the semantics of high-level labels, as well as the diversity of labels. To address the first challenge, most current algorithms focus on the design of general feature extraction, utilizing matrix factorization or attention mechanisms to extract general features. For the second challenge, graph neural network learning based on label correlation has emerged as a new learning paradigm. Graph neural network learning is a broader learning paradigm, capable of more effectively expressing the intrinsic relationships and importance of labels.

[0003] Nevertheless, effectively bridging the semantic gap between low-level features and high-level label semantics, and fully leveraging the correlations between multiple labels, remain valuable research directions. Therefore, proposing a short video multi-label classification algorithm that effectively extracts general and individual features and fully explores the correlations between multiple labels is meaningful. Summary of the Invention

[0004] This invention provides a multi-label classification method for short videos based on deep matrix factorization. Addressing the shortcomings of current multi-label classification algorithms for short videos, this invention proposes a method based on deep matrix factorization that fully mines the general and individual semantic features of video, audio, and optical flow, supplements the potential relevant information between samples and labels, and obtains a more complete feature representation of short videos, thereby enabling more accurate multi-label classification of short videos. See the description below for details:

[0005] A multi-label classification method for short videos based on deep matrix factorization, comprising the following steps:

[0006] The short video dataset is divided into a training set, a validation set, and a test set. Video features, audio features, and optical flow features of the input short video are extracted through feature extraction.

[0007] The latent relevant information of samples and labels is encoded using the index matrix, sample relationship matrix, label encoding matrix, and label relationship matrix. The video features, audio features, and optical flow features are respectively subjected to deep matrix decomposition to obtain a general encoding matrix.

[0008] Using a general coding matrix, individual decoding matrices for video features, audio features, and optical flow features are obtained through depth decomposition.

[0009] The individual decoding matrices of video features, audio features, and optical flow features are concatenated and then input into a multi-label classifier to obtain the final multi-label classification result.

[0010] The extracted video features, audio features, and optical flow features of the input short video are as follows:

[0011] The video feature extraction method for the input short video is Inception v2, the audio feature extraction method is MFCC, and the optical flow feature extraction method is to use Inception v2 on the result obtained by the Lucas-Kanade optical flow algorithm.

[0012] Furthermore, the specific steps of encoding the potential relevant information of samples and labels using the index matrix, sample relationship matrix, label encoding matrix, and label relationship matrix are as follows:

[0013] Q k ∈R N×N Z k ∈R N×N P k ∈R N×C , L∈R C×C

[0014] Where R represents the real number field, C represents the label category, N represents the number of samples in the dataset, each sample has at least one label, and the index matrix Q... k ={0,1} N×N The value of Q is only on the diagonal, if Q k,ii If the value is 1, then the i-th sample is selected; otherwise, it is not selected. For the sample relationship matrix Z... k ∈R N×N If sample i has a labels and sample j has b labels, and if sample i and sample j share n identical labels, then Z ij =n; Tag encoding matrix P k It is a One-Hot encoding matrix of sample label information, that is, for sample i in the i-th row, if it belongs to label j, then P k,ij =1, otherwise 0; Label relation matrix L∈R C×C It involves adaptive encoding of the potential relational structure information between tags.

[0015] The method of using a universal coding matrix and depth decomposition to obtain individual decoding matrices for video features, audio features, and optical flow features is as follows:

[0016]

[0017] in, v = 1, 2, ... H represents the feature matrix extracted from the sample in the v-th view, where each column represents a feature matrix with D. v A sample of dimension ||·|| F Describing the F-norm, It is a factor matrix. It is a universal encoding matrix. It is a decoding matrix.

[0018] The beneficial effects of the technical solution provided by this invention are:

[0019] 1. This invention utilizes an index matrix, a sample relationship matrix, a label encoding matrix, and a label relationship matrix to encode the latent relevant information of samples and labels, learning the deep-level relationship between samples and labels; it performs deep matrix decomposition on video features, audio features, and optical flow features respectively to mine more latent representations and obtain a universal encoding matrix;

[0020] 2. This invention utilizes deep matrix factorization to obtain the individual encoding information of each feature in the general encoding matrix. Unlike directly concatenating each feature, which can only represent the short video from a one-sided perspective, this mechanism of decoding the encoded matrix can make the final feature take into account both the general and individual information of each feature in multiple dimensions, and learn more complete content features in the short video.

[0021] 3. This invention differs from traditional matrix factorization methods. The deep matrix factorization method based on bidirectional ensemble learning of samples and labels can adaptively generate general and individual matrices, and can effectively capture the commonalities and individualities among video features, audio features and optical flow features.

[0022] 4. The method of this invention enhances the representation ability of the spliced ​​features by fusing video features, audio features, optical flow features, and relevant information between multiple tags, providing a new approach to solving multi-tag classification of short videos. Attached Figure Description

[0023] Figure 1 This is an overall flowchart of the short video multi-label classification method based on deep matrix factorization of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0025] Figure 1 This is an overall flowchart of the short video multi-label classification method based on deep matrix factorization according to an embodiment of the present invention. The process specifically includes the following steps:

[0026] Step 1: Extract video features, audio features, and optical flow features from the input short video. Use Inception v2 (the second generation of the initial network) to extract video features from the input short video, use the MFCC (Melch Cepstral Coefficients) algorithm to extract audio features, use the Lucas-Kanade optical flow algorithm to obtain intermediate features, and then use Inception v2 to extract optical flow features based on the intermediate features.

[0027] Step 2: There is a potential correlation structure between the dataset samples and labels. The potential related information between the samples and labels is encoded using the index matrix, sample relationship matrix, label encoding matrix, and label relationship matrix. Deep matrix decomposition is performed on the video features, audio features, and optical flow features respectively to mine more potential representations and obtain a general encoding matrix.

[0028] The feature representation of the three views obtained in step 1, namely video, audio, and optical flow, is as follows: Where v represents the feature matrix extracted from the dataset samples in the v-th view, and each column represents the feature matrix with D. v Dimensional samples.

[0029] To achieve bidirectional ensemble learning of sample and label information, the latent correlation structure of samples and labels is learned, and an index matrix Q is introduced respectively. k ∈R N×N Sample Relationship Matrix Z k ∈R N×N and label encoding matrix P k ∈R N×C Label relation matrix L∈R C×C Encode the potential relationship information between samples and labels.

[0030] Where the index matrix Q k ={0,1} N×N The value of Q is only on the diagonal, if Q k,ii If the index is 1, then the i-th sample is selected; otherwise, it is not selected. Each time, 2 / K × 100% of the samples are drawn from each class, for a total of K times. The latent semantic space is constructed by combining the selections from multiple index matrices. Z k ∈R N×N If sample i has a labels and sample j has b labels, and if sample i and sample j share n (n≤a,n≤b) identical labels, then Z ij =n.

[0031] To enhance the discriminability of the relationship between samples and labels, a label encoding matrix P is introduced. k P k It is a One-Hot encoding matrix of sample label information, that is, for sample i in the i-th row, if it belongs to label j, then P k,ij=1, otherwise 0. Unlike the explicit relationships between known samples, the relationships between labels are more complex and deeper, especially for unknown input data. Manually constructed label relevance structures can easily lead to poor data compatibility. To learn deeper relationships between labels, a label relationship matrix L∈R is defined. C×C Adaptive encoding is performed on the potential relational structure information between tags.

[0032] Step 3: Multi-label classification of short videos depends not only on the latent correlation structure between samples and labels, but also on the general and individual semantic features existing among the video features, audio features, and optical flow features of the short videos. Using the general encoding matrix, individual decoding matrices for video features, audio features, and optical flow features are obtained through deep matrix decomposition. The deep matrix decomposition method used to mine general and individual semantic features is calculated as follows:

[0033]

[0034] in, This represents the feature matrix extracted from the sample in the v-th view, where each column represents the feature matrix with D. v A sample of dimension. ||·|| F Describing the F-norm, It is a factor matrix. It is a universal encoding matrix. It is a decoding matrix.

[0035] The embodiments of the present invention are described as follows:

[0036] The dataset used 125,368 short videos from the Meipai Challenge, each officially tagged by Meitu, with a total of 63 tag categories. The dataset was divided into training, validation, and test sets, each tagged with D... train D valid and D test Indicated. Using L train ={l1,l2,...,l N}、L valid ={l1,l2,...,l N} and L test ={l1,l2,...,l N} represent the training set, validation set, and test set multi-labels respectively. Each item in L corresponds to a 63-dimensional vector, representing the representation of the 63 categories of the corresponding sample.

[0037] The network model is built based on the TensorFlow deep learning framework. Inception v2 is used to extract video features from the input short video, the MFCC algorithm is used to extract audio features, and the Lucas-Kanade optical flow algorithm is used to obtain intermediate features. Then, Inception v2 is used again to extract optical flow features based on the intermediate features. All other parameters are randomly initialized. The Adam optimizer is used to decompose and fuse the video, audio, and optical flow features until convergence. The trained model is then used to fine-tune its parameters on the validation set. The test set data is then fed into the finely tuned model to obtain the final multi-label classification results.

[0038] There are currently five commonly used metrics for evaluating the performance of multi-label classification, and their expressions are as follows:

[0039]

[0040]

[0041]

[0042]

[0043]

[0044] Where N represents the number of samples, l n Y represents the number of labels. ij P represents whether the i-th sample contains the j-th label. ij This represents the prediction of the j-th label in the i-th sample, where xor represents the exclusive OR operation. For tag l n The predicted probabilities are sorted in descending order of all predicted probabilities, if the sample does not contain the label l. n ,but It will be updated to 0, and I(·) counts the number of times the condition within the parentheses is applied. L i This indicates the set of labels contained in the sample. This represents the set of labels not included in the sample. P L This refers to the ranking of the predicted labels for the samples.

[0045] The experimental results are shown in Table 1. Precision, HammingLoss, Coverage, and RangingLoss are all superior to other methods, while OneError is slightly worse. Overall, the combined effect is better than other methods.

[0046] Table 1 Comparison of different methods

[0047]

[0048] In summary, this invention, through joint deep matrix factorization and encoding / decoding, extracts deeper-level general and individual feature representations, greatly enriching both general and individual features. This effectively enhances the fusion capability of video, audio, and optical flow features in short videos, ensuring feature completeness and uniqueness, and improving the accuracy of multi-label classification for short videos. This invention combines multiple concepts such as deep matrix factorization, encoding / decoding, and learning the intrinsic relationship between samples and labels, making it particularly suitable for large-scale multi-label classification of short videos in low-power scenarios.

[0049] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0050] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-label classification method for short videos based on deep matrix factorization, the method comprising the following steps: The short video dataset is divided into a training set, a validation set, and a test set. Video features, audio features, and optical flow features of the input short video are extracted through feature extraction. The latent relevant information of samples and labels is encoded using the index matrix, sample relationship matrix, label encoding matrix, and label relationship matrix. The video features, audio features, and optical flow features are respectively subjected to deep matrix decomposition to obtain a general encoding matrix. Using a general coding matrix, individual decoding matrices for video features, audio features, and optical flow features are obtained through depth decomposition. The individual decoding matrices of video features, audio features, and optical flow features are concatenated and then input into a multi-label classifier to obtain the final multi-label classification result. The specific steps for encoding the potential relevant information of samples and labels using the index matrix, sample relationship matrix, label encoding matrix, and label relationship matrix are as follows: ， ， ， ； in, Represents the real number field. Indicates the label category, The index matrix represents the number of samples in the dataset, with each sample having at least one label. The value is only on the diagonal, if Then select the first one. If there are 1 sample, otherwise no sample is selected; for the sample relationship matrix If the sample have One label, sample have Each label, if the sample and samples If they share n identical labels, then Tag encoding matrix It is the One-Hot encoding matrix of the sample label information, that is, for the first... Row samples If it belongs to the tag ,but Otherwise Tag Relationship Matrix It involves adaptive encoding of the potential relational structure information between tags; The method of using a general coding matrix and through depth decomposition to obtain individual decoding matrices for video features, audio features, and optical flow features are as follows: ； in, Indicates the first Each view extracts a feature matrix from the samples, where each column represents a feature matrix with... Dimensional samples, Denotes the F-norm, It is a factor matrix. It is a universal encoding matrix. It is a decoding matrix.

2. The short video multi-label classification method based on deep matrix factorization according to claim 1, characterized in that, The extracted video features, audio features, and optical flow features of the input short video are as follows: The video feature extraction method for the input short video is Inception v2, the audio feature extraction method is MFCC, and the optical flow feature extraction method is to use Inception v2 on the result obtained by the Lucas-Kanade optical flow algorithm.

Images