Short video multi-label classification method based on depth matrix decomposition

Abstract

This invention discloses a multi-label classification method for short videos based on deep matrix factorization (DMF). This method uses DMF as its basic framework, mining high-level semantic features hidden within short videos and tags through the parameter matrices and latent representation matrices at each layer of DMF. It enhances the expressive power of the latent representation matrix through encoding and decoding. Tag correlation information is embedded into the short video content at the source level using inverse matrix analysis. Finally, supervised constraints guide the learning of the parameter matrix and latent representation matrix. Compared with existing technologies, this invention can learn complex, structured, and hierarchical effective information from short videos, effectively mining high-level semantic features and tag correlations for multi-label classification of short videos.

Description

Technical Field

[0001] This invention relates to the field of multimedia content understanding, and in particular to a method for multi-tag classification of short videos. Background Technology

[0002] With the advent of the information age, emerging media, primarily social media platforms and short videos, are gaining increasing popularity among users, especially young people who prefer to showcase their personality and share memorable moments through short videos. Therefore, video content understanding has become an important research direction in the field of advanced semantic cognition in computer vision. Short video multi-label classification, as a crucial research area in video content understanding, is currently still in its early stages.

[0003] Unlike traditional long videos, short videos are typically short and fast-paced, ranging from 5 to 15 seconds, with more flexible shooting angles, content, and duration. However, short videos exhibit a complex hierarchy of tags (including background, actions, and characters). Due to the limitations of time and content, characterizing the relationships between tags represented by these limited short videos becomes a significant challenge. To address this challenge, deep learning models are used to construct short video content understanding models. While deep learning is a commonly used tool in visual learning, it suffers from drawbacks such as a large number of parameters and high computational requirements. Alternatively, deep models can be built from a machine learning perspective to learn the complex structured and hierarchical information of short videos, but these methods cannot acquire the same efficient intrinsic features as deep learning. In conclusion, learning the complex tag information corresponding to short video content is a crucial challenge for multi-tag classification of short videos.

[0004] Nevertheless, effectively learning the high-level semantic features of short video content and the complex relationships between the underlying tags remains an important research direction in multi-label classification. Therefore, developing a multi-label classification method for short videos that effectively mines high-level semantic features and tag relationships has significant application value. Summary of the Invention

[0005] To address the need for advanced semantic feature mining and tag association learning in short video multi-tag classification, this invention proposes a short video multi-tag classification method based on deep matrix factorization. Using deep matrix factorization as the basic framework, it mines advanced semantic features hidden in short videos and tags, enhances the expressive power of the latent representation matrix, and embeds tag association information into the short video at the source, thereby achieving short video multi-tag classification.

[0006] This invention is achieved through the following technical solution:

[0007] A multi-label classification method for short videos based on deep matrix factorization, comprising the following steps:

[0008] Step 1: Divide the short video into single-frame images and extract an equal number of keyframes from the short video frame sequence at equal time intervals.

[0009] Step 2: Use the GoogLeNet InceptionV2 network to extract depth features from the short video images in the keyframes;

[0010] Step 3: Extract the depth features of all keyframes to form a short video feature matrix, where the row vectors represent the depth features of a single frame image, and the depth feature matrix contains temporal and spatial information.

[0011] Step 4: Transform the short video feature matrix into a short video feature vector X by performing column vector mean transformation;

[0012] Step 5: Based on the deep matrix factorization framework, learn the visual representation matrix U from the short video feature vector X to supervise the multi-label classification task, and condense the classification task into the final visual representation matrix U.

[0013] Step 6: Analyze Ω using the inverse matrix. -1 The label association information contained in the label vector matrix L is mapped to the short video feature vector X in a certain way. The discriminant representation matrix V with label association information is mined based on the deep matrix factorization framework for supervising the multi-label classification task.

[0014] Step 7: Combine the visual representation matrix U and the discriminant representation matrix V, and enhance the reverse learning capability of the representation matrix within the framework of deep matrix factorization, to restore the short video feature vector X into the Hadaman product of the visual representation matrix U and the discriminant representation matrix V.

[0015] Step 8: Learn a label representation matrix A from the label matrix L that reflects the correlation information for use in supervising the multi-label classification task;

[0016] Step 9: The visual representation matrix U, the discriminant representation matrix V, the label representation matrix A, and the real label matrix Q of the short video constitute the supervision module;

[0017] Step 10: Construct a total objective function from the visual representation matrix U, the discriminant representation matrix V, the short video feature vector X, and the label representation matrix A to predict the multi-label classification of newly appearing short videos; the expression of the total objective function is as follows:

[0018]

[0019] in, The Hadaman product of the visual representation matrix U and the discriminant representation matrix V. transpose,

[0020] Let F represent the norm, and k represent the number of layers in the depth matrix factorization. D represents the Hadman product; i Let D1…D2 represent the coefficient matrix of the i-th layer. k Z represents a multi-level coefficient matrix. i Let L represent the parameter matrix of the i-th layer, and Z1…Z2 represent the label matrix. k Represents a multi-level parameter matrix, C1…C k C represents the coefficient matrix of the multi-level decomposition. i Let L represent the coefficient matrix of the i-th level decomposition. T Let L be the transpose of the label vector matrix L.

[0021] Compared with existing technologies, the beneficial technical effects achieved by this invention include learning complex, structured, and hierarchical effective information from short videos; and a multi-label classification method for short videos that effectively mines high-level semantic features and tag correlations. Attached Figure Description

[0022] Figure 1 This is an overall flowchart of the short video multi-label classification method based on deep matrix factorization of the present invention;

[0023] Figure 2 This is a diagram illustrating an embodiment of the short video multi-label classification method based on deep matrix factorization of the present invention. Detailed Implementation

[0024] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] like Figure 1 The diagram shown is an overall flowchart of the short video multi-label classification method based on deep matrix factorization of the present invention. The process specifically includes the following steps:

[0026] Step 1: Divide the short video into single-frame images, and extract an equal number of keyframes from the short video frame sequence at equal time intervals. The information contained in the keyframes represents the content of the short video.

[0027] Step 2: Use the GoogLeNet InceptionV2 network to extract the depth features of the image from the keyframes, and use the output of the first fully connected layer of GoogLeNet InceptionV2 as the depth features.

[0028] Step 3: Extract the depth features of all keyframes to form a short video feature matrix, where the row vectors represent the depth features of a single frame image, and the depth feature matrix contains temporal and spatial information.

[0029] Step 4: Transform the short video feature matrix into a feature vector X by performing column vector mean transformation;

[0030] Depending on the actual situation, the PCA algorithm can be used to reduce the dimensionality of the original feature vector of the short video when necessary, further removing redundant information contained in the original feature vector and reducing the subsequent computational load, to obtain the final short video feature vector X.

[0031] Step 5: Based on the deep matrix factorization framework, learn the visual representation matrix U from the short video feature vector X to supervise the multi-label classification task, condensing the classification task into the final visual representation matrix U. The expression for this step is as follows:

[0032]

[0033] Among them, B1…B k This represents the multi-level decomposition coefficient matrix, where k represents the total number of levels in the multi-level decomposition coefficient matrix. Let F represent the norm and k represent the number of layers in the deep matrix factorization, and U represent the visual representation matrix;

[0034] Step 6: Analyze Ω using the inverse matrix. -1 The label association information contained in the label vector matrix L is mapped to the short video feature vector X in a certain way. Based on the deep matrix factorization framework, a discriminant representation matrix V with label association information is mined for supervising the multi-label classification task. The expression for this step is as follows:

[0035]

[0036] Among them, C1…C k Represents the multilevel decomposition coefficient matrix and the discriminant characterization matrix V, C i Let L represent the coefficient matrix of the i-th level decomposition. T This represents the transpose of the label vector matrix L;

[0037] Step 7: Combine the visual representation matrix U and the discriminant representation matrix V, and enhance the reverse learning capability of the representation matrix within the framework of deep matrix factorization, to restore the short video feature vector X into the Hadaman product of the visual representation matrix U and the discriminant representation matrix V. The expression for this step is as follows:

[0038]

[0039] in, D represents the Hadman product; i Let D1…D2 represent the coefficient matrix of the i-th layer. k Represents a multi-level coefficient matrix;

[0040] Low-rank constraints are used to eliminate redundant information in the representation matrices U and V, as shown below.

[0041]

[0042] Where Rank represents the representation matrix. The purpose of this process is to eliminate the rank of the rank. This operation removes redundant information from the model, which helps improve the multi-label classification performance of the entire model.

[0043] Step 8: Learn a label representation matrix A from the label matrix L that reflects the correlation information for use in supervising the multi-label classification task. The expression for this step is as follows:

[0044]

[0045] Among them, Z i Let A represent the parameter matrix of the i-th layer, A represent the label representation matrix, L represent the label matrix, and Z1…Z2 represent the label matrix. k Represents a multi-level parameter matrix;

[0046] Step 9: Construct a supervision module using the visual representation matrix U, the discriminant representation matrix V, the label representation matrix A, and the real label matrix Q of the short video. Introduce the real label Q of the short video and... The constraint supervision term for A guides the optimization direction of the entire multi-label classification model. The expression for this step is as follows:

[0047]

[0048] Step 10: Construct the overall objective function from the visual representation matrix U, discriminant representation matrix V, short video feature vector X, and label representation matrix A to predict the multi-label classification of newly emerging short videos; that is: construct the objective function from the latent visual representation matrix U, discriminant representation matrix V, short video feature vector X, and label representation matrix A, and use ADMM (an algorithm for solving multivariate objective functions, which decomposes an objective function containing multiple variables into multiple single-variable functions and solves for each single variable by using the reciprocal method) to process each variable (U, V, A, Ω, B). i C i D i Z i The expression for the overall objective function of this invention is as follows:

[0049]

[0050] in, The Hadaman product of the visual representation matrix U and the discriminant representation matrix V. transpose, Let F represent the norm, and k represent the number of layers in the depth matrix factorization. D represents the Hadman product; i Let D1…D2 represent the coefficient matrix of the i-th layer. k Z represents a multi-level coefficient matrix. i Let L represent the parameter matrix of the i-th layer, and Z1…Z2 represent the label matrix. k Represents a multi-level parameter matrix, C1…C k C represents the coefficient matrix of the multi-level decomposition. i Let L represent the coefficient matrix of the i-th level decomposition. T The transpose of the label vector matrix L;

[0051] Formula (7) is the overall objective function of the short video multi-label classification method of this invention. By using the ADMM algorithm to solve for each variable in the objective function, when the value of the overall objective function tends to a stable convergence state, the value of each variable at this time is the optimal value for multi-label classification. When there is new short video data, the corresponding visual feature X is generated using steps 1, 2, 3 and 4, and X is compared with D1…D k Multiplying these components yields the representation matrix of the short video. By calculating the mean μ and variance σ of the representation matrix, elements in the representation matrix V that are greater than μ+3σ are set to 1. The predicted tag for the short video can be determined by querying the relationship between the position number and the tag set.

[0052] In summary, the present invention has the following advantages:

[0053] 1. By utilizing the multi-level parameter matrix and latent representation matrix of deep matrix factorization, complex, structured, and hierarchical effective information can be learned from short videos;

[0054] 2. Utilizing the concept of encoding and decoding, the process of deep matrix factorization is similar to the encoding process, which is a learning process from the data end to the high-level latent representation matrix; therefore, introducing a decoding process from the latent representation matrix to the data can effectively deepen the expressive power of the latent representation matrix.

[0055] 3. A tag matrix corresponding to the short video is generated using word vectors, and the information contained in the tag matrix is ​​embedded into the short video feature vector through inverse matrix analysis. This method can effectively enrich the information richness of the short video, and a latent representation matrix for discrimination is obtained using deep matrix factorization.

[0056] 4. Based on deep matrix factorization, the label information is embedded into the short video feature vector at the source by introducing the inverse matrix analysis method, thereby improving the information richness of the short video feature vector; further, the expressive power of the latent representation matrix obtained by deep matrix factorization is enhanced by using the idea of ​​encoding and decoding.

Claims

1. A multi-label classification method for short videos based on deep matrix factorization, characterized in that, The method includes the following steps: Step 1: Divide the short video into single-frame images and extract an equal number of keyframes from the short video frame sequence at equal time intervals. Step 2: Use the GoogLeNet InceptionV2 network to extract depth features from the short video images in the keyframes; Step 3: Extract the depth features of all keyframes to form a short video feature matrix, where the row vectors represent the depth features of a single frame image, and the depth feature matrix contains temporal and spatial information. Step 4: Transform the short video feature matrix into a short video feature vector X by performing column vector mean transformation; Step 5: Based on the deep matrix factorization framework, learn the visual representation matrix U from the short video feature vector X to supervise the multi-label classification task, and condense the classification task into the final visual representation matrix U. Step 6: Analyze using the inverse matrix The label association information contained in the label vector matrix L is mapped to the short video feature vector X in a certain way. The discriminant representation matrix V with label association information is mined based on the deep matrix factorization framework for supervising the multi-label classification task. Step 7: Combine the visual representation matrix U and the discriminant representation matrix V, and enhance the reverse learning capability of the representation matrix within the framework of deep matrix factorization to restore the short video feature vector X into the Hadaman product of the visual representation matrix U and the discriminant representation matrix V. ; Step 8: Learn a label representation matrix A from the label matrix L that reflects the correlation information for use in supervising the multi-label classification task; Step 9: The visual representation matrix U, the discriminant representation matrix V, the label representation matrix A, and the real label matrix Q of the short video constitute the supervision module; Step 10: Construct a total objective function from the visual representation matrix U, the discriminant representation matrix V, the short video feature vector X, and the label representation matrix A to predict the multi-label classification of newly appearing short videos; the expression of the total objective function is as follows: ; in, The Hadaman product of the visual representation matrix U and the discriminant representation matrix V. Transpose of; Let F be the norm, k be the depth factorization level, o be the Hadaman product, and D be the depth factorization level. i D1LD represents the coefficient matrix of the i-th layer. k Z represents a multi-level coefficient matrix. i Let Z1LZ represent the parameter matrix of the i-th layer, L represent the label matrix, and Z1LZ represent the parameter matrix of the i-th layer. k Represents a multi-level parameter matrix, C1LC k Represents the multi-level decomposition coefficient matrix, C i Let L represent the coefficient matrix of the i-th level decomposition. T B1…B2 represents the transpose of the tag vector matrix L. k This represents the coefficient matrix of the multi-level decomposition.

Images