A video classification method and apparatus
By using convolutions with different kernels for feature extraction and sorting updates in video classification, the problem of lost spatiotemporal information in video classification is solved, and higher classification accuracy is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HISENSE GRP HLDG CO LTD
- Filing Date
- 2022-06-30
- Publication Date
- 2026-07-07
AI Technical Summary
In existing technologies, the video classification process loses some spatiotemporal information, resulting in reduced classification accuracy.
Feature extraction is performed by convolution with different kernels to obtain the feature vector sequence and target feature vector corresponding to each kernel. The target feature vectors are sorted according to the size of the kernel. For any target feature vector, it is updated based on the target feature vectors adjacent to it. Finally, all feature vector sequences and updated target feature vectors are fused to obtain a classification vector representing the video category.
By reflecting the correlation between different target feature vectors and including global view information, the accuracy of video classification is improved.
Smart Images
Figure CN115223079B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video processing technology, and in particular to a video classification method and apparatus. Background Technology
[0002] With the rapid popularization of mobile internet, videos have become increasingly popular due to their rich content and diverse forms of expression. To facilitate video management, it is necessary to categorize videos, that is, to determine the category to which each video belongs.
[0003] In related technologies, two parallel convolutional neural networks (one slow channel and one fast channel) are applied to process the same video segment. The slow channel is used to analyze the static content in the video, and the fast channel is used to analyze the dynamic content in the video.
[0004] However, the above processing will lose some spatiotemporal information. For example, when constructing a slow channel stream, downsampling will cause the loss of temporal information, which will reduce the accuracy of video classification. Summary of the Invention
[0005] This application provides a video classification method and apparatus for accurate video classification.
[0006] In a first aspect, embodiments of this application provide a video classification method, the method comprising:
[0007] The video to be processed is subjected to convolution with different convolution kernels to extract features, thereby obtaining the feature vector sequence corresponding to each convolution kernel and the target feature vector, and the target feature vector is sorted based on the size of the convolution kernel;
[0008] For any target feature vector, the target feature vector is updated based on the target feature vectors adjacent to the target feature vector.
[0009] Feature fusion is performed on all feature vector sequences of the video to be processed and the updated target feature vector to obtain a classification vector representing the category of the video to be processed.
[0010] Secondly, embodiments of this application provide a video classification device, comprising:
[0011] The feature extraction module is used to extract features from the video to be processed by convolution with different convolution kernels, obtain the feature vector sequence corresponding to each convolution kernel and the target feature vector, and sort the target feature vectors based on the size of the convolution kernel;
[0012] The update module is used to update the target feature vector based on the target feature vectors adjacent to the target feature vector for any given target feature vector;
[0013] The fusion module is used to perform feature fusion on all feature vector sequences of the video to be processed and the updated target feature vector to obtain a classification vector representing the category of the video to be processed.
[0014] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory;
[0015] The memory stores program code that, when executed by the processor, causes the processor to perform the video classification method as described in any of the first aspects.
[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the video classification method as described in any of the first aspects.
[0017] Furthermore, the technical effects of any of the implementation methods in the second to fourth aspects can be found in the technical effects of different implementation methods in the first aspect, and will not be repeated here. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 A schematic flowchart illustrating the first video classification method provided in this application embodiment;
[0020] Figure 2 A first system architecture diagram provided for embodiments of this application;
[0021] Figure 3 A second system architecture diagram provided for embodiments of this application;
[0022] Figure 4 A schematic flowchart illustrating the second video classification method provided in this application embodiment;
[0023] Figure 5 A schematic flowchart illustrating the feature vector sequence and target feature vector determination method provided in the embodiments of this application;
[0024] Figure 6 A schematic flowchart illustrating the third video classification method provided in this application embodiment;
[0025] Figure 7A schematic flowchart illustrating the fourth video classification method provided in this application embodiment;
[0026] Figure 8 A schematic flowchart illustrating the target feature vector update method provided in this application embodiment;
[0027] Figure 9 A schematic flowchart illustrating the fifth video classification method provided in this application embodiment;
[0028] Figure 10 A schematic flowchart illustrating the adjustment vector determination method provided in this application embodiment;
[0029] Figure 11 A schematic diagram illustrating the process of determining the adjustment vector provided in this application embodiment;
[0030] Figure 12 A schematic flowchart illustrating the first feature fusion method provided in this application embodiment;
[0031] Figure 13 A schematic flowchart illustrating the sixth video classification method provided in this application embodiment;
[0032] Figure 14 A schematic flowchart illustrating the second feature fusion method provided in this application embodiment;
[0033] Figure 15 A schematic diagram illustrating the change in the number of vectors provided in this application embodiment;
[0034] Figure 16 A schematic flowchart illustrating the seventh video classification method provided in this application embodiment;
[0035] Figure 17 A schematic flowchart illustrating the eighth video classification method provided in this application embodiment;
[0036] Figure 18 This is a schematic diagram of the structure of the first video classification device provided in the embodiments of this application;
[0037] Figure 19 This is a schematic diagram of the structure of the second video classification device provided in the embodiments of this application;
[0038] Figure 20 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0039] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0040] In the description of this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, it can refer to a direct connection, an indirect connection through an intermediate medium, or a connection within two devices. Those skilled in the art can understand the specific meaning of the above term in this application based on the specific circumstances.
[0041] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0042] With the rapid popularization of mobile internet, videos have become increasingly popular due to their rich content and diverse forms of expression. To facilitate video management, it is necessary to categorize videos, that is, to determine the category to which each video belongs.
[0043] In related technologies, two parallel convolutional neural networks (one slow channel and one fast channel) are applied to process the same video segment. The slow channel is used to analyze the static content in the video, and the fast channel is used to analyze the dynamic content in the video.
[0044] However, the above processing will lose some spatiotemporal information. For example, when constructing a slow channel stream, downsampling will cause the loss of temporal information, which will reduce the accuracy of video classification.
[0045] See Figure 1 As shown, in some embodiments, video classification is performed in the following manner:
[0046] Step S101: Extract features from the video to be processed by convolution with different convolution kernels to obtain the feature vector sequence and target feature vector corresponding to each convolution kernel;
[0047] Step S102: Perform feature fusion on all feature vector sequences and target feature vectors through a cross-attention mechanism to obtain a classification vector, and determine the video category based on the classification vector.
[0048] See Figure 2 The diagram shows the system architecture corresponding to the above embodiments.
[0049] However, the target feature vectors in the above methods do not reflect the correlation between different target feature vectors and lack global view information; in addition, feature fusion through cross attention mechanism cannot effectively extract key information. Therefore, it is difficult to accurately determine the video type of the video to be processed based on the above classification vectors.
[0050] Based on this, embodiments of this application provide a video classification method and apparatus. The method includes: extracting features from a video to be processed by convolution with different convolution kernels to obtain a feature vector sequence and a target feature vector corresponding to each convolution kernel, and sorting the target feature vectors based on the size of the convolution kernels; updating the target feature vector based on the target feature vectors adjacent to the target feature vector for any target feature vector; and performing feature fusion on all feature vector sequences of the video to be processed and the updated target feature vectors to obtain a classification vector representing the category of the video to be processed.
[0051] See Figure 3 The diagram shows the system architecture corresponding to the above embodiments.
[0052] The above scheme, after obtaining the feature vector sequence corresponding to each convolution kernel and the target feature vector, sorts the target feature vectors based on the size of the convolution kernel. After sorting, it updates each target feature vector based on other target feature vectors adjacent to each target feature vector, so that the updated target feature vector reflects the correlation between different target feature vectors and contains global view information. Then, it performs feature fusion on the above feature vector sequence and the updated target feature vector to obtain a classification vector that can accurately represent the category of the video to be processed. Subsequently, the video can be accurately classified based on this classification vector.
[0053] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with reference to the accompanying drawings and specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0054] This application provides a second video classification method, such as... Figure 4 As shown, the method may include:
[0055] Step S401: Extract features from the video to be processed by convolution with different convolution kernels to obtain the feature vector sequence and target feature vector corresponding to each convolution kernel, and sort the target feature vectors based on the size of the convolution kernel.
[0056] In this embodiment, convolutions with different kernels (such as 3D convolutions) are set. Smaller kernels correspond to smaller tubelets (video objects) to capture fine-grained motion; larger kernels correspond to larger tubelets to capture slowly changing scenes. Therefore, feature extraction is performed through convolutions with different kernels to obtain comprehensive feature information.
[0057] Step S402: For any target feature vector, update the target feature vector based on the target feature vectors adjacent to the target feature vector.
[0058] The aforementioned target feature vectors do not reflect the correlation between different target feature vectors. Therefore, this embodiment sorts the target feature vectors according to the size of the convolution kernel, and then integrates the correlation between adjacent target feature vectors to update the target feature vectors. The updated target feature vectors reflect the correlation between different target feature vectors and include global view information. Subsequently, based on the updated target feature vectors, a more accurate classification vector representing the category of the video to be processed can be obtained.
[0059] Step S403: Perform feature fusion on all feature vector sequences of the video to be processed and the updated target feature vector to obtain a classification vector representing the category of the video to be processed.
[0060] The above scheme, after obtaining the feature vector sequence corresponding to each convolution kernel and the target feature vector, sorts the target feature vectors based on the size of the convolution kernel. After sorting, it updates each target feature vector based on other target feature vectors adjacent to each target feature vector, so that the updated target feature vector reflects the correlation between different target feature vectors and contains global view information. Then, it performs feature fusion on the above feature vector sequence and the updated target feature vector to obtain a classification vector that can accurately represent the category of the video to be processed. Subsequently, the video can be accurately classified based on this classification vector.
[0061] In some optional implementations, the above-described feature vector sequence and target feature vector determination method can be found in [reference needed]. Figure 5 As shown:
[0062] Step S501: For any convolution kernel, extract features from the video to be processed by convolving the convolution kernel to obtain multiple multidimensional matrices corresponding to the convolution kernel.
[0063] For example, the video to be processed is represented as V∈R T×H×W×CWhere T is the number of frames in the video, C is the number of channels in each frame, H is the height, and W is the width. The video to be processed is input into each convolutional layer, resulting in N multidimensional matrices output by each convolution. These matrices typically have dimensions t × h × w, and are represented as z ∈ R. N×t×h×w×C ,in,
[0064] Step S502: Perform linear transformations on the multiple multidimensional matrices respectively to obtain the feature vector sequence.
[0065] In this embodiment, each multidimensional matrix is linearly transformed to obtain a one-dimensional matrix, which forms the above-mentioned eigenvector sequence.
[0066] Step S503: Input the feature vector sequence and the preset vector into the encoder to obtain the target feature vector corresponding to the convolution kernel output by the encoder.
[0067] In this embodiment, to more fairly integrate the information in the feature vector sequence (token1, token2, ... token), N Add a learnable preset vector (token) before ) CLS Finally, position embedding is added.
[0068] will token CLS , token1, token2,…token N Input the encoder to get the token CLS ′, the token CLS ′ is the target feature vector corresponding to the convolution kernel.
[0069] Since the self-attention mechanism has quadratic complexity, it is computationally difficult to process all the above vector sequences together. Therefore, the encoder can be a multi-view encoder (Transformer), which consists of multi-head attention (MSA), layer normalization (LN), and multilayer perceptron (MLP).
[0070] For example, each set of vectors (the sequence of feature vectors and the preset vectors) is processed using a separate encoder (consisting of L Transformer layers). The transformation formula from the j-th layer to the (j+1)-th layer of the i-th vector in the Transformer is as follows:
[0071] y i,j =MSA(LN(z) i,j ))+z i,j
[0072] z i,j+1 =MLP(LN(y i,j ))+y i,j
[0073] After the view is processed by the Transformer, the token will be... CLS Corresponding vector (token) CLS ′) is used as the target feature vector.
[0074] Correspondingly, this application provides a third video classification method, such as... Figure 6 As shown, the method may include:
[0075] Step S601: For any convolution kernel, extract features from the video to be processed by convolving the convolution kernel to obtain multiple multidimensional matrices corresponding to the convolution kernel.
[0076] Step S602: Perform linear transformations on the multiple multidimensional matrices respectively to obtain the feature vector sequence.
[0077] Step S603: Input the feature vector sequence and the preset vector into the encoder to obtain the target feature vector corresponding to the convolution kernel output by the encoder.
[0078] Step S604: Sort the target feature vectors based on the size of the convolution kernel.
[0079] Step S605: For any target feature vector, update the target feature vector based on the target feature vectors adjacent to the target feature vector.
[0080] Step S606: Perform feature fusion on all feature vector sequences of the video to be processed and the updated target feature vector to obtain a classification vector representing the category of the video to be processed.
[0081] The specific implementation of steps S601 to S606 can be referred to the above embodiments, and will not be repeated here.
[0082] This application provides a fourth video classification method, such as... Figure 7 As shown, the method may include:
[0083] Step S701: Extract features from the video to be processed by convolution with different convolution kernels to obtain the feature vector sequence and target feature vector corresponding to each convolution kernel, and sort the target feature vectors based on the size of the convolution kernel.
[0084] The specific implementation of step S701 can be found in the above embodiments, and will not be repeated here.
[0085] Step S702: Input all target feature vectors of the video to be processed into the update model, and update the target feature vectors based on the target feature vectors adjacent to any target feature vector.
[0086] In this embodiment, the model is trained to learn the correlation between adjacent target feature vectors to obtain the above-mentioned updated model; then, based on the target feature vectors adjacent to each target feature vector, the updated model is used to accurately update each target feature vector.
[0087] Step S703: Perform feature fusion on all feature vector sequences of the video to be processed and the updated target feature vector to obtain a classification vector representing the category of the video to be processed.
[0088] The specific implementation of step S703 can be found in the above embodiments, and will not be repeated here.
[0089] The above scheme updates each target feature vector precisely by updating the model based on the target feature vectors adjacent to each target feature vector, so that the updated target feature vectors reflect the correlation between different target feature vectors.
[0090] In some optional implementations, the above-described target feature vector update method can be found in [reference needed]. Figure 8 As shown:
[0091] Step S801: Perform average pooling on the first feature vector of the k-th level and the second feature vector of the k-th level using the updated model to obtain the average vector of the k-th level.
[0092] Wherein, 1≤k≤K, K is the total number of levels in the iterative update model; the first feature vector of the first level is any target feature vector, and the second feature vector of the first level is the adjacent target feature vector.
[0093] In this embodiment, the update model is set with K update layers, meaning it needs to be iterated and updated K times.
[0094] For example, there are X target feature vectors in total; for the first target feature vector, the average vector z of the k-th level... avg 1,k =avg(z rep 1,k ,z rep 2,k );
[0095] For the x-th target feature vector (1≤x≤X-1), the average vector z of the k-th level avg x,k =avg(zrep x-1,k ,z rep x,k ,z rep x+1,k );
[0096] For the Xth target feature vector, the average vector z of the kth level avg X,k =avg(z rep X-1,k ,z rep X,k );
[0097] avg is the average pooling calculation, z rep x,k Let x be the first feature vector of the k-th level.
[0098] It can be understood that the second feature vector mentioned above corresponds to the adjacent target feature vectors. For the x-th target feature vector, the second feature vector of the first level is the (x+1)-th target feature vector and / or the (x-1)-th target feature vector, and the second feature vector of other levels is the (x+1)-th first feature vector and / or the (x-1)-th first feature vector calculated in the above manner.
[0099] Step S802: Perform a fully connected layer calculation on the average vector of the k-th layer to obtain the adjustment vector of the k-th layer.
[0100] After determining the average vector, a fully connected layer is needed to perform a fully connected calculation to determine the adjustment vector at that layer.
[0101] Step S803: The sum of the adjustment vector of the kth level and the first feature vector of the kth level is determined as the first feature vector of the (k+1)th level.
[0102] Wherein, the first feature vector of the Kth level is the updated target feature vector.
[0103] For example, for the x-th target feature vector (1≤x≤X-1), the first feature vector z of the (k+1)-th level avg x ,k+1 =△z x,k +z avg x,k ; where △z x,k z is the adjustment vector of the x-th target feature vector at the k-th level. avg x,k Let x be the first eigenvector.
[0104] Correspondingly, this application provides a fifth video classification method, such as... Figure 9 As shown, the method may include:
[0105] Step S901: Extract features from the video to be processed by convolution with different convolution kernels to obtain the feature vector sequence and target feature vector corresponding to each convolution kernel, and sort the target feature vectors based on the size of the convolution kernel.
[0106] Step S902: Perform average pooling on the first feature vector of the k-th level and the second feature vector of the k-th level using the updated model to obtain the average vector of the k-th level.
[0107] Step S903: Perform a fully connected layer calculation on the average vector of the k-th layer to obtain the adjustment vector of the k-th layer.
[0108] Step S904: The sum of the adjustment vector of the k-th level and the first feature vector of the k-th level is determined as the first feature vector of the (k+1)-th level; wherein, the first feature vector of the k-th level is the updated target feature vector.
[0109] Step S905: Perform feature fusion on all feature vector sequences of the video to be processed and the updated target feature vector to obtain a classification vector representing the category of the video to be processed.
[0110] The specific implementation of steps S901 to S905 can be referred to the above embodiments, and will not be repeated here.
[0111] The above scheme, for each iteration of the updated model, performs average pooling on the first feature vector and the second feature vector (adjacent feature vectors) of each level to obtain the average vector of the target feature vector at that level; then, through a fully connected layer, it calculates the adjustment vector of the target feature vector at that level; based on the adjustment vector of the target feature vector at that level, it adjusts the first feature vector of that level to obtain the first feature vector of the next level. After each layer of update iteration, more information from other target feature vectors can be aggregated, and after multiple iterations, the updated target feature vector is obtained.
[0112] In some optional implementations, the method for determining the adjustment vector at the k-th level described above can be found in [reference needed]. Figure 10 As shown:
[0113] Step S1001: Perform a first fully connected layer calculation on the average vector of the k-th level to obtain the first vector of the k-th level; and perform a second fully connected layer calculation on the average vector of the k-th level to obtain the second vector of the k-th level, and perform normalization calculation on the second vector of the k-th level to obtain the weight information of the k-th level.
[0114] In implementation, the average vector of the k-th level is input into two branches. The first branch uses a fully connected layer to perform a first fully connected layer calculation on the average vector of the k-th level to obtain the first vector of the k-th level. The second branch uses a fully connected layer and a normalization (SoftMax) layer. The fully connected layer performs a second fully connected layer calculation on the average vector of the k-th level to obtain the second vector of the k-th level. The normalization layer performs a normalization calculation on the second vector of the k-th level to obtain the weight information of the k-th level.
[0115] Step S1002: Based on the first vector of the k-th level and the weight information of the k-th level, obtain the adjustment vector of the k-th level.
[0116] For example, the average vector of the k-th level contains Y feature values, and the weight information includes the weight value corresponding to each feature value. Each feature value in the average vector of the k-th level is multiplied by its corresponding weight value to obtain the adjustment value corresponding to that feature value; the Y adjustment values constitute the aforementioned adjustment vector.
[0117] The above scheme obtains a first vector through fully connected computation; weight information is obtained through fully connected computation and normalization computation; based on the first vector and weight information, an adjustment vector representing the key information of the target feature is determined. Therefore, adjusting the first feature vector based on this adjustment vector can not only aggregate information from more other target feature vectors, but also retain the key information of the target feature vector.
[0118] See Figure 11 The diagram shown illustrates the process of determining the adjustment vector.
[0119] In some optional implementations, the above feature fusion method can be found in [reference needed]. Figure 12 As shown:
[0120] Step S1201: Concatenate all feature vector sequences of the video to be processed and the updated target feature vector to obtain an initial feature matrix.
[0121] In this embodiment, S 1×D dimensional vectors (S is the total number of the feature vector sequences of the video to be processed and the updated target feature vectors) are concatenated to obtain an initial feature matrix F, denoted as F∈i S×D .
[0122] Step S1202: Input the initial feature matrix into the fusion model, and perform feature fusion on the initial feature matrix through the fusion model to obtain a classification vector representing the category of the video to be processed.
[0123] By training the model, the relationship between the initial feature matrix and the classification vector is learned, and the above-mentioned fusion model is obtained. Then, the initial feature matrix is fused using this fusion model to accurately determine the classification vector.
[0124] Correspondingly, this application provides a sixth video classification method, such as... Figure 13 As shown, the method may include:
[0125] Step S1301: Extract features from the video to be processed by convolution with different convolution kernels to obtain the feature vector sequence and target feature vector corresponding to each convolution kernel, and sort the target feature vectors based on the size of the convolution kernel.
[0126] Step S1302: For any target feature vector, update the target feature vector based on the target feature vectors adjacent to the target feature vector.
[0127] Step S1303: Concatenate all feature vector sequences of the video to be processed and the updated target feature vector to obtain an initial feature matrix.
[0128] Step S1304: Input the initial feature matrix into the fusion model, and perform feature fusion on the initial feature matrix through the fusion model to obtain a classification vector representing the category of the video to be processed.
[0129] The specific implementation of steps S1301 to S1304 can be referred to the above embodiments, and will not be repeated here.
[0130] The above scheme uses a fusion model to fuse features from the initial feature matrix, effectively removing redundant information from the feature vectors while retaining key information, thus improving video classification accuracy.
[0131] In some optional implementations, the above feature fusion method can be found in [reference needed]. Figure 14 As shown:
[0132] Step S1401: Concatenate all feature vector sequences of the video to be processed and the updated target feature vector to obtain an initial feature matrix.
[0133] The specific implementation of step S1401 can be found in the above embodiments, and will not be repeated here.
[0134] Step S1402: Input the initial feature matrix into the fusion model, and determine the update matrix of the m-th level based on the adjacency matrix of the m-th level, the feature matrix of the m-th level, and the adjustment parameters of the m-th level.
[0135] Wherein, 1≤m≤M, M is the total number of levels in the iterative fusion of the fusion model; if 2≤m≤M, the adjacency matrix of the m-th level is determined based on the update matrix of the (m-1)-th level and the adjacency matrix of the (m-1)-th level, the adjacency matrix of the 1st level is a preset matrix, and the feature matrix of the 1st level is the initial feature matrix.
[0136] In some alternative implementations, the above fusion model is a Graph Convolutional Network (GCN).
[0137] For example, the initial feature matrix is represented as F∈i S×D The preset matrix is represented as A∈i S×S ;
[0138] The update matrix U at level m m =SoftMax[GCN(A m ,F m )];For example:
[0139] i represents the row of matrix A, and j represents the column of matrix A;
[0140] Where, Um∈R Sm×Sm+1 σ is the activation function, A m Let A be the adjacency matrix of the m-th level. m ∈R Sm×Sm A m =U m-1 T ×A m-1 ×U m-1 ;F m Let F be the feature matrix of the m-th level. m ∈R Sm×Dm w m w is the adjustment parameter for the m-th level. m ∈R Dm×Sm+1 .
[0141] Step S1403: The product of the inverse of the update matrix of the m-th level and the feature matrix of the m-th level is determined as the feature matrix of the (m+1)-th level; wherein, the feature matrix of the M-th level is the classification vector.
[0142] For example, F m+1 =U m T ×F m F m+1 ∈R Sm+1×Dm .
[0143] See Figure 15As shown, the number of vectors (nodes) in the feature matrix is continuously reduced by updating the matrix as described above, until the number of nodes is 1.
[0144] Figure 15 This example is merely illustrative of the variation in the number of nodes in the feature matrix and is not intended to be limiting.
[0145] Correspondingly, this application provides a seventh video classification method, such as... Figure 16 As shown, the method may include:
[0146] Step S1601: Extract features from the video to be processed by convolution with different convolution kernels to obtain the feature vector sequence and target feature vector corresponding to each convolution kernel, and sort the target feature vectors based on the size of the convolution kernel.
[0147] Step S1602: For any target feature vector, update the target feature vector based on the target feature vectors adjacent to the target feature vector.
[0148] Step S1603: Concatenate all feature vector sequences of the video to be processed and the updated target feature vector to obtain an initial feature matrix.
[0149] Step S1604: Input the initial feature matrix into the fusion model, and determine the update matrix of the m-th level based on the adjacency matrix of the m-th level, the feature matrix of the m-th level, and the adjustment parameters of the m-th level.
[0150] Step S1605: The product of the inverse of the update matrix of the m-th level and the feature matrix of the m-th level is determined as the feature matrix of the (m+1)-th level; wherein, the feature matrix of the M-th level is the classification vector.
[0151] The specific implementation of steps S1601 to S1605 can be referred to the above embodiments, and will not be repeated here.
[0152] In some optional implementations, embodiments of this application provide an eighth video classification method, such as... Figure 17 As shown, the method may include:
[0153] Step S1701: Extract features from the video to be processed by convolution with different convolution kernels to obtain the feature vector sequence and target feature vector corresponding to each convolution kernel, and sort the target feature vectors based on the size of the convolution kernel.
[0154] Step S1702: For any target feature vector, update the target feature vector based on the target feature vectors adjacent to the target feature vector.
[0155] Step S1703: Perform feature fusion on all feature vector sequences of the video to be processed and the updated target feature vector to obtain a classification vector representing the category of the video to be processed.
[0156] The specific implementation of steps S1701 to S1703 can be referred to the above embodiments, and will not be repeated here.
[0157] Step S1704: Based on a preset correspondence, determine the video category corresponding to the classification vector of the video to be processed; wherein, the preset correspondence includes the correspondence between the classification vector of the video and the video category.
[0158] The classification vector of the video to be processed represents the category of the video. By pre-setting the correspondence between the classification vector of the video and the video category, the video category of the video to be processed can be determined according to the correspondence.
[0159] Based on the aforementioned pre-defined correspondence, the above scheme can accurately and efficiently determine the video category (i.e., the category to which the video to be processed belongs) corresponding to the classification vector of the video to be processed.
[0160] like Figure 18 As shown, based on the same inventive concept, this application provides a video classification device 1800, comprising:
[0161] The feature extraction module 1801 is used to extract features from the video to be processed by convolution with different convolution kernels, to obtain the feature vector sequence corresponding to each convolution kernel and the target feature vector, and to sort the target feature vectors based on the size of the convolution kernel.
[0162] Update module 1802 is used to update the target feature vector based on the target feature vectors adjacent to the target feature vector for any target feature vector;
[0163] The fusion module 1803 is used to perform feature fusion on all feature vector sequences of the video to be processed and the updated target feature vector to obtain a classification vector representing the category of the video to be processed.
[0164] In some optional implementations, the update module 1802 is specifically used for:
[0165] All target feature vectors of the video to be processed are input into the update model, and the update model updates the target feature vectors based on the target feature vectors adjacent to any target feature vector.
[0166] In some optional implementations, the update module 1802 is specifically used for:
[0167] The average pooling operation is performed on the first feature vector and the second feature vector of the k-th level by the update model to obtain the average vector of the k-th level; where 1≤k≤K, K is the total number of levels updated by the update model; the first feature vector of the first level is any target feature vector, and the second feature vector of the first level is the adjacent target feature vector;
[0168] The average vector of the k-th level is calculated using a fully connected layer to obtain the adjustment vector of the k-th level;
[0169] The sum of the adjustment vector of the k-th level and the first feature vector of the k-th level is determined as the first feature vector of the (k+1)-th level; wherein, the first feature vector of the k-th level is the updated target feature vector.
[0170] In some optional implementations, the update module 1802 is specifically used for:
[0171] The average vector of the k-th level is calculated using a first fully connected layer to obtain a first vector of the k-th level; and the average vector of the k-th level is calculated using a second fully connected layer to obtain a second vector of the k-th level, and the second vector of the k-th level is normalized to obtain the weight information of the k-th level.
[0172] Based on the first vector of the k-th level and the weight information of the k-th level, the adjustment vector of the k-th level is obtained.
[0173] In some optional implementations, the fusion module 1803 is specifically used for:
[0174] The initial feature matrix is obtained by concatenating all feature vector sequences of the video to be processed and the updated target feature vector.
[0175] The initial feature matrix is input into the fusion model, and the fusion model performs feature fusion on the initial feature matrix to obtain a classification vector representing the category of the video to be processed.
[0176] In some optional implementations, the fusion module 1803 is specifically used for:
[0177] The update matrix of the m-th level is determined by the fusion model based on the adjacency matrix of the m-th level, the feature matrix of the m-th level, and the adjustment parameters of the m-th level; where 1≤m≤M, M is the total number of levels in the fusion model iterative fusion; if 2≤m≤M, the adjacency matrix of the m-th level is determined based on the update matrix of the (m-1)-th level and the adjacency matrix of the (m-1)-th level, the adjacency matrix of the 1-th level is a preset matrix, and the feature matrix of the 1-th level is the initial feature matrix;
[0178] The product of the inverse of the update matrix at level m and the feature matrix at level m is determined as the feature matrix at level (m+1); where the feature matrix at level M is the classification vector.
[0179] In some optional implementations, the feature extraction module 1801 is specifically used for:
[0180] For any convolution kernel, the video to be processed is convolved with the kernel to extract features, resulting in multiple multidimensional matrices corresponding to the convolution kernel;
[0181] The multiple multidimensional matrices are linearly transformed to obtain the feature vector sequence;
[0182] The feature vector sequence and a preset vector are input into the encoder to obtain the target feature vector corresponding to the convolution kernel output by the encoder.
[0183] See Figure 19 As shown, in some optional embodiments, this application provides another video classification device 1900, which, based on the above-described video classification device 1800, further includes a classification module 1804 for:
[0184] After the fusion module 1803 obtains the classification vector representing the category of the video to be processed, it determines the video category corresponding to the classification vector of the video to be processed based on a preset correspondence; wherein, the preset correspondence includes the correspondence between the classification vector of the video and the video category.
[0185] Since this device is the same as the device in the method of this application embodiment, and the principle of the device in solving the problem is similar to that of the method, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described again.
[0186] like Figure 20 As shown, based on the same inventive concept, this application provides an electronic device 2000, including: a processor 2001 and a memory 2002;
[0187] Memory 2002 may be volatile memory, such as random-access memory (RAM); memory 2002 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 2002 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 2002 may be a combination of the above-mentioned memories.
[0188] The processor 2001 may include one or more central processing units (CPUs), graphics processing units (GPUs), or digital processing units, etc.
[0189] This application embodiment does not limit the specific connection medium between the memory 2002 and the processor 2001. This application embodiment... Figure 20 The memory 2002 and the processor 2001 are connected via a bus 2003, and the bus 2003 is in... Figure 20 The bus 2003, represented by thick lines, can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 20 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0190] The memory 2002 stores program code, which, when executed by the processor 2001, causes the processor 2001 to perform the following processes:
[0191] The video to be processed is subjected to convolution with different convolution kernels to extract features, thereby obtaining the feature vector sequence corresponding to each convolution kernel and the target feature vector, and the target feature vector is sorted based on the size of the convolution kernel;
[0192] For any target feature vector, the target feature vector is updated based on the target feature vectors adjacent to the target feature vector.
[0193] Feature fusion is performed on all feature vector sequences of the video to be processed and the updated target feature vector to obtain a classification vector representing the category of the video to be processed.
[0194] In some optional implementations, the processor 2001 specifically performs:
[0195] All target feature vectors of the video to be processed are input into the update model, and the update model updates the target feature vectors based on the target feature vectors adjacent to any target feature vector.
[0196] In some optional implementations, the processor 2001 specifically performs:
[0197] The average pooling operation is performed on the first feature vector and the second feature vector of the k-th level by the update model to obtain the average vector of the k-th level; where 1≤k≤K, K is the total number of levels updated by the update model; the first feature vector of the first level is any target feature vector, and the second feature vector of the first level is the adjacent target feature vector;
[0198] The average vector of the k-th level is calculated using a fully connected layer to obtain the adjustment vector of the k-th level;
[0199] The sum of the adjustment vector of the k-th level and the first feature vector of the k-th level is determined as the first feature vector of the (k+1)-th level; wherein, the first feature vector of the k-th level is the updated target feature vector.
[0200] In some optional implementations, the processor 2001 specifically performs:
[0201] The average vector of the k-th level is calculated using a first fully connected layer to obtain a first vector of the k-th level; and the average vector of the k-th level is calculated using a second fully connected layer to obtain a second vector of the k-th level, and the second vector of the k-th level is normalized to obtain the weight information of the k-th level.
[0202] Based on the first vector of the k-th level and the weight information of the k-th level, the adjustment vector of the k-th level is obtained.
[0203] In some optional implementations, the processor 2001 specifically performs:
[0204] The initial feature matrix is obtained by concatenating all feature vector sequences of the video to be processed and the updated target feature vector.
[0205] The initial feature matrix is input into the fusion model, and the fusion model performs feature fusion on the initial feature matrix to obtain a classification vector representing the category of the video to be processed.
[0206] In some optional implementations, the processor 2001 specifically performs:
[0207] The update matrix of the m-th level is determined by the fusion model based on the adjacency matrix of the m-th level, the feature matrix of the m-th level, and the adjustment parameters of the m-th level; where 1≤m≤M, M is the total number of levels in the fusion model iterative fusion; if 2≤m≤M, the adjacency matrix of the m-th level is determined based on the update matrix of the (m-1)-th level and the adjacency matrix of the (m-1)-th level, the adjacency matrix of the 1-th level is a preset matrix, and the feature matrix of the 1-th level is the initial feature matrix;
[0208] The product of the inverse of the update matrix at level m and the feature matrix at level m is determined as the feature matrix at level (m+1); where the feature matrix at level M is the classification vector.
[0209] In some optional implementations, the processor 2001 specifically performs:
[0210] For any convolution kernel, the video to be processed is convolved with the kernel to extract features, resulting in multiple multidimensional matrices corresponding to the convolution kernel;
[0211] The multiple multidimensional matrices are linearly transformed to obtain the feature vector sequence;
[0212] The feature vector sequence and a preset vector are input into the encoder to obtain the target feature vector corresponding to the convolution kernel output by the encoder.
[0213] In some optional implementations, after obtaining the classification vector characterizing the category of the video to be processed, the processor 2001 further performs:
[0214] Based on a preset correspondence, the video category corresponding to the classification vector of the video to be processed is determined; wherein, the preset correspondence includes the correspondence between the classification vector of the video and the video category.
[0215] Since the electronic device is the same electronic device that executes the method in the embodiments of this application, and the principle of the electronic device in solving the problem is similar to that of the method, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be described again.
[0216] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the video classification method described above. The readable storage medium can be a non-volatile readable storage medium.
[0217] The present application has been described above with reference to block diagrams and / or flowcharts illustrating methods, apparatus (systems), and / or computer program products according to embodiments of the present application. It should be understood that a block of a block diagram and / or flowchart, as well as combinations of blocks of block diagrams and / or flowcharts, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, and / or other programmable means to produce a machine, such that the instructions, executable via the computer processor and / or other programmable means, create methods for implementing the functions / actions specified in the blocks of the block diagrams and / or flowcharts.
[0218] Accordingly, this application can also be implemented using hardware and / or software (including firmware, resident software, microcode, etc.). Furthermore, this application can take the form of a computer program product on a computer-usable or computer-readable storage medium, having computer-usable or computer-readable program code implemented in the medium for use by or in conjunction with an instruction execution system. In the context of this application, a computer-usable or computer-readable medium can be any medium that can contain, store, communicate, transmit, or deliver a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0219] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0220] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A video classification method, characterized in that, The method includes: The video to be processed is subjected to convolution with different convolution kernels to extract features, thereby obtaining the feature vector sequence corresponding to each convolution kernel and the target feature vector, and the target feature vector is sorted based on the size of the convolution kernel; For any target feature vector, the target feature vector is updated based on the target feature vectors adjacent to the target feature vector. Feature fusion is performed on all feature vector sequences of the video to be processed and the updated target feature vector to obtain a classification vector representing the category of the video to be processed; For any target feature vector, the target feature vector is updated based on the target feature vectors adjacent to it, including: All target feature vectors of the video to be processed are input into the update model, and the update model updates the target feature vectors based on the target feature vectors adjacent to any target feature vector. The update model updates the target feature vector based on the target feature vectors adjacent to any target feature vector, including: The average pooling operation is performed on the first feature vector and the second feature vector of the k-th level by the update model to obtain the average vector of the k-th level; where 1≤k≤K, and K is the total number of levels updated by the update model; the first feature vector of the first level is any of the target feature vectors, and the second feature vector of the first level is the target feature vector adjacent to the target feature vector; The average vector of the k-th level is calculated using a fully connected layer to obtain the adjustment vector of the k-th level; The sum of the adjustment vector of the k-th level and the first feature vector of the k-th level is determined as the first feature vector of the (k+1)-th level; wherein, the first feature vector of the k-th level is the updated target feature vector.
2. The method according to claim 1, characterized in that, The average vector of the k-th layer is processed by a fully connected layer to obtain the adjustment vector of the k-th layer, including: The average vector of the k-th level is calculated using a first fully connected layer to obtain a first vector of the k-th level; and the average vector of the k-th level is calculated using a second fully connected layer to obtain a second vector of the k-th level, and the second vector of the k-th level is normalized to obtain the weight information of the k-th level. Based on the first vector of the k-th level and the weight information of the k-th level, the adjustment vector of the k-th level is obtained.
3. The method according to claim 1, characterized in that, Feature fusion is performed on all feature vector sequences of the video to be processed and the updated target feature vector to obtain a classification vector representing the category of the video to be processed, including: The initial feature matrix is obtained by concatenating all feature vector sequences of the video to be processed and the updated target feature vector. The initial feature matrix is input into the fusion model, and the fusion model performs feature fusion on the initial feature matrix to obtain a classification vector representing the category of the video to be processed.
4. The method according to claim 3, characterized in that, The initial feature matrix is fused using the fusion model, including: The update matrix of the m-th level is determined by the fusion model based on the adjacency matrix of the m-th level, the feature matrix of the m-th level, and the adjustment parameters of the m-th level; where 1≤m≤M, M is the total number of levels in the iterative fusion of the fusion model; if 2≤m≤M, the adjacency matrix of the m-th level is determined based on the update matrix of the (m-1)-th level and the adjacency matrix of the (m-1)-th level, the adjacency matrix of the 1-th level is a preset matrix, and the feature matrix of the 1-th level is the initial feature matrix; The product of the inverse of the update matrix at level m and the feature matrix at level m is determined as the feature matrix at level (m+1); where the feature matrix at level M is the classification vector.
5. The method according to claim 1, characterized in that, The video to be processed is subjected to convolution with different convolution kernels to extract features, resulting in a feature vector sequence corresponding to each convolution kernel and a target feature vector, including: For any convolution kernel, the video to be processed is convolved with the kernel to extract features, resulting in multiple multidimensional matrices corresponding to the convolution kernel; The multiple multidimensional matrices are linearly transformed to obtain the feature vector sequence; The feature vector sequence and the preset vector are input into the encoder to obtain the target feature vector corresponding to the convolution kernel output by the encoder.
6. The method according to any one of claims 1 to 5, characterized in that, After obtaining the classification vector representing the category of the video to be processed, the process further includes: Based on a preset correspondence, the video category corresponding to the classification vector of the video to be processed is determined; wherein, the preset correspondence includes the correspondence between the classification vector of the video and the video category.
7. A video classification device, characterized in that, The device includes: The feature extraction module is used to extract features from the video to be processed by convolution with different convolution kernels, obtain the feature vector sequence corresponding to each convolution kernel and the target feature vector, and sort the target feature vectors based on the size of the convolution kernel; The update module is used to update the target feature vector based on the target feature vectors adjacent to the target feature vector for any given target feature vector; The fusion module is used to perform feature fusion on all feature vector sequences of the video to be processed and the updated target feature vector to obtain a classification vector representing the category of the video to be processed. The update module is specifically used for: All target feature vectors of the video to be processed are input into the update model, and the update model updates the target feature vectors based on the target feature vectors adjacent to any target feature vector. The update module is specifically used for: The average pooling operation is performed on the first feature vector and the second feature vector of the k-th level by the update model to obtain the average vector of the k-th level; where 1≤k≤K, and K is the total number of levels updated by the update model; the first feature vector of the first level is any of the target feature vectors, and the second feature vector of the first level is the target feature vector adjacent to the target feature vector; The average vector of the k-th level is calculated using a fully connected layer to obtain the adjustment vector of the k-th level; The sum of the adjustment vector of the k-th level and the first feature vector of the k-th level is determined as the first feature vector of the (k+1)-th level; wherein, the first feature vector of the k-th level is the updated target feature vector.