A multi-label video classification method based on multi-instance learning and label relation graph

Abstract

The application relates to a multi-label video classification method based on multi-instance learning and a label relation graph, which comprises the following steps: acquiring the feature of each frame of a video to be classified; regarding the feature of each frame of the video as an instance, combining all the frame features into a bag-level feature to represent the video; mapping the bag-level feature of the video into a content-aware category representation of each video classification label; learning the relation of the video classification label, constructing a label relation graph, and adopting a label relation convolutional neural network to enhance the current video feature from the corresponding content-aware category representation to obtain the probability score of each classification label; comparing the probability score of each classification label with a set threshold to determine the final label of the video to be classified. The application can dynamically learn according to a current object, thereby correcting the frequency deviation in multi-label video prediction, and making the final classification label of the video more accurate.

Description

Technical Field

[0001] This application relates to the field of video understanding technology in artificial intelligence, and in particular to a multi-label video classification method based on multi-instance learning and label relationship graph. Background Technology

[0002] In recent years, with the booming development of online video sharing platforms, the automatic understanding and processing of massive amounts of video data has become an urgent problem to be solved. In practical video classification scenarios, each video often has multiple different types of tags, such as different objects, styles, scenes, actions, etc. Multi-tag video classification has important application value in user video recommendation and video content detection.

[0003] Existing video classification technologies are mainly divided into two categories: time-based modeling methods and frame-based methods. Time-based modeling methods primarily identify the temporal correlations between video frames to obtain motion information and thus classify events. This method is suitable for action recognition, using an end-to-end network model to map video clips to individual action labels. Frame-based methods, on the other hand, utilize static frames in the video, such as classification tasks that emphasize objects and scenes rather than actions. This approach typically requires capturing keyframes in the video.

[0004] Specifically, the frame-based methods, such as the ML-GCN algorithm, learn the relationships between labels through graph convolutional neural networks to obtain a definite classification model. Then, this classification model is used to directly classify the video to be classified. That is, a classifier that has learned the co-occurrence frequency of labels is used for feature matching, and the final classification of the video is determined based on the frequency of label co-occurrence.

[0005] However, this classification method has the following problems: The classification model of the ML-GCN algorithm only learns the prior knowledge of label co-occurrence for video classification. It cannot learn dynamically based on the current object. Therefore, global label relationships may lead to frequency bias. This bias will strengthen the strong associations and ignore the weak associations, making the final classification label of the video not particularly accurate. Summary of the Invention

[0006] To improve the accuracy of video classification, this application provides a multi-label video classification method based on multi-instance learning and label relationship graphs.

[0007] Firstly, the multi-label video classification method based on multi-instance learning and label relationship graph provided in this application adopts the following technical solution:

[0008] A multi-label video classification method based on multi-instance learning and label relationship graphs includes the following steps:

[0009] Obtain the features of each frame of the video to be classified;

[0010] Each frame feature of the video is treated as an instance, and all frame features are merged into packet-level features to represent the video.

[0011] The package-level features of the video are mapped to the content-aware category representation of each video classification label;

[0012] Learn the relationship between the video classification labels, construct a label relationship graph, and use a label relationship convolutional neural network to enhance the current video features from the corresponding content-aware category representation to obtain the probability score of each classification label;

[0013] The probability score of each classification label is compared with a set threshold to determine the final label of the video to be classified.

[0014] By adopting the above-mentioned technical solution, namely the multi-attention label relationship learning convolutional neural network (also known as MALL-CNN), the multi-instance learning framework is applied to video understanding. The video is regarded as a bag, and the frames in the video are instances of the bag. By modeling the label relationship, the current video label is highlighted in the global label relationship graph. This application does not need to use the prior knowledge of label co-occurrence for video classification, but learns dynamically based on the current object, thereby correcting the frequency bias in multi-label video prediction and making the final classification label of the video more accurate.

[0015] Preferably, the packet-level features of the video are obtained through the following methods:

[0016] Use an attention mechanism to assign attention weights to each video frame;

[0017] Based on the frame features of each video frame and the attention weight, a weighted summation process is performed to obtain the packet-level features of the video.

[0018] By adopting the above technical solution, an attention mechanism is used to assign attention weights to each video frame, and then the frames are grouped into packets, making the packet-level features of the final video more accurate. Consequently, the classification results based on the packet-level features of the video are also more accurate.

[0019] Preferably, the attention weight w is assigned to each video frame using the following formula. i :

[0020]

[0021] Where P, Q, and a are the parameters of the three fully connected layers. L is the number of frames captured, D is the number of channels; F i F jFor example, the features of the i-th and j-th frames. ⊙ represents element-wise multiplication, and Sigmoid(·) and tanh(·) are non-linear activation functions.

[0022] By adopting the above technical solution, this application uses an attention mechanism instead of simply averaging the frame-level features of the video to obtain the packet-level features. This allows for the retention of as much useful frame information as possible while reducing useless frame information, resulting in more accurate video label classification. In addition, two of the three fully connected layers used in this application are followed by a non-linear activation function, which enhances the generalizability of the neural network and makes it less prone to overfitting.

[0023] Two nonlinear activation functions, Sigmoid and tanh, are introduced between the three fully connected layers to enhance the nonlinear performance of the network and improve the generalization of the proposed method.

[0024] Preferably, Softmax is used to set the sum of attention weights to 1. This ensures that the final packet-level features obtained after weighted calculation for each frame are not too large.

[0025] Preferably, a feature mapping mechanism based on label attention converter (LAT) is used to map the packet-level features of the video into a content-aware category representation of the video classification label, specifically including the following steps:

[0026] The packet-level features of the video Convert to class-specific label vectors and package-level feature vectors Where H, W, and D refer to the three dimensions of the feature, representing the feature's length, width, and number of channels, respectively; C is the total number of categories; and l C It is the C-th vector in L, and D′≤D;

[0027] Based on the class-specific tag vectors and feature vectors, a content-aware category representation is obtained. Each category represents x c The calculation method is as follows:

[0028]

[0029] In the formula, and v′ i,j These are the specific label vector L and video feature vector V′ for category c, respectively. (i,j) The weight.

[0030] By adopting the above technical solutions, the packet-level features of the video are... Convert to class-specific label vectors and package-level feature vectors Then, based on the class-specific tag vector and package-level feature vector, generate the feature vector for each class (X = {X1, X2, X3, ... X...). C This allows for the conversion of package-level features into label classes, while reducing the dimensionality of channels, thereby reducing network complexity, improving network real-time performance, and making video label classification more efficient.

[0031] Preferably, the packet-level features The bag-level feature vector is obtained through a 1x1 convolution function. Specifically, the number of output channels of the convolution function is set to D', and the packet-level features of the input video are... That is, the bag-level feature vector is obtained. The packet-level feature vector Obtain class-specific label vectors using a classifier. The package-level feature vector Each with class-specific label vectors Multiply to obtain the content-aware category representation of the video.

[0032] By adopting the above technical solution, a bag-level feature vector is obtained through a 1*1 convolution function. Obtain class-specific label vectors using a classifier. The package-level feature vector Each with class-specific label vectors Multiply to obtain the content-aware category representation of the video. This allows the tag dimension to be incorporated with minimal computation, thus improving the efficiency of the entire video tag classification process.

[0033] Preferably, the step of learning the relationship between the video classification labels, constructing a label relationship graph, and using a label relationship convolutional neural network to enhance the current video features from the corresponding content-aware category representations specifically includes:

[0034] The content-aware category representations of multiple video category tags of the video to be classified are convolved to obtain the adjacency matrix and the graph structure of the corresponding adjacency matrix.

[0035] Based on the graph structure of the adjacency matrix, the relationships between different labels are obtained on each channel to obtain a new graph structure; the new graph structure is summed with the graph structure of the adjacency matrix, and the residual graph structure is obtained through residual connection, which is the final class representation vector.

[0036] By adopting the above technical solution, three label relationship graphs (i.e., the graph structure of the adjacency matrix, the new graph structure, and the residual graph structure) are constructed. This allows for dynamic learning based on the current object, correcting frequency bias in multi-label video prediction, and making the final classification label of the video more accurate.

[0037] Preferably, the step of performing convolution processing on the content-aware category representations of multiple video classification tags of the video to be classified to obtain an adjacency matrix specifically includes:

[0038] The content-aware category representation of the multiple video category tags of the video to be classified. As input nodes, construct a correlation matrix A to reflect the relationships between the features of each node. and the weight matrix W used to update X x ,

[0039] Node The adjacency matrix can be obtained by representing a single graph convolutional network layer as follows:

[0040] X u =δ(AXW x ),

[0041] Wherein, the linear transformation weight matrix W x Obtained during training; δ(·) is a non-linear activation function.

[0042] Preferably, the new graph structure is:

[0043]

[0044] in, e ji This indicates that there is an edge connecting the i-th and j-th nodes in the graph; ε is the set of edges, D is the number of channels, and r is the number of channels. ji h represents the node at position ij. t Denotes the basis function, k t Let C represent the coefficients, C represent the total number of categories, and n represent the univariate function f(·) transformed by Fourier transform into a set of basis functions (h1, h2, ..., hn). n ) and a set of coefficients (k1, k2, ..., k n When , represents the number of basis functions and their corresponding coefficients.

[0045] By adopting the above technical solution, namely, using an improved CNN module to aggregate neighborhood information in the global graph and focusing on the current video label X to enhance the characteristics of the current video label, it is possible to aggregate local inputs with different topological structures. This allows for dynamic learning based on the current object, correcting frequency biases in multi-label video prediction, and making the final video classification label more accurate.

[0046] Preferably, the probability score for each category label is obtained using the following method:

[0047] The predicted score for each classification label is calculated by global average pooling of class-specific label vectors.

[0048] Based on the final class representation vector, the prediction score is obtained.

[0049] The predicted score 1 and predicted score 2 are averaged to obtain the final probability score for each category label.

[0050] By adopting the above technical solution, the probability score of each classification label is obtained by averaging the prediction score 1 and prediction score 2, thereby making the probability of the final classification label more accurate and the label assignment result of the video more accurate.

[0051] Preferably, during model training, after obtaining the probability score for each classification label in the step, the method further includes: supervising the probability scores and training the entire label relationship convolutional neural network using the FL loss function or the ASL asymmetric loss function.

[0052] By adopting the above technical solutions and training the entire label relationship convolutional neural network with the FL loss function or ASL loss function, the weight values ​​of weak classification labels when calculating the loss can be set in advance, thereby further improving the accuracy of the final classification label of the video.

[0053] Secondly, this application provides a multi-label video classification system based on multi-instance learning and label relationship graphs, employing the following technical solution:

[0054] A multi-label video classification system based on multi-instance learning and label relationship graphs includes:

[0055] The frame feature acquisition module is used to acquire the features of each frame of the video to be classified.

[0056] The packet-level feature acquisition module is used to treat each frame feature of the video as an instance and merge all frame features into packet-level features to represent the video.

[0057] The data mapping module is used to map the package-level features of the video to the content-aware category representation of each video classification label; the label relationship graph construction module is used to learn the relationship between the video classification labels and construct a label relationship graph.

[0058] The probability score acquisition module for classification labels is used to enhance the current video features from the corresponding content-aware category representation using a label relation convolutional neural network to obtain the probability score for each classification label.

[0059] The tag determination module for videos to be classified is used to compare the probability score of each classification tag with a set threshold to determine the final tag of the video to be classified.

[0060] Thirdly, the electronic device provided in this application adopts the following technical solution:

[0061] An electronic device includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as in any of the methods described above.

[0062] Fourthly, the computer-readable storage medium provided in this application adopts the following technical solution:

[0063] A computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described in any of the methods described above.

[0064] In summary, this application includes at least one of the following beneficial technical effects:

[0065] 1. This application employs a multi-attention label relationship learning convolutional neural network (also known as MALL-CNN) to apply a multi-instance learning framework to video understanding. Videos are treated as packets, and frames in the video are instances of packets. By modeling label relationships, the current video label is highlighted in the global label relationship graph. This application does not require prior knowledge of label co-occurrence for video classification, but learns dynamically based on the current object, thereby correcting frequency bias in multi-label video prediction and making the final video classification label more accurate.

[0066] 2. This application constructs three label relationship graphs (i.e., the graph structure of the adjacency matrix, the new graph structure, and the residual graph structure), and then uses an improved CNN module to aggregate neighborhood information in the global graph and focus on the current video label X to enhance the characteristics of the current video label. It can aggregate local inputs with different topological structures, thereby dynamically learning according to the current object and correcting the frequency bias in multi-label video prediction. Attached Figure Description

[0067] Figure 1 This is a flowchart of a method according to one embodiment of this application.

[0068] Figure 2 This is a flowchart of a method for allocating attention weights in one embodiment of this application.

[0069] Figure 3This is a flowchart of a method for mapping the packet-level features of a video to a content-aware category representation of a video classification tag, according to one embodiment of this application.

[0070] Figure 4 This is a flowchart of a method for enhancing current video features from corresponding content-aware category representations using a label relational convolutional neural network, as described in one embodiment of this application.

[0071] Figure 5 This is a schematic diagram of an overall method according to one embodiment of this application.

[0072] Figure 6 This is a comparative illustration of the features of LR-CNN before (left) and after (right) enhancement in this application, visualized using t-sne. Detailed Implementation

[0073] The following is in conjunction with the appendix Figures 1-6 This application will be described in further detail.

[0074] This application discloses a multi-label video classification method based on multi-instance learning and label relationship graphs. (Refer to...) Figure 1 A multi-label video classification method based on multi-instance learning and label relationship graphs includes the following steps:

[0075] S1, Obtain the features of each frame of the video to be classified;

[0076] S2, treat each frame feature of the video as an instance, and merge all frame features into packet-level features to represent the video;

[0077] S3, map the package-level features of the video to the content-aware category representation of each video classification label;

[0078] S4. Learn the relationship between the video classification labels, construct a label relationship graph, and use a label relationship convolutional neural network to enhance the current video features from the corresponding content-aware category representation to obtain the probability score of each classification label.

[0079] S5, compare the probability score of each classification label with a set threshold to determine the final label of the video to be classified.

[0080] In this embodiment, step S2 can specifically obtain the packet-level features of the video through the following method:

[0081] S21, uses an attention mechanism to assign attention weights to each video frame;

[0082] S22, based on the frame features of each video frame and the attention weight, perform weighted summation to obtain the packet-level features of the video.

[0083] In other embodiments, a method that does not use attention may also be used, such as averaging the features of all frames to obtain the packet-level features of the video.

[0084] In this embodiment, each frame feature is considered as an instance F. i Each original video is a packet V O ={F1, F2, ..., F N}. V O If there is at least one instance F i If it belongs to the category of this label, then it can be considered V. O This tag is included. Therefore, the generic MIL pooling function used to aggregate instance features into package-level features (i.e., video-level features) can be described as follows:

[0085]

[0086] Use the attention-weighted average operator to aggregate frame-level features in the packet.

[0087] Step S21 specifically uses the following formula to assign attention weight w to each video frame. i :

[0088]

[0089] Wherein, P, Q, and a are the parameters of the three fully connected layers, obtained through training; that is, they are obtained through training the multi-attention label relationship learning convolutional neural network model (also known as MALL-CNN) from the steps of "obtaining the features of each frame of the video to be classified" to "comparing the probability score of each classification label with a set threshold to determine the final label of the video to be classified". L is the number of frames acquired; D is the number of channels; F i F j For example, the features of the i-th and j-th frames. ⊙ denotes element-wise multiplication; Sigmoid(·) and tanh(·) are non-linear activation functions. Gating mechanisms with Sigmoid(·) and tanh(·) are used to learn complex non-linear relationships between instances in an attention network.

[0090] In multi-label video classification tasks with thousands of labels c = {1, ..., C}, it is difficult to predict all labels using only a single attention network, as shown in Equation (1). Therefore, the single attention network is extended to a multi-attention network, and a set of exponential parameters {a} is used. c P c Q c Obtain the new weights W. The expansion function is defined as:

[0091]

[0092] Finally, the original features and the assigned weights are multiplied together to obtain the final bag-level features V.

[0093] To ensure that the final packet-level features obtained after weighted calculation for each frame are not too large, this embodiment uses Softmax to set the sum of attention weights to 1. The method for allocating attention weights is as follows: Figure 2 As shown.

[0094] In step S3 of this embodiment, a feature mapping mechanism based on label attention converter (LAT) can be used to map the packet-level features of the video to a content-aware category representation of the video classification label (i.e., corresponding to...). Figure 5 The tag conversion module (in the text) specifically includes the following steps, such as... Figure 3 As shown:

[0095] S31, the packet-level features of the video. Convert to class-specific label vectors and package-level feature vectors (This feature vector is a packet-level feature vector with the number of channels reduced using a convolution function (the number of channels is reduced from D to D')). Here, H, W, and D refer to the three dimensions of the feature, representing the length, width, and number of channels, respectively. The specific values ​​depend on the network used to extract the features. For example, ResNet50 will uniformly crop the input video frames into 256 or 320 pixel images, ultimately resulting in a feature map with a length and width of 7 or 8, and a final number of channels of 2048; C represents the total number of categories, l C It is the C-th vector in L, and D′≤D;

[0096] S32, Based on the class-specific tag vector and feature vector, obtain the content-aware category representation. Each category represents x c The calculation method is as follows:

[0097]

[0098] In the formula, and v′ i,j These are the specific label vector L and video feature vector V′ for category c, respectively. (i,j) The weights are obtained through training after initialization.

[0099] Since feature information is typically stored in the channels, dimensionality transformation is usually performed along the D dimension. The number of labels is a dimension not inherent in video data and needs to be introduced separately. Therefore, while preserving video information (i.e., dimension D), the number of labels (i.e., C) needs to be introduced. This requires creating two new features: V' preserves the number of channels (i.e., the semantic information of the video), and L introduces the number of labels C.

[0100] In this embodiment, the packet-level feature The bag-level feature vector is obtained through a 1x1 convolution function. Specifically, the number of output channels of the convolution function is set to D', and the packet-level features of the input video are... That is, the bag-level feature vector is obtained. The packet-level feature vector Obtain class-specific label vectors using a classifier (such as a multi-class binary classifier implemented with fully connected layers). The package-level feature vector Each with class-specific label vectors Multiply to obtain the content-aware category representation of the video.

[0101] In other embodiments, the packet-level features of the video can also be mapped to a content-aware category representation of the video classification label using the following method:

[0102] Features Perform global average pooling to obtain

[0103] The above A classifier implemented using fully connected layers performs a classification first, while in another branch, it... Repeat C (number of categories) times to obtain

[0104] The weights of the classifier and Element-wise product is used to obtain the content-aware category representation of the label.

[0105] In step S4 of this embodiment, the relationship between the video classification tags is learned, a tag relationship graph is constructed, and a tag relationship convolutional neural network is used to enhance the current video features from the corresponding content-aware category representation, such as... Figure 4 As shown, it specifically includes:

[0106] S41, perform convolution processing on the content-aware category representations of multiple video classification labels of the video to be classified to obtain the adjacency matrix and the graph structure of the corresponding adjacency matrix (i.e., graph X).

[0107] S42, Based on the graph structure of the adjacency matrix, obtain the relationship between different labels on each channel to obtain a new graph structure (i.e., graph Y);

[0108] S43, sum the new graph structure with the graph structure of the adjacency matrix, and obtain the residual graph structure (i.e., graph Z) through residual connection, which is the final class representation vector.

[0109] The method described in this application learns the label relationships using three graphs (X, Y, Z), highlighting the current video label X step by step.

[0110] In step S41, the convolutional processing of the content-aware category representations of multiple video classification labels of the video to be classified to obtain the adjacency matrix specifically includes: first, the relevance matrix A propagates relevant information among all nodes; then, each node receives all necessary information; finally, the value of X is linearly transformed by the weight matrix W. x To update; that is:

[0111] S411, represent the content-aware category of the multiple video classification tags of the video to be classified. As input nodes, construct a correlation matrix A to reflect the relationships between the features of each node. and the weight matrix W used to update X x , The correlation matrix A and weight matrix W x It is obtained through training, that is, through the entire "neural network" model trained from the above steps of "obtaining the features of each frame of the video to be classified" to "comparing the probability score of each classification label with a set threshold to determine the final label of the video to be classified".

[0112] S412, will node The adjacency matrix can be obtained by representing a single graph convolutional network layer as follows:

[0113] X u =δ(AXW x (5)

[0114] Wherein, the linear transformation weight matrix W x Obtained during training; δ(·) is a non-linear activation function, such as Relu(·) or Sigmoid(·).

[0115] In this embodiment, the new graph structure described in step S42 is:

[0116]

[0117] According to the theory of function approximation, any univariate function f(·) can be transformed by a Fourier transform into a function derived from a set of basis functions {h1, h2, ..., h...}. n} and a set of coefficients {k1, k2, ..., k n}express; e ji This indicates that there is an edge connecting the i-th and j-th nodes in the graph (i.e., the two labels are related); ε is the set of edges, D is the number of channels, and r is the number of channels. ji h represents the node at position ij. t Denotes the basis function, k t Let C represent the coefficients, C represent the total number of categories, and n represent the univariate function f(·) transformed by Fourier transform into a function derived from a set of basis functions {h1, h2, ..., hn}. n} and a set of coefficients {k1, k2, ..., k n} represents the number of basis functions and their corresponding coefficients.

[0118] This application constructs a label relationship graph by learning the relationship between the video classification labels, and uses a label relationship convolutional neural network to process the corresponding content-aware category representation, making the feature distribution of the same category more clustered and the feature distinction of different categories more obvious, thus enhancing the current video features.

[0119] In other embodiments, other graph convolution methods can also be used to obtain the new graph structure described in step S42, such as the GCN algorithm.

[0120] In step S4 of this embodiment, the probability score of each classification label can be obtained through the following method, such as... Figures 3-5 As shown:

[0121] The prediction score for each class label is calculated by global average pooling of the class-specific label vectors (i.e., the corresponding...). Figure 5 L in a Specifically, global average pooling can be used to... Convert to That is, the predicted score 1);

[0122] Based on the final class representation vector, (a multi-class binary classifier implemented using fully connected layers) the predicted score 2 is obtained (i.e., the corresponding...). Figure 5 L in r );

[0123] The predicted score 1 and predicted score 2 are averaged to obtain the final probability score for each category label.

[0124] Figure 5The values ​​of predicted score 1 and predicted score 2 described above are merely examples; their actual values ​​are obtained using the method described above.

[0125] In other embodiments, a 1*1 convolutional function, a fully connected layer, or an MLP structure can also be used to obtain the probability score for each classification label.

[0126] In this embodiment, during model training, after obtaining the probability score for each classification label in the step, the method further includes: supervising the probability scores and training the entire label relationship convolutional neural network using the FL loss function or the ASL asymmetric loss function.

[0127] The FL loss function mentioned above is:

[0128] The aforementioned ASL asymmetric loss function, namely

[0129] In other embodiments, the following loss functions may also be used for training, such as:

[0130] Binary Cross Entropy (BCE) loss function: or

[0131] Where σ(·) is the Sigmoid(·) function, s is the class score predicted by the model, y is the true label of the video data, c is a certain class, and L(y, s) represents the loss function.

[0132] In this application, network parameters are optimized through a loss function, and the specific optimization process can be automatically completed using a deep learning framework.

[0133] The method described in this application learns the label relationships using three graphs (X, Y, Z), gradually revealing the working principle of the current video labels, such as... Figure 4 , Figure 5 As shown:

[0134] This application introduces a label-relationship convolutional neural network, namely... Figure 5 The LR-CNN in the example enhances the current video features X from the content-aware category representation. Specifically, LR-CNN progressively enhances the current video features through three steps, corresponding to three graph structures (X, Y, Z).

[0135] First, construct graph X. Graph X is the initial graph X used to model the label relationships of content-aware category representations. Given a set of features... As input nodes, construct a correlation matrix. This reflects the relationships between the features of each node. A weight matrix is ​​also constructed. To update X. Formally, a node A single graph convolutional network layer can be represented as:

[0136] X u =δ(AXW x ),

[0137] Among them, W x The value of X can be learned during training. δ(·) is a non-linear activation function, such as ReLU(·) or Sigmoid(·). In this process, firstly, the correlation matrix A propagates relevant information among all nodes, then each node receives all necessary information, and finally, the value of X is linearly transformed through the weight matrix W. x Update.

[0138] Next, we construct graph Y. Graph Y is an enhancement of graph X. The transformation from X to Y can be accomplished using an improved convolutional neural network module (i.e., an improved CNN module). Given an input... and a filter The output at vertex i is:

[0139]

[0140] Formula (7) is the traditional CNN module, which can only process Euclidean data.

[0141] Where, discrete ω j-i+m It can be fitted by a univariate continuous function f(·). f(·) always satisfies f(j-i+m)=ω j-i+m Therefore, the above formula can be equivalently rewritten as follows:

[0142]

[0143] Formula (8) can handle non-Euclidean data by fitting a univariate function.

[0144] Where f(·) is a function used to process discrete data ω j-i+m The filter function used for sampling. Relationship It can be derived from a univariate continuous function Fitting. This represents the relationship between a vertex and its neighboring vertices. This indicates that the (i-m+r)th vertex is the rth neighbor of the i-th vertex. The local structure of vertex i is represented as:

[0145]

[0146] Where m is a constant value. e ij∈ε means that the j-th vertex is a neighbor of the i-th vertex, r ji yes The element at (i,j). Therefore, the improved convolution can be defined as:

[0147]

[0148] in, Follow Variation. According to the above formula, convolution can aggregate local inputs with different topological structures.

[0149] According to the theory of function approximation, any univariate function f(·) can be transformed by Fourier transform into a function derived from a set of basis functions {h1, h2, ..., h...}. n} and a set of coefficients {k1, k2, ..., k n} represents. When x = r ji ,

[0150]

[0151] Several mathematical methods exist for local aggregation, such as Chebyshev polynomials, Legendre polynomials, Laguerre polynomials, and Hermitian polynomials. Here, Chebyshev polynomials are chosen as the local aggregation method:

[0152] T k (L)=2LT k-1 (L)-T k-2 (L), T0(L)=I, T1=L, (12)

[0153] When T k (L) is replaced with h t (r ji When ), it is rewritten as:

[0154] h t (x)=2xh k-1 (x)-h k-2 (x), h1(x)=1, h2(x)=x, (13)

[0155] In summary:

[0156]

[0157] The implementation process of this module is as follows: Figure 5 As shown. For example, the input X has D = 1024 channels (preferred setting, but other values ​​are also possible) and C = 3862 vertices (i.e., the number of categories, which can be determined according to the number of categories in the actual dataset used), and the output Y has only one channel. For each vertex i, its local structure is represented as follows: X is shared for each channel. Then, the local inputs {f1(·), f2(·), ..., f...} are aggregated using a set of convolutional kernels. n Finally, {f1(·), f2(·), ..., f n The (·)} are added together. Therefore, the formula is as follows:

[0158]

[0159] Finally, graph Z is constructed. Graph Z focuses on the feature labels of the current video on graph Y. Figure 5 In the context, nodes {z1, z2, ..., z} n} represents the final predicted label. Adding graph X and graph Y together generates the final graph z. This method highlights the current video features and reduces redundant relationships.

[0160] To verify the technical effectiveness of this application, the inventors also conducted the following comparative experiments:

[0161] Dataset

[0162] Released in June 2019, YouTube-8M features segment-level annotations. It is the world's largest semantic recognition dataset, comprising 4,716 classes, over 6.1 million videos, and more than 3,800 visual entities. The dataset is divided into three subsets: training, validation, and test sets. YouTube-8M includes three datasets: frame-level feature dataset, segment-level feature dataset, and video-level feature dataset. Furthermore, this dataset contains features from both visual and audio modalities.

[0163] Evaluation indicators

[0164] Four commonly used evaluation metrics for multi-label classification were selected: Global Average Precision (GAP, the global average precision across all labels), Mean Average Precision (MAP, the average precision per label), Precision at Equal Recall Rate (PERR, the precision at the same recall rate), and Hit Rate (Hit@1, the frequency with which the true label appears first in the predicted label sequence). For all metrics, a higher value indicates better performance in video classification.

[0165] Implementation details

[0166] On an Nvidia GeForce RTX 3090 GPU, SGD was selected as the optimizer, with momentum set to 0.9 and an initial learning rate of 0.0001. Every 10... 5The learning rate is decreased by 0.8 times. The batch size is set to 40. In MIVL, L=1024 and D=32 are set. In LAT, the number of channels for V′ is set to 1024. In LR-CNN, the number of channels for X is set to 1024, and an 80th-order Chebyshev polynomial is chosen as the approximation function for the convolution kernel.

[0167] Comparison method:

[0168] The first existing method is NeXtVLAD, which was proposed by Rongcheng Lin et al. in the paper "Nextvlad: An efficient neural network to aggregate frame-level features for large-scale video classification".

[0169] The second existing method is Axon AI, a method proposed by Joonseok Lee et al. in the paper "The 2nd youtube-8m large-scale video understanding challenge".

[0170] The third existing method is CMS-SNet, a method proposed by Renchun You et al. in the paper "Cross-modality attention with semantic graph embedding for multi-label classification".

[0171] The video tag classification results using the method of this application and the aforementioned existing methods are as follows:

[0172]

[0173] The above experiments demonstrate that, compared to traditional methods, the method proposed in this application significantly reduces the number of parameters while achieving significant improvements in all four commonly used metrics for multi-label classification, thus ensuring the speed and accuracy of multi-label video classification.

[0174] In addition, to verify the effect of the improved CNN module used in this application in generating the new graph structure Y, the effect of the improved CNN module used in this application in generating the new graph structure Y is compared with the effect of other existing GCNN techniques in generating the new graph structure Y.

[0175] Comparison method:

[0176] The first existing method is MoNets (Hybrid Model Networks), proposed by Federico Monti et al. in the paper "Geometric deep learning on graphs and manifolds using mixture model cnns".

[0177] Existing Method 2: ChebNet (a spectral network based on Chebyshev), by The method proposed by Defferrard et al. in the paper "Convolutional neural networks on graphs with fast localized spectral filtering"

[0178] The third existing method is GCN (Graph Convolutional Neural Network), a method proposed by Thomas N. Kipf and Max Welling in the paper "Semi-supervised classification with graph convolutional networks".

[0179] The comparison results are shown in the table below. As can be seen from the table, compared with other GCNN algorithms, the improved CNN module of this application generates a new graph structure Y, resulting in higher average precision for each label and the average precision for all labels in the final video label classification. Moreover, it has a significant advantage in terms of computational complexity. Compared with the GCN algorithm, the computational cost (i.e., FLOPS) of the improved CNN module of this application is reduced by half, thus achieving better performance.

[0180]

[0181] like Figure 6 As shown, t-sne is used to visualize the features before (left) and after (right) enhancement of LR-CNN in this application. It can be found that: in particular, the improved CNN module used in this application generates a new graph structure Y to enhance the current video features. The feature distribution of the same category tends to cluster, which makes the feature distribution of different categories more distinct and is conducive to improving the classification effect.

[0182] This embodiment also discloses a multi-label video classification system based on multi-instance learning and label relationship graphs, including:

[0183] The frame feature acquisition module is used to acquire the features of each frame of the video to be classified.

[0184] The packet-level feature acquisition module is used to treat each frame feature of the video as an instance and merge all frame features into packet-level features to represent the video.

[0185] The data mapping module is used to map the package-level features of the video to the content-aware category representation of each video classification label; the label relationship graph construction module is used to learn the relationship between the video classification labels and construct a label relationship graph.

[0186] The probability score acquisition module for classification labels is used to enhance the current video features from the corresponding content-aware category representation using a label relation convolutional neural network to obtain the probability score for each classification label.

[0187] The tag determination module for videos to be classified is used to compare the probability score of each classification tag with a set threshold to determine the final tag of the video to be classified.

[0188] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0189] This embodiment also discloses an electronic device. An electronic device includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as in any of the methods described above.

[0190] The electronic device can be a desktop computer, a laptop computer, or a cloud server, and includes, but is not limited to, a processor and a memory. For example, the electronic device may also include input / output devices, network access devices, and buses.

[0191] The processor in this application may include one or more processing cores. The processor executes instructions, programs, code sets, or instruction sets stored in memory, and calls data stored in memory to perform various functions and process data. The processor may be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic devices used to implement the above-described processor functions may also be other types, and the embodiments of this application do not specifically limit this.

[0192] The memory can be an internal storage unit of an electronic device, such as a hard disk or RAM, or an external storage device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD), or flash memory card (FC) equipped on the electronic device. Furthermore, the memory can be a combination of an internal storage unit and an external storage device of the electronic device. The memory is used to store computer programs and other programs and data required by the electronic device. The memory can also be used to temporarily store data that has been output or will be output. This application does not limit this.

[0193] This embodiment also discloses a computer-readable storage medium. A computer-readable storage medium stores a computer program that can be loaded by a processor and executed as in any of the methods described above.

[0194] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0195] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the methods and principles of this application should be covered within the scope of protection of this application.

Claims

1. A multi-label video classification method based on multi-instance learning and label relation graph, characterized in that, The method comprises the following steps: obtaining the feature of each frame of the video to be classified; regarding the feature of each frame of the video as an instance, and combining all the frame features into a package-level feature to represent the video; mapping the package-level feature of the video into a content-aware class representation of each video classification label; learning the relationship of the video classification labels, constructing a label relationship graph, and using a label relationship convolutional neural network to enhance the current video feature from the corresponding content-aware class representation to obtain a probability score of each classification label; comparing the probability score of each classification label with a set threshold to determine the final label of the video to be classified; wherein the package-level feature of the video is mapped into a content-aware class representation of the video classification label by using a feature mapping mechanism LAT based on a label attention converter, and the specific steps include: converting the packet-level features of the video into class-specific label vectors and packet-level feature vectors where H, W, D denote the three dimensions of the features, respectively the length, width and number of channels, C is the total number of classes, l C is the Cth vector in L, D' ≤ D; Based on the class-specific label vector and feature vector, a content-aware class representation is obtained where each class representation x c is computed as follows: wherein and v' are the specific label vector L and video feature vector V' of class c, respectively i，j and v' are the specific label vector L and video feature vector V' of class c, respectively (i，j) and v' are the specific label vector L and video feature vector V' of class c, respectively the learning of the relationship of the video classification labels, the construction of the label relationship graph, and the use of the label relationship convolutional neural network to enhance the current video feature from the corresponding content-aware class representation specifically include: convolving the content-aware class representations of multiple video classification labels of the video to be classified to obtain an adjacency matrix and a graph structure corresponding to the adjacency matrix; obtaining the relationship between different labels on each channel according to the graph structure of the adjacency matrix to obtain a new graph structure; summing the new graph structure and the graph structure of the adjacency matrix to obtain a residual graph structure through a residual connection, i.e., a final class representation vector; wherein the convolution of the content-aware class representations of multiple video classification labels of the video to be classified to obtain an adjacency matrix specifically includes: Content-aware class representation of a plurality of video classification labels of the video to be classified As an input node, a correlation matrix A for reflecting a relationship between features of each node is constructed, and a weight matrix W for updating X x , The nodes is represented by a single graph convolutional network layer as follows, resulting in an adjacency matrix: X u = δ(AXW x ), where the linear transformation weight matrix W x obtained in training; δ(·) is a nonlinear activation function; the new graph structure, i.e.: in, eji indicates that there is an edge connecting the i-th and j-th nodes in the graph; ε is the set of edges, D is the number of channels, and r ji h represents the node at position ij. t Denotes the basis function, k t Let C represent the coefficients, C represent the total number of categories, and n represent the univariate function f(·) transformed by Fourier transform into a function derived from a set of basis functions {h1h2, ..., h...}. n } and a set of coefficients {k1, k2, ..., k n } represents the number of basis functions and their corresponding coefficients.

2. The multi-label video classification method based on multi-instance learning and label relation graph according to claim 1, characterized in that, The package-level feature of the video is obtained by the following method: using an attention mechanism to assign an attention weight to each video frame; performing weighted summation processing according to the frame feature of each video frame and the attention weight to obtain the package-level feature of the video.

3. The multi-label video classification method based on multi-instance learning and label relation graph according to claim 2, characterized in that, In particular, the following formula is used to assign an attention weight w to each video frame i : where P, Q, a are parameters of three fully connected layers, L is the number of collected frames, D is the number of channels; F i , F j is an instance, i.e., the i-th, j-th frame feature, ⊙ represents element-wise multiplication, Sigmoid(·), tanh(·) are nonlinear activation functions.

4. The multi-label video classification method based on multi-instance learning and label relation graph according to claim 1, characterized in that, During model training, after obtaining the probability score of each classification label in step, the method further comprises: supervising the probability score and training the entire label relationship convolutional neural network by using a FL loss function or an ASL asymmetric loss function.

5. A multi-label video classification system based on multi-instance learning and label relation graph, characterized in that, It comprises: a frame feature acquisition module for obtaining the feature of each frame of the video to be classified; a package-level feature acquisition module for regarding the feature of each frame of the video as an instance, and combining all the frame features into a package-level feature to represent the video; a data mapping module for mapping the package-level feature of the video into a content-aware class representation of each video classification label; a label relationship graph construction module for learning the relationship of the video classification labels and constructing a label relationship graph; a classification label probability score acquisition module for using a label relationship convolutional neural network to enhance the current video feature from the corresponding content-aware class representation to obtain a probability score of each classification label; a label determination module for the video to be classified for comparing the probability score of each classification label with a set threshold to determine the final label of the video to be classified. The data mapping module adopts a feature mapping mechanism LAT based on a label attention converter to map the packet-level features of the video into content-aware class representation of video classification labels. converting the packet-level features of the video into class-specific label vectors and packet-level feature vectors where H, W, D refer to the three dimensions of the features, respectively referring to the length, width and number of channels of the features, C is the total number of classes, l C is the Cth vector in L, D'≤D; Based on the class-specific label vector and feature vector, a content-aware class representation is obtained where each class representation x c is computed as follows: wherein and v' are the specific label vector L and video feature vector V' of class c, respectively i，j and v' are the specific label vector L and video feature vector V' of class c, respectively (i，j) and v' are the specific label vector L and video feature vector V' of class c, respectively The classification label probability score acquisition module, The content-aware class representation of the multiple video classification labels of the video to be classified is subjected to convolution processing to obtain an adjacency matrix and a graph structure corresponding to the adjacency matrix. According to the graph structure of the adjacency matrix, the relationship between different labels is obtained on each channel to obtain a new graph structure. The new graph structure and the graph structure of the adjacency matrix are subjected to summation processing to obtain a residual graph structure, i.e., a final class representation vector, through a residual connection. The classification label probability score acquisition module obtains the adjacency matrix in the following manner: Content-aware class representation of a plurality of video classification labels of the video to be classified As an input node, a correlation matrix A for reflecting a relationship between features of each node is constructed, and a weight matrix W for updating X x , The nodes is represented by a single graph convolutional network layer as follows: X u = δ(AXW x ), where the linear transformation weight matrix W x obtained in training; δ(·) is a nonlinear activation function; The new graph structure is as follows: wherein, eji indicates that there is an edge connecting the i-th and j-th nodes in the graph; ε is the set of edges, D is the number of channels, r ji represents the nodes at ij, h t represents the basis functions, k t represents the coefficients, C represents the total number of classes, and n represents the number of basis functions {hi, h2,..., hn} and coefficients {ki, k2,..., kn} that the unary function f(·) is converted into by Fourier transform. n} and a set of coefficients {ki, k2,..., kn} when the unary function f(·) is converted into by Fourier transform. n} and a set of coefficients {ki, k2,..., kn} when the unary function f(·) is converted into by Fourier transform.

6. An electronic device, comprising: The device comprises a memory and a processor, and the memory stores a computer program capable of being loaded and executed by the processor and implementing any one of the methods in claims 1 to 4.

7. A computer readable storage medium characterized in that, The device stores a computer program capable of being loaded and executed by the processor and implementing any one of the methods in claims 1 to 4.

Images