A multi-label behavior recognition method and device based on binary tree structure

Abstract

The present invention discloses a multi-label behavior recognition method and device based on a binary tree structure, including: S1, extracting the parameter vector of each behavior category output by the first classifier in the trained first multi-label behavior recognition network; S2, calculating the similarity between each behavior category based on the parameter vector and sorting them from high to low according to the similarity, and recursively combining each behavior category in turn without repetition to construct a binary tree; S3, constructing a second multi-label behavior recognition network based on the feature extraction network, the second classifier and the binary tree; S4, using the trained second multi-label behavior recognition network to perform behavior recognition on the video to be tested. By adopting the multi-label behavior classification technology based on the binary tree, the similarity between behaviors of different categories can be decoupled, especially for behaviors that are highly similar visually but difficult to distinguish, thereby effectively improving the recognition accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and more specifically, relates to a multi-label behavior recognition method and device based on a binary tree structure. Background Art

[0002] In the era of Internet big data, massive amounts of videos are generated every moment. How to make computers accurately understand the content of these videos and identify the labels of the videos can not only quickly realize video classification and improve the efficiency of information dissemination, but also assist humans in more accurate abnormal monitoring in monitoring scenarios. The former aims to automatically analyze the massive original videos generated on the Internet and generate labels for them to facilitate the retrieval of semantically related information; the latter is mainly used to accurately identify abnormal behaviors such as falling, fighting, and drowning, which can reduce the probability of casualties. The analysis and understanding of video content, especially multi-label tasks, has great practical application value.

[0003] Most existing research focuses on single-label classification of videos. In this task, it is assumed that each video contains only one behavior, and the network only needs to give each video a behavior label. However, in reality, videos often contain multiple behaviors. Therefore, in order to fully analyze the video, multi-label classification of videos is an inevitable trend. In multi-label classification of videos, there is often a high degree of visual similarity between different types of behaviors. For example, the two behaviors of opening and closing the refrigerator door. To accurately distinguish these similar behaviors, the requirements of the last fully connected layer of the network are extremely high, so the two are often indistinguishable.

[0004] The tasks for multi-label classification proposed by scholars generally focus on exploring the symbiotic dependency between multiple labels, such as using graph convolution to model the semantic relationship between labels, such as the connection between different objects such as food and plates, people and backpacks, to achieve inspiration for possible labels and suppression of impossible labels. However, in the face of the high visual similarity between different labels, there is still little research on how to improve the discrimination ability of classifiers. Existing methods are difficult to solve the problem of fine-grained distinction between highly visually similar behaviors, and their general robustness is not good enough. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a multi-label behavior recognition method and device based on a binary tree structure, thereby solving the problem of difficulty in distinguishing between highly visually similar behaviors in existing multi-label behavior classification tasks.

[0006] To achieve the above object, according to a first aspect of the present invention, a multi-label behavior recognition method based on a binary tree structure is provided, comprising:

[0007] S1, extracting parameter vectors of each behavior category output by a first classifier in a trained first multi-label behavior recognition network; wherein the first multi-label behavior recognition network also includes a feature extraction network;

[0008] S2, calculating the similarity between each behavior category based on the parameter vector and sorting them from high to low according to the similarity, and recursively combining each behavior category in pairs without duplication in order to construct a binary tree;

[0009] S3, constructing a second multi-label behavior recognition network based on the feature extraction network, the second classifier and the binary tree; the output dimension of the second classifier is the same as the number of nodes of the binary tree;

[0010] S4, using the trained second multi-label behavior recognition network to perform behavior recognition on the video to be tested.

[0011] Preferably, step S2 comprises:

[0012] The similarities between the behavior categories in the first classifier are sorted from high to low, and non-repetitive recursive combinations are performed in pairs in turn to finally obtain a combination, which is the root node of the binary tree.

[0013] Preferably, the first classifier and the second classifier are any one of a fully connected layer, a multi-layer perceptron or a SVM.

[0014] Preferably, the number of layers L of the binary tree and the number C of behavior categories of the data set satisfy the following relationship:

[0015]

[0016] Preferably, the method for determining the similarity between the behavior categories is:

[0017] Calculate the Cosine similarity between the C D-dimensional feature parameters in the first classifier;

[0018] Or, the C D-dimensional feature parameters in the first classifier are normalized, and then the similarities between the C feature parameters are calculated.

[0019] Preferably, the feature extraction network is a deep learning feature extraction network or an extraction network based on manual feature operators.

[0020] Preferably, only the second classifier and the binary tree in the second multi-label behavior recognition network are trained.

[0021] According to a second aspect of the present invention, a multi-label behavior recognition device based on a binary tree structure is provided, comprising:

[0022] An extraction module, used to extract the parameter vector of each behavior category output by the first classifier in the trained first multi-label behavior recognition network; wherein the first multi-label behavior recognition network also includes a feature extraction network;

[0023] A binary tree construction module, used to sort the similarities between the behavior categories in the first classifier from high to low, and perform non-repeating recursive combinations of two by two in sequence to construct a binary tree;

[0024] A recognition network construction module, used to construct a second multi-label behavior recognition network based on the feature extraction network, the second classifier and the binary tree; the output dimension of the second classifier is the same as the number of nodes of the binary tree;

[0025] The recognition module is used to perform behavior recognition on the video to be tested using the trained second multi-label behavior recognition network.

[0026] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0027] 1. The multi-label behavior recognition method based on the binary tree structure provided by the present invention adopts the multi-label behavior classification technology and uses the trained classifier to measure the similarity between multiple behavior categories. The behavior tree constructed according to the similarity intuitively reflects the similarity between any two behavior categories from the model structure. This modeling method can make full use of the behaviors learned by the first classifier itself and the similarities between behaviors, reduce the difficulty of learning the second classifier, and thus improve the classification accuracy.

[0028] 2. The multi-label behavior recognition method based on a binary tree structure provided by the present invention adopts a multi-label behavior classification technology based on a binary tree, which can realize the decoupling of the similarities between behaviors of different categories. In particular, for behaviors that are highly similar visually but difficult to distinguish, the behavior tree technology can effectively improve the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 One of the flow charts of a multi-label behavior recognition method based on a binary tree structure provided by the present invention;

[0030] Figure 2 A schematic diagram of similarity measurement between combinations provided by the present invention;

[0031] Figure 3 A corresponding diagram of the binary tree and the second classifier provided by the present invention;

[0032] Figure 4 The second flowchart of a multi-label behavior recognition method based on a binary tree structure provided by the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0034] The embodiment of the present invention provides a multi-label behavior recognition method based on a binary tree structure. Figure 1 As shown, including:

[0035] S1, extracting the parameter vector of each behavior category output by the first classifier in the trained first multi-label behavior recognition network; wherein the first multi-label behavior recognition network also includes a feature extraction network.

[0036] Specifically, an arbitrary feature extraction network + a first classifier is used to train a multi-label behavior classification network, and a parameter vector corresponding to each category in the first classifier is taken out, and these parameter vectors are used to measure the similarity between all categories.

[0037] The output dimension of the first classifier is equal to the number C of behavior categories in the data set, that is, the output of the first classifier is a C-dimensional vector.

[0038] The behavior may be human behavior, animal behavior, machine behavior, etc.

[0039] Preferably, the feature extraction network can be any traditional deep learning feature extraction network or an extraction network based on manual feature operators.

[0040] S2, based on the parameter vector, calculate the similarity between each behavior category and sort them from high to low according to the similarity, and recursively combine each behavior category in pairs without duplication in order to construct a binary tree.

[0041] Specifically, the similarities between each behavior category are sorted from high to low, and non-repetitive pairwise recursive grouping is performed according to the similarities between different categories to form a behavior tree.

[0042] S3, constructing a second multi-label behavior recognition network based on the feature extraction network, the second classifier and the binary tree; the output dimension of the second classifier is the same as the number of nodes of the binary tree.

[0043] Preferably, the first classifier and the second classifier are any one of a fully connected layer, a multi-layer perceptron or a SVM. The first classifier may be the same as or different from the second classifier.

[0044] S4, using the trained second multi-label behavior recognition network to perform behavior recognition on the video to be tested.

[0045] Specifically, the second classifier and the constructed behavior tree are used to replace the first classifier, a second multi-label behavior recognition network is constructed, and the second multi-label behavior recognition network is trained. After the training is completed, the behavior tree is directly used for reasoning testing.

[0046] Furthermore, the number of layers L of the binary tree and the number of behavior categories C of the data set satisfy the following relationship:

[0047]

[0048] Specifically, for a data set with C classes, in order to ensure that the C classes can be completely distributed on all leaf nodes, it is necessary to design an L-layer behavior tree, which is a complete binary tree. It can be understood that the data set includes a training data set and a test data set.

[0049] A behavior tree of level L has 2 L -1 node, so in the process of implementing the behavior tree, you need to use a completely independent classifier with an output dimension of 2 L The second classifier of -1 corresponds to each node in the binary tree one by one, that is, the 2 output by the second classifier L -1-dimensional vector and binary tree 2 L -1 nodes correspond one to one, and each element in the second classifier output vector uniquely corresponds to a node in the behavior tree. Figure 2 The figure shows the correspondence between the nodes in the behavior tree and the elements in the classifier output results. Taking the second classifier as a fully connected layer as an example, C=4, L=3, that is, the number of behavior categories is 4, the number of layers of the behavior tree is 3, and the number of nodes in the behavior tree is 7. The position of the elements in the second classifier output results and the number of the nodes in the behavior tree conform to the rules of a complete binary tree.

[0050] Since C categories are placed on the four leaf nodes A, B, C, and D of the constructed binary tree, in order to obtain the probability prediction corresponding to each category, the prediction result of the leaf node corresponding to each category is multiplied by the prediction results of all the parent nodes of the leaf node as the prediction probability of the leaf node corresponding to each category; Figure 2 Taking the behavior tree in as an example, the predicted probability of node B is:

[0051] p B =σ(f4)σ(f1)σ(f0)

[0052] where f jrepresents the output of the jth element in the classifier, and σ represents the Sigmoid function. Where f4 is a leaf node, and f1 and f0 are all parent nodes of the leaf node f4.

[0053] Furthermore, the step S2 comprises:

[0054] The similarities between the behavior categories in the first classifier are sorted from high to low, and non-repetitive recursive combinations are performed in pairs in turn to finally obtain a combination, which is the root node of the binary tree.

[0055] Specifically, we perform non-repetitive recursive groupings based on the similarities between different categories to form a behavior tree. The specific steps of the combination are as follows:

[0056] (1) Sort the similarities between the behavior categories in the first classifier from high to low, and prioritize the behavior categories with high similarity to form a group, so as to ensure that the two behavior categories grouped together have the highest similarity; for example, if the similarity between behavior 1 and behavior 2 is 0.9, the similarity between behavior 1 and behavior 3 is 0.8, and the similarity between behavior 1 and behavior 4 is 0.7, then behavior 1 and behavior 2 are prioritized to be combined into a group. During the combination process, if the behavior in the current combination has been selected by other behavior combinations with higher similarity, the current combination will be abandoned. This rule allows all behavior categories to be combined in pairs without duplication. For C categories, we can get A combination of Indicates rounding up.

[0057] (2) In order to form a binary tree structure, the combinations need to be further grouped. In order to measure the similarity between the combinations, such as Figure 2 As shown in Figure 2, this method uses the similarity between behaviors in a combination to estimate the similarity of the entire combination. Figure 1 The similarity S between combination A and combination B ab Written as:

[0058]

[0059] Where S ij Represents the similarity between the i-th type of behavior and the j-th type of behavior.

[0060] (3) After step (2), the behavior categories are recursively combined in pairs, and a tree relationship is established between the behavior categories. For easy observation, it can be drawn as follows: Figure 3 The behavior tree shown.

[0061] Furthermore, the method for determining the similarity between the behavior categories is:

[0062] Calculate the Cosine similarity between the C D-dimensional feature parameters in the first classifier;

[0063] Or, the C D-dimensional feature parameters in the first classifier are normalized, and then the similarities between the C feature parameters are calculated.

[0064] Specifically, measure the similarity between all categories, including:

[0065] Assume that the parameter shape in the fully connected layer is D×C, where D directly represents the feature dimension of each category, C represents the number of categories to be classified in the data set, and the similarity measure between these C categories is a C×C similarity matrix. The similarity value in this matrix is ​​calculated as follows:

[0066] Directly calculate the Cosine similarity between C D-dimensional feature parameters;

[0067] Alternatively, the C D-dimensional feature parameters are normalized to a unit vector, and then the similarities between the C feature parameters are calculated.

[0068] Furthermore, only the second classifier and the binary tree in the second multi-label behavior recognition network are trained.

[0069] Specifically, the first multi-label behavior recognition network includes a feature extraction network and a first classifier. Since the feature extraction network has been obtained and trained during the training process of the first multi-label behavior recognition network, in order to improve the modeling speed, when training the second label behavior recognition network, the parameters of the feature extraction network can be fixed, that is, the parameters in the feature extraction network are no longer trained, and only the parameters of the second classifier and the binary tree in the second multi-label behavior recognition network are trained for classification tasks. The trained feature extractor is directly used in the second multi-label behavior recognition network to extract image features of the video to be tested.

[0070] Furthermore, for a batch of data in each training iteration, the BCE loss function is used for optimization, and the training loss function is:

[0071]

[0072] Where N represents the number of data in each training iteration, y ij represents the correct label of the jth category in the i-th data, p ij It is the network's predicted probability for the jth category in the i-th data.

[0073] The multi-label behavior recognition device based on a binary tree structure provided by the present invention is described below. The multi-label behavior recognition device based on a binary tree structure described below and the multi-label behavior recognition method based on a binary tree structure described above can be referenced to each other.

[0074] The method provided by the present invention is described below with a specific example.

[0075] like Figure 4 As shown, the multi-label classification method provided by the present invention specifically includes the following steps:

[0076] (1) First, a multi-label video classification model is trained according to the traditional process. In this example, Slowfast50 is used as the feature extraction network. 16 frames of 224 pixels × 224 pixels of continuous video images are used as input. Slowfast50 inputs the extracted 2304-dimensional features into a fully connected layer for classification. In this example, 100 categories are assumed, so the parameter matrix of the fully connected layer has a shape of 2304 × 100. The BCE loss function is used for training and optimization.

[0077] (2) After the training of the multi-label video classification model based on the fully connected layer is completed, the 2304×100 fully connected layer parameter matrix is ​​taken out, and the similarity is calculated for each of the 100 categories to obtain a 100×100 similarity matrix.

[0078] (3) Sort the similarity matrix from large to small, and select the behavior category pairs in order to form a combination. If one or more of the new category pairs have been selected by the previous combination, the current category pair combination is abandoned. Ultimately, it is necessary to ensure that there are no duplicate classes between all category pairs.

[0079] (4) Calculate the similarity between the new combinations and repeat the method in step (3) to combine the new combinations again until only one combination exists. This combination is the root node of the behavior tree.

[0080] (5) Based on the generated behavior tree, reinitialize a classifier with an output dimension of 2 that is different from the first classifier. L -1, the output elements of this classifier will be interpreted according to the correspondence between each element and the nodes in the behavior tree, that is, the product of each behavior's leaf node and all its parent nodes is used as the classification result of each class. In order to train this second classifier, first take out the previously mentioned trained Slowfast50 feature extraction network, and use the features output by the feature extraction network to train the second classifier. During the training of the second classifier, the parameters in the feature extraction network are not updated, and only the parameters in the second classifier are trained.

[0081] (6) Use the trained Slowfast50 feature extraction network + the second classifier to infer the input video to be tested, and then interpret the output elements of the second classifier according to the behavior tree to obtain the confidence probability of each class, completing the multi-label classification process of the video.

[0082] The embodiment of the present invention provides a multi-label behavior recognition device based on a binary tree structure, comprising:

[0083] An extraction module, used to extract the parameter vector of each behavior category output by the first classifier in the trained first multi-label behavior recognition network; wherein the first multi-label behavior recognition network also includes a feature extraction network;

[0084] A binary tree construction module, used to sort the similarities between the behavior categories in the first classifier from high to low, and perform non-repeating recursive combinations of two by two in sequence to construct a binary tree;

[0085] A recognition network construction module, used to construct a second multi-label behavior recognition network based on the feature extraction network, the second classifier and the binary tree; the output dimension of the second classifier is the same as the number of nodes of the binary tree;

[0086] The recognition module is used to perform behavior recognition on the video to be tested using the trained second multi-label behavior recognition network.

[0087] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-label behavior recognition method based on a binary tree structure, characterized in that: include: S1, extracting parameter vectors of each behavior category output by a first classifier in a trained first multi-label behavior recognition network; wherein the first multi-label behavior recognition network also includes a feature extraction network; S2, calculating the similarity between each behavior category based on the parameter vector and sorting them from high to low according to the similarity, and recursively combining each behavior category in pairs without duplication in order to construct a binary tree; S3, constructing a second multi-label behavior recognition network based on the feature extraction network, the second classifier and the binary tree; the output dimension of the second classifier is the same as the number of nodes of the binary tree; Wherein, each element in the output vector of the second classifier uniquely corresponds to a node in the binary tree; The prediction probability of the leaf node corresponding to each category is obtained by multiplying the prediction result of the leaf node corresponding to each category with the prediction results of all parent nodes of the leaf node; S4, using the trained second multi-label behavior recognition network to perform behavior recognition on the video to be tested.

2. The multi-label behavior recognition method based on a binary tree structure as claimed in claim 1, characterized in that: Step S2 includes: The similarities between the behavior categories in the first classifier are sorted from high to low, and non-repetitive recursive combinations are performed in pairs in turn to finally obtain a combination, which is the root node of the binary tree.

3. The multi-label behavior recognition method based on a binary tree structure according to claim 1 or 2, characterized in that: The first classifier and the second classifier are any one of a fully connected layer, a multi-layer perceptron or a SVM.

4. The multi-label behavior recognition method based on a binary tree structure as claimed in claim 1, characterized in that: The number of layers L of the binary tree and the number of behavior categories C of the data set satisfy the following relationship:

5. The multi-label behavior recognition method based on a binary tree structure as claimed in claim 1, characterized in that: The method for determining the similarity between the behavior categories is: Calculate the Cosine similarity between the C D-dimensional feature parameters in the first classifier; Or, the C D-dimensional feature parameters in the first classifier are normalized, and then the similarities between the C feature parameters are calculated.

6. The multi-label behavior recognition method based on a binary tree structure according to claim 1 or 2, characterized in that: The feature extraction network is a deep learning feature extraction network or an extraction network based on manual feature operators.

7. The multi-label behavior recognition method based on a binary tree structure as claimed in claim 1, characterized in that: Only the second classifier and the binary tree in the second multi-label behavior recognition network are trained.

8. A multi-label behavior recognition device based on a binary tree structure, characterized in that: include: An extraction module, used to extract the parameter vector of each behavior category output by the first classifier in the trained first multi-label behavior recognition network; wherein the first multi-label behavior recognition network also includes a feature extraction network; A binary tree construction module, used to sort the similarities between the behavior categories in the first classifier from high to low, and perform non-repeating recursive combinations of two by two in sequence to construct a binary tree; A recognition network construction module, used to construct a second multi-label behavior recognition network based on the feature extraction network, the second classifier and the binary tree; the output dimension of the second classifier is the same as the number of nodes of the binary tree; A recognition module, used for performing behavior recognition on the video to be tested using the trained second multi-label behavior recognition network; Wherein, each element in the output vector of the second classifier uniquely corresponds to a node in the binary tree; The prediction probability of the leaf node corresponding to each category is obtained by multiplying the prediction result of the leaf node corresponding to each category by the prediction results of all parent nodes of the leaf node.

Images