A human action recognition method and a computer readable storage medium

Abstract

The present application relates to a kind of human action recognition method and computer readable storage medium, human action recognition method includes: obtaining video frame sequence and optical flow sequence;Video frame sequence is formed into first training set and test set;Optical flow sequence is formed into second training set;Knowledge distillation model is constructed, specifically for: constructing two teacher networks and a student network, using first training set and second training set to the teacher network constructed is trained, and first teacher network and second teacher network are obtained;First teacher network, second teacher network and the student network constructed are just composed of knowledge distillation model;The constructed knowledge distillation model is trained multiple times, and the student network after training is completed is obtained;In test set, an arbitrary test sample is selected, and is input into the student network after training, i.e. action recognition result is obtained.The action recognition accuracy of the method is better, and has higher recognition rate.

Description

Technical Field

[0001] This invention relates to the field of motion recognition technology, and in particular to a human motion recognition method and a computer-readable storage medium. Background Technology

[0002] Video information is an important part of people's daily lives, and methods for processing and analyzing it have been extensively studied. Video human motion recognition has wide applications, not only in motion analysis but also in advanced human-computer interaction, video surveillance, and virtual reality.

[0003] There are many existing methods for human motion recognition. For example, Chinese invention patent application number CN 202410210350.4 (publication number CN118015704A) discloses a motion recognition method and system based on three-dimensional convolutional networks; another example is Chinese invention patent application number CN202311764970.4 (publication number CN117523675A), which discloses a human motion recognition method, device, and medium based on neural networks. Although many existing methods can achieve human motion recognition, the accuracy and speed of motion recognition are significantly affected by factors such as the complexity of existing research models. Therefore, further improvements to existing technologies are needed. Summary of the Invention

[0004] The first technical problem to be solved by the present invention is to provide a human motion recognition method with good recognition accuracy and high recognition speed, in contrast to the above-mentioned prior art.

[0005] The second technical problem to be solved by the present invention is to provide a computer-readable storage medium capable of performing the above-described human motion recognition method.

[0006] The technical solution adopted by the present invention to solve the first technical problem mentioned above is: a human motion recognition method, characterized by comprising the following steps:

[0007] S1. Obtain multiple videos containing human actions, and perform sparse sampling on each video to obtain multiple sparsely sampled video frame sequences.

[0008] S2. Calculate the optical flow graphs corresponding to two adjacent video frames in each video frame sequence, and construct an optical flow sequence from all the optical flow graphs.

[0009] S3. The video frame sequences in S1 are used to form the first dataset, and the first dataset is divided into a first training set and a test set; the optical flow sequences in S2 are used to form the second training set.

[0010] S4. Construct a knowledge distillation model;

[0011] The specific process of constructing the knowledge distillation model is as follows:

[0012] Two teacher networks and one student network are constructed. The constructed teacher networks are trained using the first training set and the second training set, respectively, to obtain the first teacher network and the second teacher network. The first teacher network, the second teacher network and the constructed student network constitute the knowledge distillation model.

[0013] S5. Train the knowledge distillation model constructed in S4 multiple times to obtain the trained student network.

[0014] The specific process of training the constructed knowledge distillation model once is as follows:

[0015] S5-1. Select multiple training samples from the first training set and the second training set respectively. The training samples in the first training set are denoted as the first training sample, and the training samples in the second training set are denoted as the second training sample. The first training sample includes a video frame sequence and the real label corresponding to the video frame sequence. The second training sample includes an optical flow sequence and the real label corresponding to the optical flow sequence.

[0016] S5-2. Perform the following operations respectively:

[0017] All the first training samples are input into the first teacher network to obtain the first feature map set output by the first teacher network, and the first feature map set is passed through a fully connected layer to obtain the first classification vector set;

[0018] All the first training samples are input into the student network to obtain the second feature map set output by the student network, and the second feature map set is passed through a fully connected layer to obtain the second classification vector set;

[0019] All the second training samples are input into the second teacher network to obtain the third feature map set output by the second teacher network, and the third feature map set is passed through a fully connected layer to obtain the third classification vector set;

[0020] S5-3. Merge the first feature map set and the third feature map set to obtain a fused feature map set; and merge the first classification vector set and the third classification vector set to obtain a fused classification vector set.

[0021] S5-4. Construct a structural knowledge network by inputting the fused feature map set and the second feature map set into the constructed structural knowledge network and calculating the first loss function.

[0022] S5-5. Obtain the predicted label corresponding to each second classification vector in the second classification vector set, and calculate the second loss function based on the predicted label and the true label; in addition, calculate the third loss function based on the second classification vector set and the fused classification vector set, and finally add the first loss function, the second loss function and the third loss function in a weighted sum to obtain the total loss function.

[0023] S5-6. Update the constructed student network using the total loss function to obtain the student network after one training cycle.

[0024] S6. Randomly select a test sample from the test set and input it into the trained student network to obtain the action recognition result corresponding to the test sample.

[0025] Preferably, both teacher networks use ResNet101 networks.

[0026] Preferably, the student network adopts a ResNet18 network.

[0027] To enable the student network to acquire more information, in step S5-4, the fused feature map set and the second feature map set are input into the constructed structural knowledge network, and the specific process of calculating the first loss function is as follows:

[0028] The second feature map set is subjected to average pooling, so that the size of the second feature map set becomes ( , );in The total number of samples in the second feature map set, where each sample has one frame and each frame is a 1*1 pixel second feature map. The number of channels in the second feature map of each frame;

[0029] The Huber loss function is used to calculate the similarity between two adjacent channels of the second feature map in each frame, obtaining ( -1) Output of dimension size, which is the output of the structural knowledge network corresponding to the second feature map of each frame;

[0030] The fused feature map set is subjected to average pooling, so that the size of the fused feature map set becomes ( , );in This represents the total number of samples in the fused feature map set. Each sample has one frame, and each frame is a 1*1 pixel fused feature map. The number of channels in the fused feature map for each frame;

[0031] Divide the channel dimensions of the fused feature map of each frame to obtain Group channels; the Huber loss function is used to calculate the similarity between adjacent groups of channels in the fused feature map of each frame, obtaining ( -1) Output of dimension size, which is the output of the structural knowledge network corresponding to the fused feature map of each frame;

[0032] Finally, the outputs of the structural knowledge network corresponding to all the second feature maps and the outputs of the structural knowledge network corresponding to all the fused feature maps are substituted into the KL divergence loss function to calculate the first loss function. .

[0033] In the above scheme, the total loss function in step S5-5 The calculation formula is:

[0034]

[0035] in, For the second loss function, For the third loss function, , and All are preset weights and .

[0036] Preferably, the second loss function The cross-entropy loss function is used.

[0037] Preferably, the third loss function The calculation method is as follows:

[0038] Each second-class vector in the second-class vector set is input into a softmax function with a temperature coefficient T to obtain the first output probability;

[0039] Each fused classification vector in the fused classification vector set is input into a softmax function with a temperature coefficient T to obtain the second output probability;

[0040] Then, the first and second output probabilities are substituted into the KL divergence loss function to obtain the third loss function.

[0041] The technical solution adopted by the present invention to solve the second technical problem mentioned above is: a computer-readable storage medium storing a computer program that can be read and executed by a processor, characterized in that: when the computer program is executed by the processor, it implements the human motion recognition method described above.

[0042] Compared with existing technologies, the advantages of this invention are as follows: By establishing two teacher networks and one student network, video frame sequences and optical flow sequences are respectively input into the corresponding teacher networks, and the feature maps output by the two teacher networks are fused, thereby enabling the teacher networks to learn more information; in addition, the fused feature maps and the feature maps output by the student network are input into the constructed structural knowledge network, utilizing the potential structural information between adjacent channel features, this structural knowledge is transferred from the teacher network to the student network, allowing the student network to focus more on more information from the teacher network. Therefore, the method has better action recognition accuracy and a higher recognition rate. Attached Figure Description

[0043] Figure 1 This is a structural diagram of the knowledge distillation model in an embodiment of the present invention. Detailed Implementation

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0045] The human motion recognition method in this embodiment includes the following steps:

[0046] S1. Obtain multiple videos containing human actions, and perform sparse sampling on each video to obtain multiple sparsely sampled video frame sequences.

[0047] In this embodiment, to reduce redundancy between video frames, the video is first sampled at equal intervals, with one frame sampled every 16 frames. Then, the sampled video images are sampled a second time, that is, the video frames are divided into N segments, and one frame is randomly sampled from each segment, for a total of N frames. During the training phase of the two teacher networks, each frame after the second sampling is augmented with data augmentation techniques such as horizontal flipping, random cropping, color dithering, and normalization. During the verification and testing phase of the two teacher networks, no data augmentation techniques are used.

[0048] In this embodiment, the HMDB51 dataset is used as the first dataset. The HMDB51 dataset is an existing open-source dataset with a total of 6766 videos and 51 categories, mainly including general facial movements, facial movements manipulated by objects, general body movements, body movements and object interactions, and body movements of human-to-human interactions.

[0049] S2. Calculate the optical flow graphs corresponding to two adjacent video frames in each video frame sequence, and construct an optical flow sequence from all the optical flow graphs.

[0050] S3. The video frame sequences in S1 are used to form the first dataset, and the first dataset is divided into a first training set and a test set; the optical flow sequences in S2 are used to form the second training set.

[0051] S4. Construct a knowledge distillation model;

[0052] The specific process of constructing the knowledge distillation model is as follows:

[0053] Two teacher networks and one student network are constructed. The constructed teacher networks are trained using the first training set and the second training set, respectively, to obtain the first teacher network and the second teacher network. The first teacher network, the second teacher network and the constructed student network constitute the knowledge distillation model.

[0054] In this embodiment, both teacher networks use ResNet101 networks; the student network uses ResNet18 networks; the aforementioned ResNet101 and ResNet18 networks are existing technologies and will not be described in detail here.

[0055] S5. Train the knowledge distillation model constructed in S4 multiple times to obtain the trained student network.

[0056] like Figure 1 As shown, the specific process of training the constructed knowledge distillation model once is as follows:

[0057] S5-1. Select multiple training samples from the first training set and the second training set respectively. The training samples in the first training set are denoted as the first training sample, and the training samples in the second training set are denoted as the second training sample. The first training sample includes a video frame sequence and the real label corresponding to the video frame sequence. The second training sample includes an optical flow sequence and the real label corresponding to the optical flow sequence.

[0058] S5-2. Perform the following operations respectively:

[0059] All the first training samples are input into the first teacher network to obtain the first feature map set output by the first teacher network, and the first feature map set is passed through a fully connected layer to obtain the first classification vector set;

[0060] All the first training samples are input into the student network to obtain the second feature map set output by the student network, and the second feature map set is passed through a fully connected layer to obtain the second classification vector set;

[0061] All the second training samples are input into the second teacher network to obtain the third feature map set output by the second teacher network, and the third feature map set is passed through a fully connected layer to obtain the third classification vector set;

[0062] In this embodiment, the first feature map set is the feature map set output by the front end of the fully connected layer in the first teacher network (ResNet101 network), the second feature map set is the feature map set output by the front end of the fully connected layer in the student network (ResNet18 network), and the third feature map set is the feature map set output by the front end of the fully connected layer in the second teacher network (ResNet101 network).

[0063] S5-3. Merge the first feature map set and the third feature map set to obtain a fused feature map set; and merge the first classification vector set and the third classification vector set to obtain a fused classification vector set.

[0064] S5-4. Construct a structural knowledge network by inputting the fused feature map set and the second feature map set into the constructed structural knowledge network and calculating the first loss function.

[0065] In this embodiment, the specific process of inputting the fused feature map set and the second feature map set into the constructed structural knowledge network and calculating the first loss function is as follows:

[0066] The second feature map set is subjected to average pooling, so that the size of the second feature map set becomes ( , );in The total number of samples in the second feature map set, where each sample has one frame and each frame is a 1*1 pixel second feature map. The number of channels in the second feature map of each frame;

[0067] The Huber loss function is used to calculate the similarity between two adjacent channels of the second feature map in each frame, obtaining ( -1) Output of dimension size, which is the output of the structural knowledge network corresponding to the second feature map of each frame;

[0068] The fused feature map set is subjected to average pooling, so that the size of the fused feature map set becomes ( , );in This represents the total number of samples in the fused feature map set. Each sample has one frame, and each frame is a 1*1 pixel fused feature map. The number of channels in the fused feature map for each frame;

[0069] Divide the channel dimensions of the fused feature map of each frame to obtain Group channels; the Huber loss function is used to calculate the similarity between adjacent groups of channels in the fused feature map of each frame, obtaining ( -1) Output of dimension size, which is the output of the structural knowledge network corresponding to the fused feature map of each frame;

[0070] Finally, the outputs of the structural knowledge network corresponding to all the second feature maps and the outputs of the structural knowledge network corresponding to all the fused feature maps are substituted into the KL divergence loss function to calculate the first loss function. ;

[0071] The Huber loss function and KL divergence loss function mentioned above are existing technologies and will not be elaborated further here;

[0072] S5-5. Obtain the predicted label corresponding to each second classification vector in the second classification vector set, and calculate the second loss function based on the predicted label and the true label; in addition, calculate the third loss function based on the second classification vector set and the fused classification vector set, and finally add the first loss function, the second loss function and the third loss function in a weighted sum to obtain the total loss function.

[0073] Total loss function The calculation formula is:

[0074]

[0075] in, For the second loss function, For the third loss function, , and All are preset weights and In this embodiment, , , ;

[0076] Second loss function Employ the cross-entropy loss function;

[0077] Third loss function The calculation method is as follows:

[0078] Each second-class vector in the second-class vector set is input into a softmax function with a temperature coefficient T to obtain the first output probability;

[0079] In this embodiment, T = t, where t is any value. Figure 1 The meaning of softmax(T=t) in this context is: the value of T is the softmax function corresponding to t; Figure 1 The meaning of softmax(T=1) in the text is: the softmax function corresponding to the value of T being 1;

[0080] Each fused classification vector in the fused classification vector set is input into a softmax function with a temperature coefficient T to obtain the second output probability;

[0081] Then, the first and second output probabilities are substituted into the KL divergence loss function to obtain the third loss function;

[0082] S5-6. Update the constructed student network using the total loss function to obtain the student network after one training cycle.

[0083] S6. Randomly select a test sample from the test set and input it into the trained student network to obtain the action recognition result corresponding to the test sample.

[0084] This embodiment also relates to a computer-readable storage medium storing a computer program that can be read and executed by a processor. When the computer program is executed by the processor, it implements the above-mentioned human motion recognition method.

[0085] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for recognizing human actions, characterized in that... Includes the following steps: S1. Obtain multiple videos containing human actions, and perform sparse sampling on each video to obtain multiple sparsely sampled video frame sequences. S2. Calculate the optical flow graphs corresponding to two adjacent video frames in each video frame sequence, and construct an optical flow sequence from all the optical flow graphs. S3. The video frame sequences in S1 are used to form the first dataset, and the first dataset is divided into a first training set and a test set; the optical flow sequences in S2 are used to form the second training set. S4. Construct a knowledge distillation model; The specific process of constructing the knowledge distillation model is as follows: Two teacher networks and one student network are constructed. The constructed teacher networks are trained using the first training set and the second training set, respectively, to obtain the first teacher network and the second teacher network. The first teacher network, the second teacher network and the constructed student network constitute the knowledge distillation model. S5. Train the knowledge distillation model constructed in S4 multiple times to obtain the trained student network. The specific process of training the constructed knowledge distillation model once is as follows: S5-1. Select multiple training samples from the first training set and the second training set respectively. The training samples in the first training set are denoted as the first training sample, and the training samples in the second training set are denoted as the second training sample. The first training sample includes a video frame sequence and the real label corresponding to the video frame sequence. The second training sample includes an optical flow sequence and the real label corresponding to the optical flow sequence. S5-2. Perform the following operations respectively: All the first training samples are input into the first teacher network to obtain the first feature map set output by the first teacher network, and the first feature map set is passed through a fully connected layer to obtain the first classification vector set; All the first training samples are input into the student network to obtain the second feature map set output by the student network, and the second feature map set is passed through a fully connected layer to obtain the second classification vector set; All the second training samples are input into the second teacher network to obtain the third feature map set output by the second teacher network, and the third feature map set is passed through a fully connected layer to obtain the third classification vector set; S5-3. Merge the first feature map set and the third feature map set to obtain a fused feature map set; and merge the first classification vector set and the third classification vector set to obtain a fused classification vector set. S5-4. Construct a structural knowledge network by inputting the fused feature map set and the second feature map set into the constructed structural knowledge network and calculating the first loss function. S5-5. Obtain the predicted label corresponding to each second classification vector in the second classification vector set, and calculate the second loss function based on the predicted label and the true label; in addition, calculate the third loss function based on the second classification vector set and the fused classification vector set, and finally add the first loss function, the second loss function and the third loss function in a weighted sum to obtain the total loss function. S5-6. Update the constructed student network using the total loss function to obtain the student network after one training cycle. S6. Randomly select a test sample from the test set and input it into the trained student network to obtain the action recognition result corresponding to the test sample.

2. The human motion recognition method according to claim 1, characterized in that: Both teacher networks use the ResNet101 network.

3. The human motion recognition method according to claim 1, characterized in that: The student network uses a ResNet18 network.

4. The human motion recognition method according to claim 1, characterized in that: In step S5-4, the fused feature map set and the second feature map set are input into the constructed structural knowledge network, and the specific process of calculating the first loss function is as follows: The second feature map set is subjected to average pooling, so that the size of the second feature map set becomes (n, c', 1, 1, 1); where n is the total number of samples in the second feature map set, each sample has 1 frame, and each frame is a 1*1 pixel second feature map, and c' is the number of channels in each frame of the second feature map. The Huber loss function is used to calculate the similarity between two adjacent channels of the second feature map in each frame, and an output of size (n, c′-1) is obtained. This output is the output of the structural knowledge network corresponding to the second feature map in each frame. The fused feature map set is subjected to average pooling, so that the size of the fused feature map set becomes (n, c, 1, 1, 1); where n is the total number of samples in the fused feature map set, each sample has 1 frame, and each frame is a 1*1 pixel fused feature map, and c is the number of channels in each frame of the fused feature map. The channel dimension of each frame's fused feature map is divided to obtain c′ groups of channels; The Huber loss function is used to calculate the similarity between two adjacent channels of the fused feature map in each frame, and an output of size (n, c′-1) is obtained. This output is the output of the structural knowledge network corresponding to the fused feature map in each frame. Finally, the outputs of the structural knowledge network corresponding to all the second feature maps and the outputs of the structural knowledge network corresponding to all the fused feature maps are substituted into the KL divergence loss function to calculate the first loss function stru_loss.

5. The human motion recognition method according to claim 4, characterized in that: The formula for calculating the total loss function Loss in step S5-5 is as follows: Loss=α·hard_loss+β·soft_loss+δ·stru_loss Here, hard_loss is the second loss function, soft_loss is the third loss function, and α, β and δ are preset weights with α+β+δ=1.

6. The human motion recognition method according to claim 5, characterized in that: The second loss function, hard_loss, uses the cross-entropy loss function.

7. The human motion recognition method according to claim 5, characterized in that: The third loss function, soft_loss, is calculated as follows: Each second-class vector in the second-class vector set is input into a softmax function with a temperature coefficient T to obtain the first output probability; Each fused classification vector in the fused classification vector set is input into a softmax function with a temperature coefficient T to obtain the second output probability; Then, the first and second output probabilities are substituted into the KL divergence loss function to obtain the third loss function.

8. A computer-readable storage medium storing a computer program that can be read and executed by a processor, characterized in that: When the computer program is executed by the processor, it implements the human motion recognition method as described in any one of claims 1 to 7.

Images