Human action recognition algorithm in compressed video based on optical flow distillation

Through optical flow distillation technology, optical flow characteristics are migrated from the time flow network to the spatial flow network. Combined with the MPEG-4 video compression coding algorithm, the problem of excessive time spent extracting optical flow characteristics in the dual-stream network is solved, real-time human behavior is achieved Identify.

CN114708646BActive Publication Date: 2024-11-29广州新华学院
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202111329644.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-10
Publication Date
2024-11-29
Estimated Expiration
2041-11-10

AI Technical Summary

Technical Problem

When using optical flow features, the dual-stream network takes a long time to extract motion information, which makes the model's time-consuming consumption during inference cannot meet the requirements of practical applications, making it difficult to achieve real-time behavior recognition.

Method used

Using compressed video human behavior recognition algorithm with optical flow distillation, the optical flow characteristics are distilled into the spatial flow network by using the time flow network as a teacher model, and the RGB image frame and motion information are extracted using the MPEG-4 video compression encoding algorithm to perform Supervise training to reduce optical flow extraction time.

Benefits of technology

Optical flow features are extracted during training, and only video frames are needed for inference and testing, which reduces optical flow extraction time, improves the performance and efficiency of the algorithm, and realizes real-time behavior recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114708646B_ABST
    Figure CN114708646B_ABST
Patent Text Reader

Abstract

The present invention provides a human behavior recognition algorithm for compressed videos based on optical flow distillation. The present invention uses optical flow to supervise network training by means of knowledge distillation. The time stream network in a dual-stream network is used as a teacher model. The motion information extracted by optical flow in the teacher model is distilled into a student model of a spatial stream. The student model takes RGB image frames as input (i.e., I frames, motion vectors and residuals in the video are extracted in MPEG-4 video compression encoding format, and P frames generated by stacking and I frames extracted from the original video are used as feature inputs of the network model). In this way, optical flow features can be extracted only during training, while only video frames need to be used as input during reasoning and testing, thereby reducing the optical flow extraction time and improving the performance of the algorithm while ensuring efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of human behavior recognition, and in particular to a compressed video human behavior recognition algorithm based on optical flow distillation. Background Art

[0002] The two-stream network is the most representative network framework for human behavior recognition. The two-stream network framework uses the appearance stream (spatial stream) network and the motion stream (temporal stream) network to model the video respectively, and fuses the results of the two streams as the final recognition result. The spatial features in the two-stream framework refer to the appearance and background information used to describe the moving human body in the video, usually the RGB image of the video frame, which contains more appearance details; while the temporal features are used to describe the motion changes and displacement information between video frames; the two-stream network mainly uses optical flow as the temporal feature, where the optical flow uses the changes in the pixels of the image in the time domain in the video frame sequence and the correlation with the adjacent frames to calculate the correspondence between the previous frame and the current frame. Compared with the motion vector and residual calculated according to the block displacement between video frames in the video compression coding feature, the optical flow has pixel-level accuracy and contains more action details and richer motion information.

[0003] Given the above, the two-stream network can achieve higher algorithm accuracy by using optical flow as temporal feature; however, since the temporal flow network that expresses motion information usually uses optical flow features, it takes a long time for the two-stream network to extract optical flow features under the existing hardware conditions, resulting in the model's time consumption during the reasoning process being unable to meet the requirements of practical applications, making it difficult to achieve real-time behavior recognition.

[0004] Therefore, it is necessary to provide a compressed video human behavior recognition algorithm based on optical flow distillation to solve the above technical problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a compressed video human behavior recognition algorithm based on optical flow distillation, which is used to solve the problem that in the recognition algorithm using optical flow as the temporal feature in the dual-stream network, since the temporal flow network that expresses motion information usually uses optical flow features, it takes a long time for the dual-stream network to extract optical flow features under the existing hardware conditions, resulting in the time consumption of the model in the reasoning process cannot meet the requirements of practical applications, and it is difficult to achieve real-time behavior recognition.

[0006] The present invention provides a human behavior recognition algorithm for compressed videos based on optical flow distillation, which comprises the following steps:

[0007] Step 1: First, use a two-stream network consisting of exactly the same temporal stream network and spatial stream network to model the compressed video sample, and use the temporal stream network as the teacher model and the spatial stream network as the student model;

[0008] Step 2: Use optical flow as a high-level feature to pre-train the teacher model of the temporal flow network obtained in step 1, then freeze the network weights to extract motion information;

[0009] Step 3: Use MPEG-4 video compression coding algorithm to extract RGB image frames from the compressed video samples, and input the RGB image frames into the student model;

[0010] Step 4: Distill the motion information extracted by the optical flow method in the teacher model into the student model of the spatial flow network through knowledge distillation, and perform supervised training on the network.

[0011] Preferably, the MPEG-4 video compression encoding algorithm described in step 3 includes the following operation steps:

[0012] a: Use intra-frame prediction compression algorithm to segment the compressed video samples, calculate the similarity of adjacent frames, merge the highly correlated frames according to the threshold, and obtain the group of pictures (GOP);

[0013] b: Using a motion estimation and compensation algorithm to perform inter-frame prediction compression on the group of pictures (GOP) obtained in step a, obtain a motion vector (MV) and a residual (R), and record the motion vector (MV) and the residual (R) of the current frame as a P frame or a B frame;

[0014] c: The motion vectors and residuals are encoded by discrete cosine transform (DCT) and entropy coding and stored as MPEG-4 video.

[0015] Preferably, since a single P frame in a group of pictures (GOP) itself only describes the changes between the previous and next two frames, the signal-to-noise ratio is relatively low and it is difficult to model. In addition, it may also have a chain dependency relationship with the P frames at multiple previous moments. Extracting only a single P frame for modeling will destroy this chain dependency. Therefore, all motion vectors in the P frame are traced back to the I frame at the previous moment through a decoupling model, and multiple similar P frames after the I frame in each group of pictures (GOP) are stacked to form a stacked P frame structure, so that each P frame only depends on the I frame but not on other P frames, and the noise of the P frame image is reduced by aggregating multiple P frames.

[0016] Preferably, the RGB image frame described in step three is an I frame, motion vector and residual extracted by an MPEG-4 video compression encoding algorithm.

[0017] Preferably, step three also includes a P frame restoration strategy, specifically: extracting a P frame every K frames from the sampled independent P frames and restoring it to a complete RGB image frame as the input of the spatial stream of the I frame, so as to increase the sample size of the I frame network input and avoid a small number of I frames extracted in a short video. If the number of I frames is small, it is easy to cause underfitting of the model.

[0018] Preferably, step four also includes a distillation model used for knowledge distillation, and the total loss of the distillation model is composed of soft labels and hard labels generated by the teacher model, wherein the soft label is a classification label and the hard label is a label of the actual classification layer. The soft label is the knowledge that needs to be "taught" from the teacher model to the student model, and the hard label refers to the actual classification label in the data, which usually comes from the data set.

[0019] Compared with the related art, the human behavior recognition algorithm for compressed video based on optical flow distillation provided by the present invention has the following beneficial effects:

[0020] The present invention uses optical flow to supervise network training by means of knowledge distillation, takes the time stream network in the dual-stream network as the teacher model, and distills the motion information extracted by optical flow in the teacher model into the student model of the spatial stream. The student model takes RGB image frames as input (i.e., the I frames, motion vectors and residuals in the video are extracted in the MPEG-4 video compression encoding format, and the stacked P frames and the I frames extracted from the original video are used as feature inputs of the network model). This method can extract optical flow features only during training, and only video frames need to be used as input during reasoning and testing, which reduces the optical flow extraction time and improves the performance of the algorithm while ensuring efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the MPEG-4 video compression encoding algorithm of the present invention;

[0022] Figure 2 This is a network framework diagram of optical flow distillation of the present invention; DETAILED DESCRIPTION

[0023] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0024] A human behavior recognition algorithm for compressed videos based on optical flow distillation includes the following steps:

[0025] Step 1: First, use a two-stream network consisting of exactly the same temporal stream network and spatial stream network to model the compressed video sample, and use the temporal stream network as the teacher model and the spatial stream network as the student model;

[0026] Step 2: Use optical flow as a high-level feature to pre-train the teacher model of the temporal flow network obtained in step 1, then freeze the network weights to extract motion information;

[0027] Step 3: Use MPEG-4 video compression coding algorithm to extract RGB image frames from the compressed video samples, and input the RGB image frames into the student model;

[0028] Step 4: Distill the motion information extracted by the optical flow method in the teacher model into the student model of the spatial flow network through knowledge distillation, and perform supervised training on the network.

[0029] Knowledge distillation principle

[0030] Knowledge distillation technology originated from the field of transfer learning. It refers to the technology of transferring knowledge from a high-performance complex model to a simple lightweight model. The high-performance complex model and the lightweight simple model are called the teacher model and the student model respectively. Generally, the trained knowledge is transferred from the teacher model to the student network by making the probability distribution of the student model classification close to the distribution predicted by the teacher model. Compared with the original student model, the distilled student model has richer knowledge and better performance, and has fewer parameters and lower model complexity. Knowledge distillation usually trains the teacher model by minimizing the loss function of the teacher model, and then minimizes the difference loss between the student model and the teacher model, so that the loss function of the student model fits the teacher model.

[0031] Principle of MPEG-4 video compression encoding algorithm

[0032] The MPEG-4 video compression encoding method divides the video into I frames (intra-frame compression frames), P frames (inter-frame prediction frames) and B frames (bidirectional prediction frames). Figure 2 As shown, the I frame is a key frame in the video, usually an RGB image containing complete information of the video frame; the P frame is an inter-frame prediction frame, each P frame is predicted by the previous P frame or I frame, and the B frame is a bidirectional prediction frame, which is predicted by the previous I or P frame and the following P frame. Both the P frame and the B frame are calculated using motion estimation and compensation methods. The complete compressed video will be stacked by I frames, P frames and B frames.

[0033] When the video compression coding algorithm performs video coding, the processing of the video mainly includes two parts: intra-frame prediction compression and inter-frame prediction compression. The compression algorithm first divides each frame of the image, cuts the video frame image into several strips horizontally, and then cuts it into several blocks vertically. The divided pixel blocks are called macroblocks. Usually, 16×16 macroblocks are used as the basic operation unit. Then the video frames are grouped, and the similarity of macroblocks between two adjacent frames is calculated. The video frames with high correlation are merged into a group according to the threshold. In each group of frames, only the complete image data of the first frame at the beginning is retained, which is called I frame. The other frames are calculated with reference to the previous frame, which is called P frame or B frame. The encoded video frame group is called GOP.

[0034] Then, the motion estimation and compensation algorithm is used to perform inter-frame prediction compression on the GOP. First, the displacement change of the same macroblock between two frames of the video is calculated. The direction and distance of the change of this part of the macroblock between the two frames are called motion vectors (Motion Vector, MV). Then, according to the motion vector, the same image part as the previous frame is deleted to obtain the final compensated image, called residual (Residual, R). The motion vector and residual of the current frame are recorded as P frame or B frame, indicating the change between video frames in the GOP. Finally, the compression coding algorithm encodes the motion vector and residual through discrete cosine transform (DCT) and entropy coding and stores them as MPEG-4 video. In this way, the I frame and residual R obtained by the compression algorithm are three-channel (RGB) images with the same resolution as the original video, and the motion vector MV is a vector of two channels containing horizontal displacement and vertical displacement.

[0035] P frames are composed of motion vectors and residuals, and the motion vectors and residuals are calculated unidirectionally with reference to the P frames of the previous moment. B frames can be regarded as special P frames. Similar to P frames, they calculate motion vectors and residuals bidirectionally with reference to the P frames of the previous moment and the next moment. However, since B frames contain information at future moments, they are not of much help to the behavior recognition task itself. The present invention only uses I frames and P frames as inputs to the model.

[0036] At time t in the video, the P frame consists only of the motion vector T(t) and the residual Δ(t) at that moment. The P frame at time t in the original video frame is reconstructed from the P frame or I frame at time t-1. The image definition of the P frame is shown in formula (1).

[0037]

[0038] Where i represents the i-th pixel of the video frame

[0039] However, a single P-frame in the GOP itself only describes the changes between the two adjacent frames, usually with a relatively low signal-to-noise ratio, making it difficult to model. Moreover, there may be a chain of dependencies with P-frames at multiple previous times. Extracting only a single P-frame for modeling will break this chain of dependencies. The present invention adopts a decoupled model of P-frames, accumulates all motion vectors back to the I-frame of the previous moment, stacks multiple similar P-frames after the I-frame in each GOP to form a stacked P-frame structure, so that each P-frame only depends on the I-frame and does not depend on other P-frames, and also reduces the noise of the P-frame image through the aggregation of multiple P-frames.

[0040] For a given P-frame t in the video, define μT(t)(i):=i - Ti(t) to represent the position referenced by pixel i in the previous frame. Each frame k < t traced back from video frame t is defined as shown in formula (2).

[0041]

[0042] Therefore, the calculation formulas for the motion vector D(t) and the residual R(t) accumulated on video frame t are as shown in formulas (3) and (4).

[0043]

[0044]

[0045] The calculation of the P-frame image redefined by formula (1) is as shown in formula (5).

[0046]

[0047] In this way, each P-frame is calculated by accumulating the previous P-frames and only depends on the initial I-frame, which can contain richer motion information and is more robust to noise and camera motion.

[0048] The present invention uses FFMEPG to extract the I-frames, motion vectors (MVs), and residuals (Rs) in the video from the MPEG-4 video compression coding format, and takes the P-frames generated by stacking through formula (5) and the I-frames extracted from the original video as the feature inputs of the network model.

[0049] However, in the case of using FFMPEG for compression coding format extraction, the number of I-frames extracted from a video with a short duration is small, which easily leads to model underfitting. The present invention proposes a restoration strategy for P-frames, extracts a P-frame every K frames from the sampled independent P-frames and restores it to a complete RGB image frame as the input of the spatial stream of the I-frame, increasing the sample size of the I-frame network input.

[0050] It can be seen that the I frame itself contains complete video frame spatial information, and different I frames are significantly different, meeting the requirements of sparse sampling, and can be used as a representation feature of video frame appearance information; the P frame records the motion of macroblocks in the video frame. Although the feature accuracy is at the macroblock level, it can also be used as a weak modality representation of video motion information. In particular, since the I frame, motion vector (MV) and residual (R) can be directly obtained from video compression coding, compared to the optical flow features that need to be recalculated, the features of the present invention can be directly extracted from the compressed video coding format, which greatly reduces the amount of computational effort in extracting features, thereby greatly improving the efficiency of the algorithm.

[0051] In the specific implementation process, Figure 2 As shown in the figure, the total loss of the distillation model consists of two parts. In the present invention, the classification label generated by the teacher model is called a soft label, and the label of the actual classification layer (ground truth) is called a hard label. The soft label is the knowledge that needs to be "taught" from the teacher model to the student model, and the hard label refers to the actual classification label in the data, which usually comes from the data set.

[0052] The present invention uses optical flow as a high-level feature training teacher network model, and guides the temporal flow network composed of motion vectors and residuals for supervision. In the process of network training, two identical spatiotemporal decomposition networks are first constructed as the teacher model and the student model respectively, and then the teacher model is pre-trained using optical flow, and then the network weights are frozen, and then the same network is trained using P frames as input, and the frozen network weights in the teacher model are used as initial values. The feature map output by the teacher model and the result of the real classification will be used together as the supervision of the network; however, during reasoning, the teacher network model of the optical flow is discarded, and only the student network model with P frames as input is used.

[0053] The total loss function of the student network with P frames as input consists of two parts. The first part is the difference between the hard labels of the student network and the classification layer (ground truth), and the second part is the difference between the soft labels generated by the student network and the teacher network. The temporal flow network loss function It is expressed as formula (6)

[0054]

[0055] in represents the cross-entropy loss of the temporal stream network and hard labels, The feature f of the P frame output by the temporal stream network P and optical flow feature f oThe Euclidean distance between them, λ represents the weight of the loss function.

[0056] In the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction. Therefore, they cannot be understood as limitations on the present invention.

[0057] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A human behavior recognition algorithm for compressed video based on optical flow distillation, characterized in that: The steps include: Step 1: First, use a two-stream network consisting of exactly the same temporal stream network and spatial stream network to model the compressed video sample, and use the temporal stream network as the teacher model and the spatial stream network as the student model; Step 2: Use optical flow as a high-level feature to pre-train the teacher model of the temporal flow network obtained in step 1, then freeze the network weights to extract motion information; Step 3: Extract RGB image frames from the compressed video samples using the MPEG-4 video compression coding algorithm, and input the RGB image frames into the student model, wherein the RGB image frames include I frames, motion vectors and residuals extracted by the MPEG-4 video compression coding algorithm. Wherein, inputting the RGB image frame into the student model includes: Through the decoupling model, all motion vectors in the P frame are traced back to the I frame at the previous moment. At the same time, multiple similar P frames after the I frame in each picture group are stacked to form a stacked P frame structure, so that each P frame only depends on the I frame but not on other P frames. The stacked P frames and the extracted I frames are used as feature inputs of the student model. In step 3, a P frame restoration strategy is also included. Specifically, the restoration strategy is to extract a P frame from the sampled independent P frame every K frames and restore it to a complete RGB image frame as the input of the spatial stream of the I frame, so as to increase the sample volume of the I frame network input; Step 4: Distill the motion information extracted by the optical flow method in the teacher model into the student model of the spatial flow network through knowledge distillation, and perform supervised training on the network.

2. According to claim 1, a human behavior recognition algorithm based on compressed video based on optical flow distillation is characterized in that: The MPEG-4 video compression encoding algorithm described in step 3 includes the following steps: a: Use intra-frame prediction compression algorithm to segment the compressed video samples, calculate the similarity of adjacent frames, merge the frames with high correlation according to the threshold, and obtain the picture group; b: Using a motion estimation and compensation algorithm to perform inter-frame prediction compression on the picture group obtained in step a, obtain a motion vector and a residual, and record the motion vector and residual of the current frame as a P frame or a B frame; c: Motion vectors and residuals are encoded by discrete cosine transform and entropy coding and stored as MPEG-4 video.

3. According to claim 1, a human behavior recognition algorithm based on compressed video based on optical flow distillation is characterized in that: Step 4 also includes a distillation model used for knowledge distillation, where the total loss of the distillation model consists of soft labels and hard labels generated by the teacher model, where the soft labels are classification labels and the hard labels are labels of the actual classification layer.

Citation Information

Patent Citations

  • Space-time double-flow segmented network behavior identification method and system based on knowledge distillation

    CN112446331A

  • Method for detecting and recognizing weak target in airport panoramic video

    CN112712052A

  • Optical flow estimation method and system based on guided learning strategy

    CN113327265A

  • System and method for knowledge distillation between neural networks

    US20200302295A1