A picture enhancement decoding method for real-time video analysis

Abstract

The present invention discloses a picture enhancement decoding method for real-time video analysis, comprising the following steps: after the video code stream is transmitted to the server through the network, the scheduler divides all frames into three categories, and each category executes one of the three pipelines. The three pipelines include 1. using a super-resolution model to enhance a group of low-resolution anchor frames into high-resolution frames and then input them into the inference model; 2. using code stream information (such as frame reference relationship, motion vector and residual) to diffuse the gain of the anchor frame to the non-anchor frame and input the non-anchor frame into the inference model; 3. using the code stream information motion vector to reuse the result of the inference model on the non-inference frame. The scheduler uses the picture features extracted from the key frame and the frame difference features extracted from the code stream information to perform reasonable frame classification; picture enhancement improves the inference accuracy of video analysis, and the use of code stream information avoids frame-by-frame picture enhancement and inference of the entire video, reducing latency.

Description

Technical Field

[0001] The present invention relates to the field of video analysis, and in particular to a picture enhancement decoding method for real-time video analysis. Background Art

[0002] With the advancement of computer vision, videos captured by popular cameras now have the opportunity to be analyzed autonomously. Deep neural networks can significantly improve the accuracy of various visual tasks, but they require high computing resources. Due to the insufficient computing resources of commercial cameras, the distributed Video Analytics Pipeline (VAP) transmits videos to powerful servers for inference.

[0003] However, even the most advanced distributed VAPs have difficulty in performing highly accurate analysis of videos. Since most current video analysis methods rely on high-resolution videos, it is difficult to analyze low-quality videos, such as object detection at low resolutions. For example, Faster R-CNN is an inference method based on modern deep neural networks (DNNs), which can only achieve an accuracy of about 56% for 360p videos and about 61% for 540p videos. However, existing surveillance systems inevitably collect low-quality videos. One reason is that existing low-quality cameras can only collect low-resolution frames. For example, the New York City Department of Transportation has made videos collected by 752 traffic cameras public; however, due to the default configuration of the cameras, these videos are transmitted at a very low resolution (240p). Another reason is that current video streaming protocols over-compress or trim videos due to upstream bandwidth limitations. For example, AWStream reduces the resolution of videos from 540p to 360p and the frame rate from 1 to 0.83 in order to save bandwidth. AWStream ultimately saved 66% of bandwidth but reduced accuracy from 61% to 54%.

[0004] To address these challenges, some VAPs attempt to enhance images using image enhancement models such as Super Resolution (SR) and Generative Adversarial Network (GAN) before the images are input into the inference model. This idea was inspired by observations in the computer vision community: running tasks related to object recognition on high-resolution images can improve detection accuracy. However, image enhancement introduces additional latency, resulting in an end-to-end latency of approximately 500ms, which is far from the real-time requirement (less than 30ms).

[0005] Although DNN-aware video enhancement is a promising approach, there is still much room for improvement. First, previous image enhancement mechanisms are largely agnostic about video content and treat each received frame equally. But in practice, it is not necessary to enhance all frames. For example, only frames containing vehicles are valuable for traffic flow analysis; conversely, enhancing frames with only empty streets is worthless and will increase system latency. Therefore, content-agnostic enhancement mechanisms are inevitably suboptimal. Second, although new DNN frameworks are designed to accurately identify important frames, the computational overhead is too high to achieve low latency. Third, decoding all frames for analysis is computationally intensive and time-consuming, and video encoding contains a lot of unexploited but convenient information to capture important frames, such as motion vectors (MVs) and residuals. Summary of the invention

[0006] Purpose of the invention: In view of the problems and shortcomings of the above-mentioned prior art, the purpose of the present invention is to propose a picture enhancement decoding method for real-time video analysis to simultaneously achieve high precision, limited delay and low resource goals.

[0007] Technical solution: To achieve the above-mentioned purpose, the present invention proposes a picture enhancement decoding method for real-time video analysis, comprising the following steps:

[0008] (1) After the video stream is transmitted to the server through the network, the scheduler divides all frames into three categories, and each category executes one of the three pipelines;

[0009] (2) The first pipeline uses the SR model to enhance a set of low-resolution (LR) anchor frames into high-resolution (HR) frames and then inputs them into the inference model; the second pipeline uses the bitstream information (such as frame reference relationship, motion vector and residual) to diffuse the gain of the anchor frames to non-anchor frames and input the non-anchor frames into the inference model;

[0010] (3) The third pipeline uses the bitstream information motion vector to reuse the results of the inference model (such as the bounding box (Bbox) in target detection) on the non-inference frame.

[0011] Furthermore, in step (1), the scheduler must provide an adaptive threshold setting to decide which pipeline to classify the frame into. In order to adaptively set the threshold, the above problem is formulated as a Markov decision process, in which the scheduler makes a threshold setting decision in the system. The MDP is a discrete-time stochastic process defined as a four-tuple<S,A,R,P> ; Where S is the state set, A is the action set, R is the reward set, and P is the transition from state S to state S when action A occurs. ′When processing frames, the scheduler’s goal is to cluster these frames into three pipelines (i.e., A) to maximize the expected long-term reward The state consists of two components: the content features of the keyframe and the difference between the current frame and the last inference frame; the action is to set two thresholds tr for each video segment 1 and tr 2 : The first threshold tr 1 It is used to select anchor frames for SR and diffuse the quality of anchor frames to other frames; the second threshold tr 2 It is used to select the inference frames to be analyzed by the inference DNN; the rest of the frames then reuse the inference results through the frame reference relationship. The reward contains two aspects: the average accuracy of the video segment and the latency required to obtain the inference results of the video segment.

[0012] Furthermore, in the step (2), the anchor frame is first enhanced using the SR model and the output is cached; then for the target block, the system selects a reference block in the cached anchor frame according to the reference index, and amplifies the MV with the same amplification factor as the SR; using the MV, the system diffuses the SR gain of the reference block in the cached frame to the target block; finally, the system amplifies the residual through lightweight interpolation (such as bilinear or bicubic), accumulates the residual to the diffused block to output the HR block, and pastes it to the non-anchor frame.

[0013] Furthermore, in step (3), the inference model is used to infer the inference frame and cache the result, the average value of all MVs residing in each Bbox is calculated, and each Bbox is moved to the current position using the average value to infer the non-inference frame; first, noisy MVs are filtered from static backgrounds and outliers; second, in order to cope with the change in Bbox size caused by the movement of the target, the MV calculation area is expanded to one macroblock (16 pixels) in each direction.

[0014] Beneficial effects: The present invention achieves high precision, limited delay and low resource goals at the same time through a picture enhancement decoding method for real-time video analysis: first, a group of LR anchor frames are enhanced into HR frames, thereby achieving the high precision goal; second, diffusion and reuse share the computational overhead of SR and inference of the entire video, thereby achieving the low delay goal; third, by utilizing the content features of key frames and changes in codec information, the delay and computational cost are reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is the diffusion process diagram of SR gain;

[0016] Figure 2 It is the correlation diagram between the frame and the residual and the difference value and the Bbox change;

[0017] Figure 3 Extract temporal cost maps for frame and residual features;

[0018] Figure 4 is a structural diagram of the present invention;

[0019] Figure 5 It is the relationship diagram between MV and Bbox; DETAILED DESCRIPTION

[0020] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0021] The present invention proposes a picture enhancement decoding method for real-time video analysis, and the specific implementation methods are described as follows:

[0022] Classifying all frames into three categories and ensuring the trade-off between accuracy and latency corresponds to step (1) of the technical solution. The specific implementation method is: using a scheduler. The key to the trade-off between accuracy and latency is to optimally allocate frames to decoding pipelines (i.e., SR, inference, and reuse) at a fine spatial granularity. Different frame decoding and analysis pipelines will result in different degrees of accuracy and latency. For each frame in the SR category, the selected anchor frame is enhanced by the SR model; after that, the scheduler inputs the upsampled frame into the inference DNN (e.g., target detection) for inference. The frames in the inference category all come from Figure 1 The SR frames referenced in are then fed into the inference DNN model for inference. Diffusion is very fast (the time cost is the same as normal frame decoding) because it only additionally includes bilateral interpolation on the residual of each frame compared to normal frame decoding. For those frames in the reuse category, such as object detection, the Bbox of each object in the last (playback order) detection frame is obtained, the average of all MVs residing in the Bbox is calculated, and the average is used to move the Bbox from the previous position to the current position.

[0023] The present invention formulates the adaptive pipeline selection problem to maximize the accuracy under delay constraints. Let F denote a frame set. Given a video containing a frame set F, one of the three pipelines is selected for each frame, as follows.

[0024]

[0025]

[0026] Where f represents a frame in the frame set F, x = {x 1 ,...,F} is the selection set, x f ∈{1,2,3} is used for pipeline selection, Acc is the accuracy of one frame, Latency is the delay of one frame given the selected pipeline, and τ is the delay tolerance of the frame set F.

[0027] Since the search space is very large, it reaches 3 |F| , so it is difficult to find the optimal pipeline selection. The present invention introduces two thresholds tr on the frame difference 1 and tr 2 , to cluster the frames into three pipelines, where tr 1 is the threshold of the first pipeline, tr 2 is the threshold of the second pipeline. The constraints of real-time video analysis (e.g., analysis speed ≥ 30fps) limit the invention from extracting lightweight features to classify frames. The residual and the Laplacian matrix on the frame (i.e., edge features) have a high correlation in inference accuracy (see Figure 2 ). At the same time, performing the Laplacian on the residual saves 34% of the time compared to performing it on the frame (see Figure 3 ). An intuitive reason is that the information about the residual is sparse and de-redundant. The residual preserves the differences between frames while not being too dense to process, so it can effectively classify frames.

[0028] Classifying frames into three pipeline categories is not simple. The optimal threshold combination is different in different videos. In addition, the optimal thresholds for frame feature differences (such as pixel and residual differences) between different video segments in a video vary greatly. In order to set the thresholds adaptively, the above problem is formulated as a Markov decision process (MDP), where the scheduler makes the threshold setting decision in the present invention. MDP is a discrete-time stochastic process defined as a four-tuple<S,A,R,P> Among them, S is the state set, A is the action set, R is the reward set, and P is the transition from state S to state S when action A occurs. ′ When processing frames, the scheduler’s goal is to cluster these frames into three pipelines (i.e., A) to maximize the expected long-term reward The parameters that define the MDP are as follows.

[0029] State: The state consists of two components: the content features of the keyframe and the difference between the current frame and the last inference frame. First, the keyframe (i.e., the first frame of the current video segment) is extracted through the 1×1×1000 fully connected layer of VGG16. Since the dimension of this feature is too large, the system uses principal component analysis to reduce the dimension of the feature to 128 dimensions. Next, the system calculates the inter-frame difference between every two frames of each video segment, that is, the difference in edge features (i.e., the residual of applying the Laplacian operator to each frame). Taking into account the continuity between video segments, the information between video segments is also added, that is, the difference in edge features between the keyframe and the last inference frame in the previous video segment.

[0030] Action: The action is to set two thresholds tr for each video segment 1 and tr 2 The first threshold tr 1 It is used to select anchor frames for SR and diffuse the quality of anchor frames to other frames. The second threshold tr 2 Used to select the inference frames to be analyzed by the inference DNN. The rest of the frames then reuse the inference results through the frame reference relationship.

[0031] Reward: The goal of the present invention is to maximize the accuracy of inference within a tolerable delay time. Therefore, the reward is designed as two aspects, namely the average accuracy of the video segment and the delay required to obtain the inference result of the video segment. The goal of the present invention is to achieve real-time inference, so the current video segment is analyzed before the next video segment arrives (for example, within 1 second for each video segment); and exceeding the specified time will be penalized. The reward r for each video segment t t The definitions are as follows.

[0032]

[0033] Where F represents a frame set, |F| represents the number of frames in frame set F, f represents a frame in frame set F, Acc f is the accuracy of frame f, α 1 and α 2 is a weight coefficient that balances the preference for latency and accuracy. 1 and α 2 The value in the reward is adjusted according to different service preferences and needs. t is the penalty function of video segment t when the delay exceeds the tolerance τ.

[0034]

[0035] Latency t is the delay of video segment t.

[0036] The process of selecting frames in Deep Reinforcement Learning (DRL) is as follows: At each time t, the agent observes the current state s of the interaction environment t , and give an action a according to the strategy t The environment then returns a reward r t As feedback, and according to the transition probability P(s t+1 | t ,) Transition to the next state s t+1 Therefore, the goal of finding the optimal policy is to maximize the expected cumulative discounted return The mathematical problem is: where γ∈[0,1] is the discount factor for future rewards; r k is the reward at each step, and T is the time frame until the video analysis is completed.

[0037] The SR gain is diffused to the non-anchor frame corresponding to step (2) of the technical solution. The specific implementation method is: In order to fully reuse, the system uses the SR model to enhance the anchor frame and cache the output (see Figure 1 ① in ); the system then diffuses the enhanced benefits to non-anchor frames through reference information and cached output (see Figure 1 Because most of the latency of SR occurs in the last few layers, caching and reusing the final output (i.e., high-resolution images) is most effective in achieving low latency.

[0038] Figure 1 The process of diffusing the gain of SR to non-anchor frames of inter-frame coding type is described. Modern video codecs encode or decode frames based on non-overlapping intra-frame or inter-frame coded blocks. The present invention uses the reference index, MV and residual in the codec information to decode the target block. Except for the additional SR, upscaling and image interpolation modules, the process is the same as normal decoding. First, the system selects a reference block in the cached anchor frame according to the reference index. Next, the system upscales the MV with the same upscaling factor as SR (for example, the upscaling factor from 270p to 1080p is 4). Using the MV, the system diffuses the SR gain of the reference block in the cached frame to the target block. Finally, the system upscales the residual by lightweight interpolation (such as bilinear or bicubic), accumulates the residual to the diffused block to output the HR block, and pastes it on the non-anchor frame. For those intra-frame coded blocks without cached reference anchor frames, the system directly decodes and upsamples by interpolation.

[0039] Reusing the inference results to the non-inference frames corresponds to step (3) of the technical solution. The specific implementation method is: the system uses the inference model to infer the inference frame and caches the results; then, the system uses the MV and the cached results to infer the non-inference frame (see Figure 4③ in the figure). MV represents the offset between the target block and the reference block. Here we take the target detection task as an example, the goal is to identify the target (i.e. the location and category of the target) on each frame in the video. Figure 5 It shows that the MV between the last inference frame and the current frame can perfectly match the motion of the target Bbox (i.e., the target detection result).

[0040] The reuse module in the third pipeline obtains the result of the last inference frame ( Figure 4 The dashed line in the figure calculates the average of all MVs residing in each Bbox and uses the average to move each Bbox to its current position.

[0041] The present invention first filters noisy MVs from static backgrounds and outliers. The system filters some MVs whose values ​​are equal to 0 or greater than the mean value in the Bbox to which they belong plus 0.8 times the standard deviation; secondly, in order to cope with the change in Bbox size caused by the movement of the target, the system expands the MV calculation area to one macroblock (16 pixels) in each direction.

[0042] Some previous works have used lightweight MV-based methods to reuse analysis results and speed up inference. However, unlike these works that calculate MVs between consecutive frames, the reuse in the present invention can work in a compressed video space. That is, the reference blocks used to calculate the MV of the target block can be distributed throughout the video (the target block can even refer to future frames under the playback sequence, called backward reference frames), but the inference results should be output in the playback order. To address this mismatch, the present invention maintains a graph to map the encoding order to the playback order and accumulates MVs along the edges.

Claims

1. A picture enhancement decoding method for real-time video analysis, comprising the following steps: (1) After the video stream is transmitted to the server through the network, the scheduler divides all frames into three categories, and each category executes one of the three pipelines; (2) The first pipeline uses a super-resolution model to enhance a set of low-resolution anchor frames into high-resolution frames and then inputs them into the inference model; the second pipeline uses bitstream information to diffuse the gains of the anchor frames to non-anchor frames and inputs the non-anchor frames into the inference model; (3) The third pipeline uses the bitstream information motion vector to reuse the results of the inference model on the non-inference frame; In step (1), the scheduler provides an adaptive threshold setting to determine which pipeline to classify the frame into. In order to adaptively set the threshold, the above problem is formulated as a Markov decision process, in which the scheduler makes a threshold setting decision in the system. The Markov decision process is a discrete-time random process defined as a four-tuple<S,A,R,P> ; Where S is the state set, A is the action set, R is the reward set, and P is the transition from state S to state S when action A occurs. ′ When processing frames, the scheduler’s goal is to cluster these frames into three pipelines to maximize the expected long-term reward The state consists of two components: Content features of key frames and the difference between the current frame and the last inference frame; The action is to set two thresholds tr1 and tr2 for each video segment: the first threshold tr1 is used to select anchor frames for super-resolution and diffuse the quality of anchor frames to other frames; the second threshold tr2 is used to select inference frames analyzed by inference deep neural network; then, the remaining frames reuse the inference results through frame reference relationship; the reward includes two aspects: the average accuracy of the video segment and the delay required to obtain the inference result of the video segment; in the step (2), the anchor frame is first enhanced using the super-resolution model and the output is cached; then for the target block, the system selects the reference block in the cached anchor frame according to the reference index and amplifies the motion vector with the same amplification factor as the super-resolution; Using the motion vector, the system diffuses the super-resolution gain of the reference block in the cached frame to the target block; finally, the system amplifies the residual by lightweight interpolation, accumulates the residual to the diffused block to output a high-resolution block, and pastes it to the non-anchor frame; in the step (3), the inference model is used to infer the inference frame and cache the result, the average of all motion vectors residing in each bounding box is calculated, and the average is used to move each bounding box to the current position to infer the non-inference frame; first, noisy motion vectors are filtered from static backgrounds and outliers; second, in order to cope with the change in bounding box size caused by the movement of the target, the motion vector calculation area is expanded to one macroblock in each direction.

Images