Video hierarchical coding and decoding system and method based on region perception

Abstract

The invention discloses a video hierarchical coding and decoding system and method based on area perception, and the method comprises the steps: an area perception stage: dividing an obtained monitoring video into video segments, processing video frames based on the video segments, and recognizing the area information of subsequent continuous frames through a reference frame processing result fed back by a server, the frame set of each video segment is spatially divided into a plurality of video sub-segments of different levels according to the application of the video; a coding stage: coding the video sub-segments into video sub-streams with different qualities by using different configuration parameters; and a decoding stage: decoding and restoring the video sub-streams into frames corresponding to the video sub-segments, and combining the frames into video frames. According to the invention, the time redundancy and the space redundancy of the video are fully considered, and the bandwidth of the transmitted video is greatly reduced; besides, the realized regional sensing mechanism overcomes the limitation of camera resources, not only meets the real-time processing requirement of video application, but also can be deployed on most cameras without GPU resources.

Description

Technical Field

[0001] The present invention belongs to the technical field of video coding and decoding, and in particular relates to a video hierarchical coding and decoding system and method based on region perception. Background Art

[0002] The video data generated by a large number of surveillance cameras deployed in cities and parks has enormous value and is widely used in public safety, traffic management, fire monitoring and other fields. Due to the limited resources on the camera side, the typical video analysis pipeline is to offload the data on the camera side to the cloud or edge server for processing by a high-computing graphics processor (Graphics Processing Unit, GPU). However, sending all data to the server will consume too much network bandwidth, causing the communication infrastructure to be overwhelmed. In addition, the transmission of high data volumes will also increase the transmission delay of the network and the inference delay of the server, making real-time processing more difficult. Therefore, it is necessary to provide a new video encoding and decoding mechanism that can significantly filter and compress video data to reduce transmission costs and processing delays while ensuring the accuracy of video analysis.

[0003] Currently, the work in the field of video analysis generally filters unchanged or application-irrelevant frames, and only encodes the frames containing the target and uploads them to the server, which is called frame-level filtering method. However, video applications focus on the region of interest (RoI) in the frame rather than the background area. Frame-level filtering does not consider the distribution differences of the region, and uses the same configuration parameters for unified encoding, resulting in additional transmission bandwidth and storage consumption. Although some researchers have proposed using a lightweight DNN (Deep Neural Network) model at the edge node to detect the RoI area, and implement region-aware filtering (i.e., region-level filtering method) based on this. However, running the DNN model requires a lot of computing resources, and how to achieve real-time region awareness on resource-scarce camera nodes is a major challenge.

[0004] Based on the existing H264 / 265 standards, some researchers have proposed Scalable Video Coding (SVC) and Region of Interest (RoI-based Encoding). SVC encoding can encode multiple video streams of different qualities at the same time and transmit them through different links (such as WiFi, 5G, etc.). However, although each video stream of SVC is based on segmented encoding, it does not consider the spatial redundancy caused by regional distribution within the frame. RoI-based encoding can implement different compression ratio encoding for the region of interest and the background area. Existing surveillance cameras generally pre-specify the RoI area. However, in actual scenarios, the movement of the target makes the RoI area change, and the manual specification method cannot adapt to the real-time changes of the video stream; furthermore, RoI-based encoding only uses differentiated quantization parameters in the frame area to control the local encoding quality, without considering the impact of resolution on different areas, and is incomplete in utilizing spatial redundancy.

[0005] In view of this, it is necessary to combine the regional-level filtering mechanism for video analysis and propose a video hierarchical encoding and decoding system and method based on regional awareness. The feedback information of the high-accuracy DNN model (such as Faster R-CNN and YOLO) on the server side is used to realize real-time regional awareness on the resource-constrained edge nodes; then, each video segment is spatially divided into multiple sub-segments of different levels according to the specific video application, and these sub-segments are encoded into sub-streams of different qualities using different configuration parameters; finally, they are decoded and restored to the original video frame on the server side. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a video hierarchical coding and decoding system and method based on region perception, so as to solve the problem that in the prior art, real-time region perception cannot be performed on camera nodes with limited resources. The present invention uses currently commonly used cameras without GPU resources, and can cooperate with servers (edge ​​or cloud servers) to achieve real-time region perception and video hierarchical coding and decoding.

[0007] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A video hierarchical coding and decoding system based on region perception of the present invention comprises: an edge node, a transmission network, and a server;

[0009] The edge node is used to obtain surveillance video data in real time, hierarchically encode the video stream into multiple video sub-streams based on regional perception and set the reference frame, and then send the video sub-stream with the specified reference frame to the server through the transmission network;

[0010] The server is used to receive the hierarchically encoded video substreams and complete the decoding and synthesis of video frames; process the reference frame through the DNN model and feed back the processing results to the edge node through the transmission network.

[0011] Further, the edge node includes: a camera, a first processor, an encoder and a first communication interface;

[0012] Camera, used to collect surveillance video stream data;

[0013] The first processor uses the reference frame processing result sent by the server to perform region tracking and region recognition on each frame in the monitoring video data, and divides the video segment into a plurality of video sub-segments, and specifies the reference frame;

[0014] An encoder, used for encoding the plurality of video sub-segments into video sub-streams of different qualities using different configuration parameters;

[0015] The first communication interface is used to send the video substream of the designated reference frame to the server, and receive the reference frame processing result sent by the server.

[0016] Further, the server includes: a decoder, a second processor, a graphics processor, and a second communication interface;

[0017] A graphics processor, used for processing the reference frame using the DNN model to obtain a reference frame processing result;

[0018] A decoder, used for decoding the video substream into video subsegments;

[0019] The second processor is used to merge the video sub-segments, merge the multiple video sub-segments into a video frame according to the frame sequence; receive the reference frame processing result and encapsulate it;

[0020] The second communication interface is used to receive the video substream sent by the edge node, and send the encapsulated reference frame processing result to the edge node.

[0021] The present invention provides a video hierarchical coding and decoding method based on region perception, based on the above system, and the method includes: a region perception stage, a coding stage and a decoding stage; wherein:

[0022] Regional perception stage: The acquired surveillance video is divided into video segments, and the video frames are processed based on the video segments. The regional information of subsequent continuous frames is identified through the reference frame processing results fed back by the server. The frame set of each video segment is spatially divided into multiple video sub-segments of different levels according to the application of the video.

[0023] Encoding stage: Encode video sub-segments into video sub-streams of different qualities using different configuration parameters;

[0024] Decoding stage: Decode the video substreams to restore them to frames corresponding to the video subsegments and merge them into video frames.

[0025] Furthermore, the specific steps of the region perception stage are:

[0026] (11) Initially, the edge node sends a certain period of surveillance video data to the server, the server creates a background image, and sends the background image to the edge node;

[0027] (12) The edge node divides the surveillance video into video segments, performs a differential operation on each frame in the video segment and the background image to generate a differential image; then extracts the outer contour of the differential image to generate a potential target area;

[0028] (13) The edge node performs RoI marking through region tracking based on the reference frame processing results sent by the server; merges adjacent regions of the frame based on the set relative distance threshold, and generates multiple video sub-segments.

[0029] Furthermore, the step (11) specifically includes:

[0030] The server receives the surveillance video data sent by the edge node, creates a background image based on the background modeling algorithm, uses the DNN model to detect targets in the background image, and replaces the target area with the background area at the corresponding position in the sequence frame; and sends the background image to the edge node;

[0031] The background image is updated periodically or triggered. For the triggered update, the server checks the area of ​​the background image where the target exists based on the reference frame processing result during the processing cycle of each video segment. If the target moves, the background image is updated using the corresponding area of ​​the reference frame.

[0032] Furthermore, the step (12) specifically includes:

[0033] (121) The first processor groups the cached frames into video segments based on a set processing cycle and specifies a reference frame; the set of all frames in the video segment is denoted as F seg ;

[0034] (122) reducing the resolution of each frame in the video segment by downsampling, and performing the same downsampling operation on the background image; performing a difference operation between each frame and the background image to generate a difference image;

[0035] (123) Generate outer contours for all differential images and delete outer contours that are smaller than an area threshold. For stationary targets in video frames, use the target detection results of the background image to generate corresponding areas on the current frame to achieve the generation of all potential target areas in the video segment.

[0036] Furthermore, the step (13) specifically includes:

[0037] (131) Use the DNN model to detect the target in the reference frame and identify the area where the target is located, which is recorded as the processing result R dnn ; Process the result R dnn Send to the corresponding edge node;

[0038] (132) The set of potential target regions generated by the reference frame based on differential operation is R ref , then RoI marking is performed based on the region overlap; suppose any two regions are r1 and r2, and define the region overlap O of r1 and r2 as:

[0039]

[0040] In the formula, area(r1∩r2) represents the area of ​​the intersection, and min(area(r1),area(r2)) represents the minimum area of ​​the two regions;

[0041] R ref The i-th region of R dnn The jth region of The region overlap threshold is T, and r is set i ref The RoI is marked as but The definition is as follows:

[0042]

[0043] R ref Any area according to Perform RoI marking, R ref It is divided into two sub-regions: RoI region and NRoI region; R after RoI marking ref Denoted as R ref* ;

[0044] (133) Based on the labeling result of the reference frame, the regions of all frames of the current video segment are marked with RoI by using the region tracking method; the adjacent regions of each video frame are merged based on the set relative distance threshold, and multiple video sub-segments are generated.

[0045] Furthermore, the step (133) specifically includes:

[0046] (1331) Based on the region overlap, the sequence frames in the video segment are tracked and the R ref* The reference frame is used as the current frame, and the subsequent frames are continuously tracked by the previous frame RoI mark, and the regions of all frames in the current video segment are marked with RoI;

[0047] (1332) Merge adjacent regions with RoI markers in the video frames based on the set relative distance threshold, and then divide all video frames in the current video segment into different video sub-segments based on the RoI markers, including at least the RoI sub-segment and NRoI subsegment

[0048] Furthermore, the specific steps of the encoding stage are:

[0049] (21) The server performs configuration profiling on the surveillance video data for a certain period of time, generates a configuration table regarding resolution, quantization parameters, and accuracy, and sends the configuration table to the edge node;

[0050] (22) The edge node selects configuration parameters for hierarchical encoding, encodes the divided multiple video sub-segments into video sub-streams of different qualities, and sends the encoded video sub-streams to the server.

[0051] Furthermore, the specific steps of step (21) are:

[0052] The server uses the DNN model to perform configuration profiling, generates label data for the acquired surveillance video data for a certain period of time, and then obtains video data of different qualities by combining resolution and quantization parameters. The DNN model is used to perform accuracy detection on the video data to obtain a configuration table;

[0053] Among them, the configuration portrait adopts periodic update or triggered update; for triggered update, the server detects the brightness and data distribution of each video segment. If the detection result exceeds a certain threshold, the edge node is notified to send video data for a certain period of time again to reconfigure the portrait.

[0054] Furthermore, the specific steps of step (22) are:

[0055] (221) The first processor selects an entry from the configuration table based on the target accuracy to encode the video sub-segment: RoI sub-segment It has a great impact on the accuracy of video analysis. It is encoded into RoI sub-stream using high-quality configuration parameters; NRoI sub-segment The accuracy of video analysis is slightly affected, and low-quality configuration parameters are used to encode it into an NRoI substream;

[0056] (222) Based on the reference frame processing result, it is evaluated whether the configuration parameters required for the current video segment meet the accuracy requirements. Under a certain target detection confidence threshold, if the number of targets below the confidence threshold in the reference frame is large, then setting high configuration parameters will improve the encoding quality; conversely, if the confidence of the targets in the reference frame is higher than the confidence threshold, then setting low configuration parameters will reduce the encoding quality.

[0057] Furthermore, the specific steps of the decoding stage are:

[0058] (31) decoding the video substream into corresponding video subsegments, and then increasing the resolution of all low-quality video subsegments to the resolution of the highest-quality video subsegments by upsampling;

[0059] (32) The upsampled video sub-segments are merged according to the frame sequence to generate a video frame set Γ seg , the merging operation is performed by pixel addition or maximum pixel method, as follows:

[0060]

[0061] In the formula, the operator ⊙ represents the operation of adding pixels or finding the maximum value of pixels, and the ellipsis represents the existence of multiple sub-segments. represents the decoded RoI subsegment, Represents the decoded NRoI subsegment.

[0062] Beneficial effects of the present invention:

[0063] The present invention fully considers the temporal redundancy and spatial redundancy of the video, and greatly reduces the transmitted video bandwidth; in addition, the implemented regional perception mechanism overcomes the limitation of camera resources, not only meets the real-time processing requirements of video applications, but can also be deployed on most cameras without GPU resources.

[0064] 1. The present invention uses different configuration parameters at the edge node to encode multiple video sub-segments into sub-streams of different qualities, and then decodes them back to the original video frames at the server end. This encoding mechanism makes full use of the spatial redundancy and temporal redundancy of the video, which can significantly reduce the bandwidth of the transmitted video. Compared with the standard segment-based encoding mechanism, when transmitting high-definition video of traffic monitoring, the segment length is set to 10 frames, and the video hierarchical encoding and decoding mechanism can save up to 40% of bandwidth (the accuracy of video analysis does not drop by more than 3%).

[0065] 2. The present invention uses the processing results of the DNN model and only uses low-level video features at the edge node to spatially divide each video segment into multiple sub-segments of different levels. This not only meets the real-time processing requirements of video applications, but also overcomes the limitations of camera resources and can be deployed on most cameras without GPUs.

[0066] 3. When selecting the encoding parameters of a sub-segment, the present invention can continuously perceive changes in video content, link status, etc. by configuring profiling, regional evaluation mechanisms, etc., and can adaptively adjust the encoding parameters (resolution, quantization parameters, etc.) of the sub-segment, which is particularly suitable for changeable edge heterogeneous environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a system architecture diagram of the present invention;

[0068] Figure 2 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0069] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings. The contents mentioned in the implementation modes are not intended to limit the present invention.

[0070] Reference Figure 1 As shown, a video hierarchical coding and decoding system based on region perception of the present invention includes: an edge node, a transmission network, and a server;

[0071] The edge node is used to obtain surveillance video data in real time, hierarchically encode the video stream into multiple video sub-streams based on regional perception and set the reference frame, and then send the video sub-stream with the specified reference frame to the server through the transmission network;

[0072] Wherein, the edge node includes: a camera, a first processor, an encoder and a first communication interface;

[0073] Camera, used to collect surveillance video stream data;

[0074] The first processor uses the reference frame processing result sent by the server to perform region tracking and region recognition on each frame in the monitoring video data, and divides the video segment into a plurality of video sub-segments, and specifies the reference frame;

[0075] An encoder, used for encoding the plurality of video sub-segments into video sub-streams of different qualities using different configuration parameters;

[0076] The first communication interface is used to send the video substream of the designated reference frame to the server, and receive the reference frame processing result sent by the server.

[0077] The server is used to receive the hierarchically encoded video substreams and complete the decoding and synthesis of video frames; process the reference frame through the DNN model and feed back the processing results to the edge node through the transmission network.

[0078] Wherein, the server includes: a decoder, a second processor, a graphics processing unit (GPU), and a second communication interface;

[0079] A graphics processor, used for processing the reference frame using the DNN model to obtain a reference frame processing result;

[0080] A decoder, used for decoding the video substream into video subsegments;

[0081] The second processor is used to merge the video sub-segments, merge the multiple video sub-segments into a video frame according to the frame sequence; receive the reference frame processing result and encapsulate it;

[0082] The second communication interface is used to receive the video substream sent by the edge node, and send the encapsulated reference frame processing result to the edge node.

[0083] Reference Figure 2 As shown, a video hierarchical coding and decoding method based on region perception of the present invention is based on the above system, and the method includes: a region perception stage, a coding stage and a decoding stage; wherein,

[0084] Regional perception stage: Divide the acquired surveillance video into video segments, process the video frames based on the video segments, and identify the regional information of subsequent continuous frames through the reference frame processing results fed back by the server. According to the application of the video (such as traffic monitoring, pedestrian recognition, etc.), the frame set of each video segment is spatially divided into multiple video sub-segments of different levels; the specific steps are as follows:

[0085] (11) Initially, the edge node sends a certain period of time (e.g., 5 minutes) of surveillance video data to the server, and the server creates a background image and sends the background image to the edge node;

[0086] (12) The edge node divides the surveillance video into video segments, performs a differential operation on each frame in the video segment and the background image to generate a differential image; then extracts the outer contour of the differential image to generate a potential target area;

[0087] (13) The edge node performs RoI marking through region tracking based on the reference frame processing results sent by the server; merges adjacent regions of the frame based on the set relative distance threshold, and generates multiple video sub-segments.

[0088] Wherein, the step (11) specifically comprises:

[0089] The server receives the surveillance video data sent by the edge node, creates a background image based on a background modeling algorithm (such as the median method), uses a DNN model (such as YOLOv5) to detect targets on the background image, and replaces the target area with the background area at the corresponding position in the sequence frame; and sends the background image to the edge node;

[0090] The background image is updated periodically (e.g., every hour) or in a triggered manner. For the triggered update, the server checks the area of ​​the background image where the target exists based on the reference frame processing result during the processing cycle of each video segment. If the target moves, the background image is updated using the corresponding area of ​​the reference frame.

[0091] Wherein, the step (12) specifically comprises:

[0092] (121) The first processor groups the cached frames into video segments based on a set processing period (e.g., 3s) and specifies a reference frame (generally the last frame of the current segment); the set of all frames in the video segment is denoted as F seg ;

[0093] (122) reducing the resolution of each frame in the video segment by downsampling (for example, reducing the resolution from 1920×1080 to 640×360), and performing the same downsampling operation on the background image; performing a difference operation between each frame and the background image to generate a difference image;

[0094] (123) Generate outer contours for all differential images and delete outer contours that are smaller than an area threshold. For stationary targets in video frames, use the target detection results of the background image to generate corresponding areas on the current frame to achieve the generation of all potential target areas in the video segment.

[0095] Wherein, the step (13) specifically comprises:

[0096] (131) Use the DNN model to detect the target in the reference frame and identify the target area information (bounding box, classification, credibility, etc.), which is recorded as the processing result R dnn ; Process the result R dnn Send to the corresponding edge node;

[0097] (132) The set of potential target regions generated by the reference frame based on differential operation is R ref , then RoI marking is performed based on the region overlap; suppose any two regions are r1 and r2, and define the region overlap O of r1 and r2 as:

[0098]

[0099] In the formula, area(r1∩r2) represents the area of ​​the intersection, and min(area(r1),area(r2)) represents the minimum area of ​​the two regions;

[0100] R ref The i-th region of R dnn The jth region of The region overlap threshold is T (usually 0.5), set The RoI is marked as but The definition is as follows:

[0101]

[0102] R ref Any area according to Perform RoI marking, R ref It is divided into two sub-regions: RoI region and NRoI region; R after RoI marking ref Denoted as R ref* ;

[0103] (133) Based on the labeling result of the reference frame, the regions of all frames of the current video segment are marked with RoI by using the region tracking method; the adjacent regions of each video frame are merged based on the set relative distance threshold, and multiple video sub-segments are generated.

[0104] Specifically, the step (133) includes:

[0105] (1331) Based on the region overlap, the sequence frames in the video segment are tracked and the R ref* The reference frame is used as the current frame, and the subsequent frames are continuously tracked by the previous frame RoI mark, and the regions of all frames in the current video segment are marked with RoI;

[0106] (1332) Merge adjacent regions with RoI markers in the video frames based on the set relative distance threshold, and then divide all video frames in the current video segment into different video sub-segments based on the RoI markers, including at least the RoI sub-segment and NRoI subsegment

[0107] For example, for general video applications such as traffic monitoring, F seg Divide into two video sub-segments: RoI sub-segment and NRoI subsegment

[0108] For certain video applications such as pedestrian recognition, F seg Divided into three video sub-segments: pedestrian sub-segment Other target sub-segments and NRoI subsegment

[0109] Encoding stage: Use different configuration parameters to encode video sub-segments into video sub-streams of different qualities; the specific steps are:

[0110] (21) The server performs a configuration profile on the surveillance video data for a certain period of time, generates a configuration table regarding resolution, quantization parameter (QP), and accuracy, and sends the configuration table to the edge node;

[0111] (22) The edge node selects configuration parameters (such as resolution, quantization parameters, etc.) for hierarchical encoding, encodes the divided multiple video sub-segments into video sub-streams of different qualities, and sends the encoded video sub-streams to the server.

[0112] The specific steps of step (21) are as follows:

[0113] The server uses the DNN model to perform configuration profiling, generates label data (ground truth) for the acquired surveillance video data of a certain period of time, and then obtains video data of different qualities by combining resolution and quantization parameters. The DNN model is used to perform accuracy detection on the video data to obtain a configuration table;

[0114] The configuration portrait is updated periodically (such as every 4 hours) or triggered. For triggered updates, the server detects the brightness of each video segment (such as changes in weather conditions) and data distribution (such as too many small targets). If the detection result exceeds a certain threshold, the edge node is notified to send video data for a certain period of time again to reconfigure the portrait.

[0115] The specific steps of step (22) are as follows:

[0116] (221) The first processor selects an entry from the configuration table based on the target accuracy to encode the video sub-segment: RoI sub-segment It has a great impact on the accuracy of video analysis. It is encoded into RoI sub-stream using high-quality configuration parameters; NRoI sub-segment The accuracy of video analysis is slightly affected, and low-quality configuration parameters are used to encode it into an NRoI substream;

[0117] For example, taking the transmission of high-definition video as an example, the RoI sub-segment can be selected while meeting the target accuracy (such as 90%). Use high configuration s1920×1080qp26 (resolution 1920×1080, quantization parameter 26) to encode as RoI sub-stream, for NRoI sub-segment Use the low profile s640×360qp30 (resolution 640×360, quantization parameter 30) to encode as the NRoI substream.

[0118] (222) Based on the reference frame processing results, the configuration parameters required for the current video segment are evaluated to see whether they meet the accuracy requirements. Under a certain target detection confidence threshold (the default value is 0.3), if the number of targets below the confidence threshold in the reference frame is greater, then setting higher configuration parameters will improve the encoding quality. On the contrary, if the confidence of the targets in the reference frame is higher than the confidence threshold, then setting lower configuration parameters will reduce the encoding quality.

[0119] Decoding stage: Decode the video substreams to restore the frames corresponding to the video subsegments and merge them into video frames. The specific steps are as follows:

[0120] (31) decoding the video substream into corresponding video subsegments, and then increasing the resolution of all low-quality video subsegments to the resolution of the highest-quality video subsegments by upsampling;

[0121] For example, taking the reception of high-definition video as an example, after receiving the RoI substream and NRoI substream in step (221), the server first decodes them into RoI subsegments and NRoI subsegment Then put The resolution of each frame is upsampled from 640×360 to 1920×1080, denoted as

[0122] (32) The upsampled video sub-segments are merged according to the frame sequence to generate a video frame set Γ seg , the merging operation is performed by pixel addition or maximum pixel method, as follows:

[0123]

[0124] In the formula, the operator ⊙ represents the operation of adding pixels or finding the maximum value of pixels, and the ellipsis represents the existence of multiple sub-segments. represents the decoded RoI subsegment, Represents the decoded NRoI subsegment.

[0125] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principle of the present invention. These improvements should also be regarded as the protection scope of the present invention.

Claims

1. A video hierarchical coding and decoding system based on region perception, characterized in that: include: Edge nodes, transmission networks, servers; The edge node is used to obtain surveillance video data in real time, hierarchically encode the video stream into multiple video sub-streams based on regional perception and set the reference frame, and then send the video sub-stream with the specified reference frame to the server through the transmission network; The server is used to receive the hierarchically encoded video substreams and complete the decoding and synthesis of video frames; process the reference frame through the DNN model and feed back the processing results to the edge node through the transmission network.

2. The region-aware video hierarchical coding and decoding system according to claim 1, characterized in that: The edge node includes: a camera, a first processor, an encoder and a first communication interface; Camera, used to collect surveillance video stream data; The first processor uses the reference frame processing result sent by the server to perform region tracking and region recognition on each frame in the monitoring video data, and divides the video segment into a plurality of video sub-segments, and specifies the reference frame; An encoder, used for encoding the plurality of video sub-segments into video sub-streams of different qualities using different configuration parameters; A first communication interface, used to send a video substream of a designated reference frame to a server, and receive a reference frame processing result sent by the server; The server includes: a decoder, a second processor, a graphics processor, and a second communication interface; A graphics processor, used for processing the reference frame using the DNN model to obtain a reference frame processing result; A decoder, used for decoding the video substream into video subsegments; The second processor is used to merge the video sub-segments, merge the multiple video sub-segments into a video frame according to the frame sequence; receive the reference frame processing result and encapsulate it; The second communication interface is used to receive the video substream sent by the edge node, and send the encapsulated reference frame processing result to the edge node.

3. A video hierarchical coding and decoding method based on region perception, based on the system described in any one of claims 1-2, characterized in that: Methods include: The regional perception stage, encoding stage, and decoding stage; Regional perception stage: The acquired surveillance video is divided into video segments, video frames are processed based on the video segments, and the regional information of subsequent continuous frames is identified through the reference frame processing results fed back by the server. The frame set of each video segment is spatially divided into multiple video sub-segments of different levels according to the application of the video; Encoding stage: Encode video sub-segments into video sub-streams of different qualities using different configuration parameters; Decoding stage: Decode the video substreams to restore them to frames corresponding to the video subsegments and merge them into video frames.

4. The method according to claim 3, characterized in that The specific steps of the area perception stage are: (11) Initially, the edge node sends a certain period of surveillance video data to the server, the server creates a background image, and sends the background image to the edge node; (12) The edge node divides the surveillance video into video segments, performs a differential operation on each frame in the video segment and the background image, and generates a differential image; Then extract the outer contour of the differential image to generate a potential target area; (13) The edge node performs RoI marking through region tracking based on the reference frame processing results sent by the server; merges adjacent regions of the frame based on the set relative distance threshold, and generates multiple video sub-segments.

5. The method according to claim 4, characterized in that The step (11) specifically comprises: The server receives the surveillance video data sent by the edge node, creates a background image based on the background modeling algorithm, uses the DNN model to detect targets in the background image, and replaces the target area with the background area at the corresponding position in the sequence frame; and sends the background image to the edge node; The background image is updated periodically or in a triggered manner. For the triggered update, the server checks the area where the target exists in the background image based on the reference frame processing result during the processing cycle of each video segment. If the target moves, the background image is updated using the corresponding area of ​​the reference frame. The step (12) specifically comprises: (121) The edge node groups the cached frames into video segments based on the set processing cycle and specifies the reference frame; the set of all frames in the video segment is denoted as F seg ; (122) reducing the resolution of each frame in the video segment by downsampling, and performing the same downsampling operation on the background image; performing a difference operation between each frame and the background image to generate a difference image; (123) Generate outer contours for all differential images, and delete outer contours that are smaller than an area threshold; for stationary targets in the video frame, use the target detection results of the background image to generate corresponding areas on the current frame, so as to achieve the generation of all potential target areas in the video segment; The step (13) specifically comprises: (131) Use the DNN model to detect the target in the reference frame and identify the area where the target is located, which is recorded as the processing result R dnn ; Process the result R dnn Send to the corresponding edge node; (132) The set of potential target regions generated by the reference frame based on differential operation is R ref , then RoI marking is performed based on the region overlap; suppose any two regions are r1 and r2, and define the region overlap O of r1 and r2 as: In the formula, area(r1∩r2) represents the area of ​​the intersection, and min(area(r1),area(r2)) represents the minimum area of ​​the two regions; R ref The i-th region of i ref , R dnn The jth region of The region overlap threshold is T, and r is set i ref The RoI is marked as but The definition is as follows: R ref Any area according to Perform RoI marking, R ref It is divided into two sub-regions: RoI region and NRoI region; R after RoI marking ref Denoted as R ref* ; (133) Based on the labeling result of the reference frame, the regions of all frames of the current video segment are marked with RoI by using the region tracking method; the adjacent regions of each video frame are merged based on the set relative distance threshold, and multiple video sub-segments are generated.

6. The method according to claim 5, characterized in that The step (133) specifically includes: (1331) Based on the region overlap, the sequence frames in the video segment are tracked and the R ref* The reference frame is used as the current frame, and the subsequent frames are continuously tracked by the previous frame RoI mark, and the regions of all frames in the current video segment are marked with RoI; (1332) Merge adjacent regions with RoI markers in the video frames based on the set relative distance threshold, and then divide all video frames in the current video segment into different video sub-segments based on the RoI markers, including at least the RoI sub-segment and NRoI subsegment 7. The method according to claim 4, characterized in that The specific steps of the encoding stage are: (21) The server performs configuration profiling on the surveillance video data for a certain period of time, generates a configuration table regarding resolution, quantization parameters, and accuracy, and sends the configuration table to the edge node; (22) The edge node selects configuration parameters for hierarchical encoding, encodes the divided multiple video sub-segments into video sub-streams of different qualities, and sends the encoded video sub-streams to the server.

8. The method according to claim 7, characterized in that The specific steps of step (21) are: The server uses the DNN model to perform configuration profiling, generates label data for the acquired surveillance video data for a certain period of time, and then obtains video data of different qualities by combining resolution and quantization parameters. The DNN model is used to perform accuracy detection on the video data to obtain a configuration table; Among them, the configuration portrait adopts periodic update or triggered update; for triggered update, the server detects the brightness and data distribution of each video segment. If the detection result exceeds a certain threshold, the edge node is notified to send video data for a certain period of time again to reconfigure the portrait.

9. The method according to claim 7, characterized in that: The specific steps of step (22) are: (221) The first processor selects an entry from the configuration table based on the target accuracy to encode the video sub-segment: RoI sub-segment It has a great impact on the accuracy of video analysis. It is encoded into RoI sub-stream using high-quality configuration parameters; NRoI sub-segment The accuracy of video analysis is slightly affected, and low-quality configuration parameters are used to encode it into an NRoI substream; (222) Based on the reference frame processing result, it is evaluated whether the configuration parameters required for the current video segment meet the accuracy requirements. Under a certain target detection confidence threshold, if the number of targets below the confidence threshold in the reference frame is large, then setting high configuration parameters will improve the encoding quality; conversely, if the confidence of the targets in the reference frame is higher than the confidence threshold, then setting low configuration parameters will reduce the encoding quality.

10. The method according to claim 7, characterized in that The specific steps of the decoding stage are: (31) decoding the video substream into corresponding video subsegments, and then increasing the resolution of all low-quality video subsegments to the resolution of the highest-quality video subsegments by upsampling; (32) The upsampled video sub-segments are merged according to the frame sequence to generate a video frame set Γ seg , the merging operation is performed by pixel addition or maximum pixel method, as follows: In the formula, the operator ⊙ represents the operation of adding pixels or finding the maximum value of pixels, and the ellipsis represents the existence of multiple sub-segments. represents the decoded RoI subsegment, Represents the decoded NRoI subsegment.

Images