Video coding and decoding method

Abstract

The invention discloses a video encoding and decoding method. The method comprises the following steps: taking a video data frame shot by a fixed camera as input data; sending the input data into a moving target trajectory analysis model to obtain a moving object and trajectory information thereof in the video data; sending the input data into a background sampling model to obtain a background image; and taking the moving object, the track information of the moving object and the background image as coding and decoding contents of the coder and decoder. According to the invention, efficient video coding and decoding based on the fixed camera can be provided, and the efficiency and the quality are improved.

Description

Technical Field

[0001] The present invention relates to the field of coding and decoding, and in particular to a video coding and decoding method. Background Art

[0002] There are more than 100 million surveillance cameras deployed in various urban road networks across the country. With the increase in the number of surveillance cameras, video storage and query processing face huge challenges. Effective video compression technology is needed to solve the problems of storage cost and query efficiency. Traditional video codec technology adopts predictive coding architecture. They divide video frames into macroblocks and use various tools, including intra-frame prediction, inter-frame prediction and entropy coding, to exploit the temporal redundancy in video sequences to encode the residuals between frames for compression.

[0003] Although the above methods solve the problem of excessive video data to a certain extent, they invest a lot of unnecessary processing resources in background information, which has limited value for video analysis. However, at this stage, there are also methods based on deep neural networks that use residual coding to encode and decode video content. Summary of the invention

[0004] The purpose of the present invention is to provide a video encoding and decoding method to address the deficiencies of the prior art, so as to achieve efficient encoding and decoding of video data based on a fixed surveillance camera.

[0005] The objective of the present invention is achieved through the following technical solution: A video encoding and decoding method, characterized in that it includes:

[0006] The video data frames captured by a fixed camera are used as input data;

[0007] Sending the input data into a moving target trajectory analysis model to obtain moving objects and their trajectory information in the video data;

[0008] Sending the input data into a background sampling model to obtain a background image;

[0009] The moving object and its trajectory information as well as the background image are used as the encoding and decoding content of the codec.

[0010] Furthermore, the background sampling model includes a background sampling network and a background noise elimination network; the background sampling network uses the background image in the video as clean data and the moving foreground object as noise data to construct a loss function and perform training, and the background sampling network is divided into two parts: encoding and decoding. The encoding stage uses a neural network to extract feature information of the foreground and background of the image, and the decoding stage outputs only background prediction information based on the difference in feature information; the background noise elimination network is also divided into two parts: encoding and decoding. The encoding stage uses a neural network to extract feature information from the obtained background prediction information and the original video frame, and the decoding stage replaces the feature information of the original video frame with the feature information of the background prediction information in different hidden layers to finally obtain the background image.

[0011] Furthermore, the format of the video data frame is a lossless data compression format.

[0012] Furthermore, the step of sending the input data into a moving target trajectory analysis model to obtain moving objects and their trajectory information in the video data includes:

[0013] Get the detection frame of the moving object in each frame: send the image to the multi-target detection task composed of a multi-layer neural network to obtain the candidate region of each moving object, and then send the candidate region to the classification layer and regression layer to obtain the detection frame data of the moving object, including category, horizontal and vertical coordinates, length and width;

[0014] Associating the detection frames of the moving object in each frame of the image: calculating the confidence of the detection frames of the moving object in each frame of the image, associating the detection frames whose confidence is greater than or equal to a first preset threshold, and obtaining the trajectory of the same moving object in different frames;

[0015] Sampling and merging different trajectory information representing the same moving object: Iteratively sample and filter the obtained trajectory information, and merge the motion trajectory pairs representing the same moving object in the video.

[0016] Furthermore, the neural network includes a backbone network for extracting features from an image and generating a feature map; the backbone network includes a convolution layer, a Leaky ReLU activation function layer, and a pooling layer; the convolution layers are connected in sequence, and a Leaky ReLU activation function layer is connected after each convolution layer; in addition, a maximum pooling layer is connected after every two convolution layers, and every two convolution layers are connected using a bypass connection;

[0017] Then, the RPN architecture is used to generate candidate regions. The RPN quickly generates candidate regions by using multiple predefined anchor boxes on the feature map. The RPN predicts an objectness score and a bounding box offset for each anchor box to determine whether it contains a moving object and the location of the moving object.

[0018] The loss function of RPN includes target loss and bounding box regression loss, and the network parameters are optimized through back propagation. The generated candidate regions are further screened through post-processing to reduce overlap and improve detection accuracy; the post-processing includes non-maximum suppression.

[0019] Furthermore, the neural network includes a backbone network composed of Darknet5 to extract image features, and a neck network composed of an SPP-PAN structure to further extract multi-scale features; the detection head of the network is an Anchor-free detection head, which realizes moving object detection by predicting category probabilities and offsets of bounding boxes; the target detection task uses a composite loss function including classification loss, positioning loss and confidence loss, and adopts automatic learning rate adjustment, data enhancement and / or label smoothing techniques for training.

[0020] Furthermore, the step of calculating the confidence of the moving object detection frame of each frame image, associating the detection frames whose confidence is greater than or equal to a first preset threshold, and obtaining the trajectory of the same moving object in different frames includes:

[0021] The movement of the same moving object will span multiple image frames. The similarity of the moving objects in adjacent image frames is calculated and the moving objects with a confidence level greater than or equal to a first preset threshold are associated to obtain the motion trajectory information of the moving object. The motion trajectory information includes the video frame ID, object identification ID, and the horizontal and vertical coordinates and length and width of the candidate box.

[0022] Furthermore, the trajectory information obtained by iterative sampling and screening is merged to form a pair of motion trajectories representing the same moving object in the video, including:

[0023] If the motion trajectory of the same moving object is identified as multiple segments of motion trajectory information representing different object identification IDs, different motion trajectory pairs are associated, and Beta distribution is initialized for each motion trajectory pair; the idea of ​​iterative sampling is used to Beta sample the motion trajectory pairs, and the trajectory pair with the smallest value is selected. The target recognition neural network model is used to calculate the feature values ​​of different candidate boxes in the trajectory pair, and the normalized feature values ​​are used to update the Beta distribution parameters representing the trajectory pair; finally, the trajectory pairs whose expected values ​​are greater than or equal to the second preset threshold are selected for merging.

[0024] Furthermore, the encoding and decoding content of the codec using the moving object and its trajectory information and the background image includes:

[0025] The moving object and its trajectory information as well as the background image are used as the coded storage content;

[0026] The background image and the moving object of each frame image are combined according to the trajectory information to obtain the decoded image.

[0027] As can be seen from the above, when the scheme provided by the present invention is used for encoding and decoding video data, the target detection task performance of the deep neural network model can be fully utilized to separate the foreground and background of the input video image, and multi-scale parameters are provided to help it encode more efficiently. On the other hand, during decoding, the dynamic foreground and fixed background are fitted according to the saved trajectory information, which improves the efficiency of decoding and improves the quality of the subsequent decoded video data.

[0028] In addition, after completing the network training, the present invention does not require an initial solution and repeated iterative calculations in the design phase. Only one forward prediction and one reverse derivation are needed to extract the performance change trend, which greatly shortens the design time and greatly improves the design quality.

[0029] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention.

[0030] The advantages of the present invention are that high-efficiency video encoding and decoding based on a fixed camera (such as a monitoring camera) can be realized at low cost, thereby improving efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0032] Figure 1 A schematic diagram of a process of efficient encoding and decoding based on fixed camera video analysis according to an embodiment of the present invention;

[0033] Figure 2 A network structure diagram of a target detection task composed of a neural network in an embodiment of the present invention;

[0034] Figure 3 is a flow chart of a background sampling model based on an input video image in an embodiment of the present invention;

[0035] Figure 4 A network structure diagram of a background sampling model based on an input video image in an embodiment of the present invention;

[0036] Figure 5 This is a visualization image of the decoded image quality evaluation index in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field based on this application belong to the scope of protection of the present invention.

[0038] In order to improve the encoding and decoding efficiency of video data, an embodiment of the present invention provides a video encoding and decoding method, see Figure 1 , including the following steps:

[0039] Step 1 uses the video data frame captured by a fixed camera as input data;

[0040] Step 2: Input the input data into the moving target trajectory analysis model to obtain the moving object and its trajectory information in the video data;

[0041] Step 3: Input the input data into the fixed background sampling model to obtain the background image;

[0042] Step 4 uses the moving object and its trajectory information as well as the background image as the encoding and decoding content of the codec.

[0043] Step 5 uses peak signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM) as measures of decoded image quality; Step 6 calculates the impact of the memory size of the moving object and background image on the decoded image quality, forming a trend of parameter changes on image performance indicators.

[0044] The video data frame described in step 1 is in a lossless data format, which may be a YUV format or a PNG format.

[0045] In one embodiment, step 2 includes:

[0046] Get the detection frame of the moving object in each frame: send the image to the multi-target detection task composed of a multi-layer neural network to obtain the candidate region of each moving object, and then send the candidate region to the classification layer and regression layer to obtain the detection frame data of the moving object, including category, horizontal and vertical coordinates, length and width;

[0047] Associating the detection frames of the moving object in each frame of the image: calculating the confidence of the detection frames of the moving object in each frame of the image, associating the detection frames whose confidence is greater than or equal to a first preset threshold, and obtaining the trajectory of the same moving object in different frames;

[0048] Sampling and merging different trajectory information representing the same moving object: Iteratively sample and filter the obtained trajectory information, and merge the motion trajectory pairs representing the same moving object in the video.

[0049] The moving target trajectory analysis model described in step 2 may include a multi-target detection task composed of a multi-layer neural network and a post-processing trajectory association module.

[0050] The neural network described in step 2 includes a backbone network composed of Darknet5 to effectively extract image features, and a neck network composed of an SPP-PAN structure to further extract multi-scale features; the detection head of the network is a simplified Anchor-free detection head that does not rely on predefined anchor boxes. The detection head implements object detection by predicting the category probability and the offset of the bounding box. A composite loss function including classification loss, positioning loss and confidence loss is used, and training is carried out using techniques including automatic learning rate adjustment, data enhancement and label smoothing, which helps to improve the generalization ability and robustness of the model.

[0051] The post-processing trajectory module described in step 2 associates the object candidate frames of adjacent frames, calculates the object confidence represented by the candidate frames, and takes the detection frames with confidence greater than or equal to a first preset threshold (such as 0.6) as adjacent trajectories of the same moving object, thereby generating the motion trajectory of the moving object across the video.

[0052] Associating different trajectory pairs as described in step 2, initialize the Beta distribution for each trajectory pair. Using the idea of ​​iterative sampling, Beta sampling is performed on the trajectory pairs, the trajectory pair with the smallest value is selected, and the feature values ​​of different candidate boxes in the trajectory pair are calculated using the target recognition neural network model. The normalized feature values ​​are used to update the Beta distribution parameters representing the trajectory pair. Repeat the sampling 10,000 times and update the Beta distribution parameters each time. Finally, the trajectory pairs with expected values ​​greater than or equal to the second preset threshold (such as 0.8) are selected for merging.

[0053] The actual input data described in step 3 is passed into the fixed background sampling model, and the model output is used to remove the fixed background of the moving object.

[0054] The background sampling model includes a background sampling network and a background noise elimination network; the background sampling network uses the background image in the video as clean data and the moving foreground object as noise data to construct a loss function and perform training, the background sampling network is divided into two parts, encoding and decoding, the encoding stage uses a neural network to extract feature information of the foreground and background of the image, and the decoding stage outputs only background prediction information based on the difference in feature information; the background noise elimination network is also divided into two parts, encoding and decoding, the encoding stage uses a neural network to extract feature information from the obtained background prediction information and the original video frame, and the decoding stage replaces the feature information of the original video frame with the feature information of the background prediction information in different hidden layers, and finally obtains the background image.

[0055] In one embodiment, step (4) includes:

[0056] The moving object and its trajectory information as well as the background image are used as the coded storage content;

[0057] The background image and the moving object of each frame image are combined according to the trajectory information to obtain the decoded image.

[0058] Peak signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM) are used to measure the quality of the decoded image. The formula is as follows:

[0059]

[0060]

[0061] Where I(i,j) represents the pixel value of the original image coordinate (i,j), and K(i,j) represents the pixel value of the decoded image coordinate (i,j). MAX1 represents the number of bits represented by the image pixel and is set to 255.

[0062] From the above, it can be seen that when the solution provided by the embodiment of the present invention is used to encode and decode a video based on a fixed camera, the moving object and the fixed background can be separated according to different parameters, the target detection task can be used to detect the trajectory of the moving object in the input image frame, and the autoencoder can be used to output the background image frame with the moving foreground removed, thereby helping designers to compress the video more efficiently.

[0063] In addition, after completing the network training, the present invention does not require an initial solution and repeated iterative calculations in the design phase. Only one forward prediction and one reverse derivation are needed to extract the performance change trend, which greatly shortens the design time and greatly improves the design quality.

[0064] The multi-target detection network used in the above model, the background autoencoder process, and the subsequent decoding performance comparison will be Figure 2 , Figure 3 , Figure 4 In the description.

[0065] See also Figure 2 , a backbone network for target detection tasks composed of a neural network provided by an embodiment of the present invention, including a convolutional layer, a leaky ReLU activation function layer, and a pooling layer; the convolutional layers are connected in sequence, and a leaky ReLU activation function layer is connected after each convolutional layer; in addition, a maximum pooling layer (max-pool) is connected after every two convolutional layers, and every two convolutional layers are connected using a bypass connection (shortcut), that is, a residual mechanism. Multi-layer processing enables the network to extract sufficiently high-dimensional features and generate feature maps.

[0066] As can be seen from the above, the introduction of Leaky ReLU greatly reduces the adverse effects of non-negligible negative inputs on network training in this task, and retains the good performance of ReLU to a certain extent; the introduction of the residual mechanism reduces the problem of gradient disappearance and inability to train when the network is too deep. Both make network training more efficient and stable, and improve network performance.

[0067] And the RPN architecture is used to generate candidate regions. RPN quickly generates candidate regions by using multiple predefined anchor boxes on the feature map. These anchor boxes cover different sizes and ratios to ensure that targets of various sizes can be captured. Using the feature map extracted by the pre-trained convolutional neural network, RPN predicts an object score and bounding box offset for each anchor box to determine whether it contains a moving object and the location of the moving object. The loss function of RPN includes object loss and bounding box regression loss, and the network parameters are optimized by back propagation. The generated candidate regions are further screened through post-processing steps such as non-maximum suppression (NMS) to reduce overlap and improve detection accuracy. The key advantage of RPN is that it can share convolutional features with subsequent target detection networks, reducing the amount of computation and speeding up detection. This method enables RPN to perform well when processing a large number of candidate regions, providing an effective solution for real-time target detection tasks.

[0068] The network also divides the classification layer and the regression layer into two parts and integrates them during prediction. The classification layer is used to classify the moving objects in the candidate area, determine which category the moving objects belong to, and use convolutional neural networks to extract features and identify the objects. The regression layer is used to locate the detected moving objects and determine their positions in the image. The regression layer outputs information such as the center coordinates, width, and height of the moving object. The network performs trajectory post-processing on the moving objects and their trajectory information to associate the detection frames representing the same moving objects between different image frames, and obtain trajectory information representing the same moving object.

[0069] See also Figure 3 , is a flowchart of a background sampling model based on an input video image provided by an embodiment of the present invention. The process reconstructs the background by modeling a video frame sequence as a low-dimensional manifold and using an autoencoder to learn this manifold, while predicting the background noise to calculate the pixel-level threshold of each frame, thereby achieving more accurate foreground segmentation. The model does not rely on time or motion information and performs well when processing videos shot by a fixed camera. In addition, a robust loss function and a strategy to prevent overfitting are introduced to further enhance the generalization ability and practicality of the model.

[0070] See also Figure 4, a network structure based on a background sampling model of an input video image provided by an embodiment of the present invention, the structure includes two main networks, namely a background sampling network and a background noise elimination network. The background sampling network uses the background image in the video as clean data and the moving foreground object as noise data to construct a loss function and perform training. The network is divided into two parts: encoding and decoding. The encoding stage uses a neural network to extract feature information of the foreground and background of the image, and the decoding stage outputs only background prediction information based on the difference in feature information. The background noise elimination network adopts the idea of ​​a residual network in order to better adapt to larger dynamic background differences. In the encoding stage, a neural network is used to extract feature information from the obtained background prediction information and the original video frame and perform self-training. In the decoding stage, the feature information of the background prediction image is replaced in different hidden layers according to the feature information of the original video frame, and finally the background image is obtained.

[0071] See also Figure 5 , which is a visualization image of a decoded image quality evaluation index provided by an embodiment of the present invention, is compared with the current mainstream video encoding and decoding content by controlling the variable method. It can be seen that under the same encoding memory size, the image quality index after decoding by the method of the present invention is the highest. This shows that the efficient encoding and decoding technology based on fixed camera video analysis can be well used in video compression methods.

[0072] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A video encoding and decoding method, characterized in that: include: The video data frames captured by a fixed camera are used as input data; Sending the input data into a moving target trajectory analysis model to obtain moving objects and their trajectory information in the video data; Sending the input data into a background sampling model to obtain a background image; The moving object and its trajectory information as well as the background image are used as the encoding and decoding content of the codec.

2. The method according to claim 1, characterized in that The background sampling model includes a background sampling network and a background noise elimination network; the background sampling network uses the background image in the video as clean data and the moving foreground object as noise data to construct a loss function and perform training, the background sampling network is divided into two parts, encoding and decoding, the encoding stage uses a neural network to extract feature information of the foreground and background of the image, and the decoding stage outputs only background prediction information based on the difference in feature information; the background noise elimination network is also divided into two parts, encoding and decoding, the encoding stage uses a neural network to extract feature information from the obtained background prediction information and the original video frame, and the decoding stage replaces the feature information of the original video frame with the feature information of the background prediction information in different hidden layers, and finally obtains the background image.

3. The method according to claim 1, characterized in that The format of the video data frame is a lossless data compression format.

4. The method according to claim 1, characterized in that: The step of sending the input data into a moving target trajectory analysis model to obtain moving objects and their trajectory information in the video data includes: Get the detection frame of the moving object in each frame: send the image to the multi-target detection task composed of a multi-layer neural network to obtain the candidate region of each moving object, and then send the candidate region to the classification layer and regression layer to obtain the detection frame data of the moving object, including category, horizontal and vertical coordinates, length and width; Associating the detection frames of the moving object in each frame of the image: calculating the confidence of the detection frames of the moving object in each frame of the image, associating the detection frames whose confidence is greater than or equal to a first preset threshold, and obtaining the trajectory of the same moving object in different frames; Sampling and merging different trajectory information representing the same moving object: Iteratively sample and filter the obtained trajectory information, and merge the motion trajectory pairs representing the same moving object in the video.

5. The method according to claim 4, characterized in that The neural network includes a backbone network for extracting features from an image and generating a feature map; the backbone network includes a convolution layer, a Leaky ReLU activation function layer, and a pooling layer; the convolution layers are connected in sequence, and a Leaky ReLU activation function layer is connected after each convolution layer; in addition, a maximum pooling layer is connected after every two convolution layers, and every two convolution layers are connected using a bypass connection; Then, the RPN architecture is used to generate candidate regions. The RPN quickly generates candidate regions by using multiple predefined anchor boxes on the feature map. The RPN predicts an objectness score and a bounding box offset for each anchor box to determine whether it contains a moving object and the location of the moving object. The loss function of RPN includes target loss and bounding box regression loss, and the network parameters are optimized by back propagation; the generated candidate regions are further screened by post-processing to reduce overlap and improve detection accuracy; the post-processing includes non-maximum suppression.

6. The method according to claim 4, characterized in that The neural network includes a backbone network composed of Darknet5 to extract image features, and a neck network composed of an SPP-PAN structure to further extract multi-scale features; the detection head of the network is an Anchor-free detection head, which realizes moving object detection by predicting category probabilities and offsets of bounding boxes; the target detection task uses a composite loss function including classification loss, positioning loss and confidence loss, and adopts automatic learning rate adjustment, data enhancement and / or label smoothing techniques for training.

7. The method according to claim 4, characterized in that The step of calculating the confidence of the moving object detection frame of each frame image, associating the detection frames whose confidence is greater than or equal to a first preset threshold, and obtaining the trajectory of the same moving object in different frames includes: The movement of the same moving object will span multiple image frames. The similarity of the moving objects in adjacent image frames is calculated and the moving objects with a confidence level greater than or equal to a first preset threshold are associated to obtain the motion trajectory information of the moving object. The motion trajectory information includes the video frame ID, object identification ID, and the horizontal and vertical coordinates and length and width of the candidate box.

8. The method according to claim 4, characterized in that The trajectory information obtained by iterative sampling and screening is merged to represent the motion trajectory pairs of the same moving object in the video, including: If the motion trajectory of the same moving object is identified as multiple segments of motion trajectory information representing different object identification IDs, different motion trajectory pairs are associated, and Beta distribution is initialized for each motion trajectory pair; the idea of ​​iterative sampling is used to Beta sample the motion trajectory pairs, and the trajectory pair with the smallest value is selected. The target recognition neural network model is used to calculate the feature values ​​of different candidate boxes in the trajectory pair, and the normalized feature values ​​are used to update the Beta distribution parameters representing the trajectory pair; finally, the trajectory pairs whose expected values ​​are greater than or equal to the second preset threshold are selected for merging.

9. The method according to claim 1, characterized in that: The codec content using the moving object and its trajectory information and the background image as the codec includes: The moving object and its trajectory information as well as the background image are used as the coded storage content; The background image and the moving object of each frame image are combined according to the trajectory information to obtain the decoded image.

Images