A method of coding a video

Abstract

The application discloses a video coding and decoding method, comprising: taking video data frames captured by a fixed camera as input data; sending the input data into a moving target trajectory analysis model to obtain moving objects and trajectory information of the moving objects in the video data; sending the input data into a background sampling model to obtain a background image; and taking the moving objects, the trajectory information of the moving objects and the background image as coding and decoding contents of a coder and decoder. The application can provide efficient video coding based on the fixed camera and improve efficiency and quality.

Description

Technical Field

[0001] The present invention relates to the field of coding and decoding, and in particular to a video coding and decoding method. Background Art

[0002] Over 100 million surveillance cameras are deployed across urban road networks nationwide. As the number of cameras increases, video storage and query processing pose significant challenges. Effective video compression technologies are needed to address storage costs and query efficiency. Traditional video codecs employ predictive coding architectures. They divide video frames into macroblocks and utilize various tools, including intra-frame prediction, inter-frame prediction, and entropy coding, to exploit temporal redundancy in video sequences to encode the residuals between frames for compression.

[0003] While the above methods address the issue of excessive video data to some extent, they also invest a significant amount of unnecessary processing resources in background information, which has limited value for video analysis. Currently, there are also methods using residual coding based on deep neural networks to encode and decode video content. Summary of the Invention

[0004] The purpose of the present invention is to provide a video encoding and decoding method to address the deficiencies of the existing technology, so as to achieve efficient encoding and decoding of video data based on fixed surveillance cameras.

[0005] The object of the present invention is achieved through the following technical solution: A video encoding and decoding method, characterized by comprising:

[0006] The video data frames captured by a fixed camera are used as input data;

[0007] The input data is fed into a moving object trajectory analysis model to obtain moving objects and their trajectory information in the video data;

[0008] Sending the input data into a background sampling model to obtain a background image;

[0009] The moving object and its trajectory information as well as the background image are used as the codec content.

[0010] Further, the background sampling model comprises a background sampling network and a background noise elimination network; the background sampling network takes the background image in the video as clean data and the moving foreground object as noise data to construct a loss function and perform training, the background sampling network is divided into two parts of encoding and decoding, the feature information of the foreground and the background of the image is extracted by using a neural network in the encoding stage, and only the background prediction information is output according to the feature information difference in the decoding stage; the background noise elimination network is also divided into two parts of encoding and decoding, the feature information of the obtained background prediction information and the original video frame is extracted by using a neural network in the encoding stage, the feature information of the background prediction information is replaced into the feature information of the original video frame in different hidden layers in the decoding stage, and finally the background image is obtained.

[0011] Further, the format of the video data frame is a lossless data compression format.

[0012] Further, the input data is sent into the moving object trajectory analysis model to obtain the moving object and the trajectory information thereof in the video data, comprising:

[0013] obtaining the detection box of the moving object in each frame of image: the image is sent into a multi-target detection task composed of a plurality of neural networks to obtain the candidate region of each moving object, and the candidate region is sent to a classification layer and a regression layer to obtain the detection box data of the moving object, including the category, the horizontal and vertical coordinates and the length and width;

[0014] associating the detection box of the moving object in each frame of image: the confidence of the detection box of the moving object in each frame of image is calculated, the detection box with the confidence greater than or equal to a first preset threshold is associated, and the trajectory of the same moving object in different frames is obtained;

[0015] sampling and merging different trajectory information representing the same moving object: the obtained trajectory information is iteratively sampled and screened, and the trajectory pair representing the same moving object in the video is merged.

[0016] Further, the neural network comprises a backbone network for extracting features from the image and generating a feature map; the backbone network comprises a convolution layer, a Leaky ReLU activation function layer and a pooling layer; the convolution layers are connected in sequence, and a Leaky ReLU activation function layer is connected after each convolution layer; in addition, a maximum pooling layer is connected after every two convolution layers, and every two convolution layers are connected by a bypass connection;

[0017] Then, a candidate region is generated by using an RPN architecture, the RPN quickly generates the candidate region by using a plurality of predefined anchor frames on the feature map; wherein, the RPN predicts a target score and a boundary box offset for each anchor frame, so as to determine whether it contains a moving object and the position of the moving object;

[0018] The RPN loss function includes target-orientedness loss and bounding box regression loss, and network parameters are optimized through backpropagation. The generated candidate regions are further filtered through post-processing to reduce overlap and improve detection accuracy; this post-processing includes non-maximum suppression.

[0019] Furthermore, the neural network includes a backbone network composed of Darknet5 to extract image features, and a neck network composed of an SPP-PAN structure to further extract multi-scale features; the detection head of the network is an anchor-free detection head, which realizes moving object detection by predicting category probabilities and bounding box offsets; the target detection task uses a composite loss function including classification loss, positioning loss and confidence loss, and adopts automatic learning rate adjustment, data enhancement and / or label smoothing techniques for training.

[0020] Furthermore, the step of calculating the confidence of the moving object detection frame of each frame image, associating the detection frames whose confidence is greater than or equal to a first preset threshold, and obtaining the trajectory of the same moving object in different frames includes:

[0021] The movement of the same moving object will span multiple image frames. The similarity of the moving objects in adjacent image frames is calculated and the moving objects with a confidence level greater than or equal to a first preset threshold are associated to obtain the motion trajectory information of the moving object. The motion trajectory information includes the video frame ID, object identification ID, and the horizontal and vertical coordinates and length and width of the candidate box.

[0022] Furthermore, the trajectory information obtained by iterative sampling and filtering is merged to form a pair of motion trajectories representing the same moving object in the video, including:

[0023] If the motion trajectory of the same moving object is identified as multiple segments of motion trajectory information representing different object identification IDs, different motion trajectory pairs are associated, and a Beta distribution is initialized for each motion trajectory pair. Using the idea of ​​iterative sampling, Beta sampling is performed on the motion trajectory pairs, and the trajectory pair with the smallest value is selected. The target recognition neural network model is used to calculate the feature values ​​of different candidate boxes in the trajectory pair, and the normalized feature values ​​are used to update the Beta distribution parameters representing the trajectory pair. Finally, trajectory pairs with expected values ​​greater than or equal to a second preset threshold are selected for merging.

[0024] Furthermore, the encoding and decoding content of the codec using the moving object and its trajectory information and the background image includes:

[0025] The moving object and its trajectory information as well as the background image are used as the encoded storage content;

[0026] The decoded image is obtained by combining the background image and the moving object of each frame image according to the trajectory information.

[0027] As can be seen from the above, when using the solution provided by this invention for video data encoding and decoding, it can fully utilize the object detection task performance of the deep neural network model to separate the foreground and background of the input video image, while also providing multi-scale parameters to facilitate more efficient encoding. Furthermore, during decoding, the dynamic foreground and fixed background are aligned based on the stored trajectory information, improving decoding efficiency and, consequently, the quality of the subsequently decoded video data.

[0028] In addition, after completing the network training, the present invention does not require an initial solution and repeated iterative calculations in the design phase. Only one forward prediction and one reverse derivation are needed to extract the performance change trend, which greatly shortens the design time and greatly improves the design quality.

[0029] Of course, it is not necessary to achieve all of the advantages described above simultaneously in order to implement any product or method of the present invention.

[0030] The advantages of the present invention are that it can realize high-efficiency video encoding and decoding based on fixed cameras (such as surveillance cameras) at low cost, thereby improving efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 Schematic diagram of the process of efficient encoding and decoding based on fixed camera video analysis according to an embodiment of the present invention;

[0033] Figure 2 This is a network structure diagram of a target detection task composed of a neural network in an embodiment of the present invention;

[0034] Figure 3 This is a flow chart of a background sampling model based on an input video image in an embodiment of the present invention;

[0035] Figure 4 This is a network structure diagram of a background sampling model based on an input video image in an embodiment of the present invention;

[0036] Figure 5 This is a visualization image of the decoded image quality evaluation index in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of the present invention.

[0038] In order to improve the encoding and decoding efficiency of video data, the embodiment of the present invention provides a video encoding and decoding method, see Figure 1 , including the following steps:

[0039] Step 1 uses the video data frames captured by a fixed camera as input data;

[0040] Step 2: Input the input data into the moving target trajectory analysis model to obtain the moving object and its trajectory information in the video data;

[0041] Step 3: Input the input data into the fixed background sampling model to obtain the background image;

[0042] In step 4, the moving object and its trajectory information as well as the background image are used as the encoding and decoding content of the codec.

[0043] Step 5 uses peak signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM) as measures of decoded image quality; Step 6 calculates the impact of the memory size of the moving object and background image on the decoded image quality, forming a trend of parameter changes on image performance indicators.

[0044] The video data frame described in step 1 is in a lossless data format, which can be a YUV format or a PNG format.

[0045] In one embodiment, step 2 includes:

[0046] Obtain the detection frame of the moving object in each frame: Send the image to the multi-target detection task composed of a multi-layer neural network to obtain the candidate region of each moving object. After sending the candidate region to the classification layer and regression layer, the detection frame data of the moving object is obtained, including the category, horizontal and vertical coordinates, and length and width;

[0047] Associating the detection frames of the moving object in each frame of the image: calculating the confidence of the detection frames of the moving object in each frame of the image, associating the detection frames whose confidence is greater than or equal to a first preset threshold, and obtaining the trajectory of the same moving object in different frames;

[0048] Sampling and merging different trajectory information representing the same moving object: Iteratively sample and filter the obtained trajectory information, and merge the motion trajectory pairs representing the same moving object in the video.

[0049] The moving target trajectory parsing model described in step 2 may include a multi-target detection task composed of a multi-layer neural network and a post-processing trajectory association module.

[0050] The neural network described in step 2 includes a Darknet5 backbone network to effectively extract image features, and a SPP-PAN-based neck network to further extract multi-scale features. The network's detection head is a simplified anchor-free detection head that does not rely on predefined anchor boxes. It detects objects by predicting class probabilities and bounding box offsets. A composite loss function consisting of classification loss, localization loss, and confidence loss is used, and training techniques including automatic learning rate adjustment, data augmentation, and label smoothing are employed to improve the model's generalization and robustness.

[0051] The post-processing trajectory module described in step 2 associates the object candidate frames of adjacent frames, calculates the object confidence represented by the candidate frames, and regards the detection frames with confidence greater than or equal to a first preset threshold (such as 0.6) as adjacent trajectories of the same moving object, thereby generating the motion trajectory of the moving object across the video.

[0052] Associating different track pairs as described in step 2, initialize a Beta distribution for each track pair. Using iterative sampling, perform Beta sampling on the track pairs, select the track pair with the smallest value, and use the target recognition neural network model to calculate the feature values ​​of the different candidate boxes in the track pair. Use the normalized feature values ​​to update the Beta distribution parameters representing the track pair. Repeat this sampling process 10,000 times, updating the Beta distribution parameters each time. Finally, select track pairs with an expected value greater than or equal to a second preset threshold (e.g., 0.8) for merging.

[0053] The actual input data described in step 3 is passed into the fixed background sampling model, and the model output is used to remove the fixed background of the moving object.

[0054] The background sampling model includes a background sampling network and a background noise elimination network; the background sampling network uses the background image in the video as clean data and the moving foreground object as noise data to construct a loss function and perform training. The background sampling network is divided into two parts: encoding and decoding. The encoding stage uses a neural network to extract feature information of the foreground and background of the image, and the decoding stage outputs only background prediction information based on the difference in feature information; the background noise elimination network is also divided into two parts: encoding and decoding. The encoding stage uses a neural network to extract feature information from the obtained background prediction information and the original video frame. The decoding stage replaces the feature information of the original video frame with the feature information of the background prediction information in different hidden layers to finally obtain the background image.

[0055] In one embodiment, step (4) includes:

[0056] The moving object, the trajectory information thereof and the background image are stored as encoding storage contents;

[0057] The background image and the moving object of each frame image are combined according to the trajectory information to obtain a decoded image.

[0058] The peak signal-to-noise ratio (PSNR) and the multi-scale structural similarity (MS-SSIM) are used as the measurement of the quality of the decoded image, and the formula is as follows:

[0059]

[0060]

[0061] wherein I(i,j) represents the pixel value of the pixel point of the original image coordinate (i,j), K(i,j) represents the pixel value of the pixel point of the decoded image coordinate (i,j), and MAX1 represents the number of bits represented by the image pixel, which is set to 255.

[0062] As can be seen from the above, when the scheme provided by the embodiment of the present application is used for video encoding and decoding based on a fixed camera, the moving object and the fixed background can be separated according to different parameters, the trajectory of the moving object in the input image frame is detected using a target detection task, and the background image frame without the moving foreground is output using a self-encoder, thereby helping the designers to more efficiently compress the video.

[0063] In addition, after the training of the network is completed, the initial solution and the repeated iterative calculation are not required in the design link, and only one forward prediction and one backward derivation are required to extract the performance change trend, thereby greatly shortening the design time consumption and greatly improving the design quality.

[0064] The multi-target detection network, the background self-encoder process and the subsequent decoding performance comparison used in the above model will be described in Figure 2 、 Figure 3 、 Figure 4 .

[0065] Referring to Figure 2 , the backbone network of a target detection task composed of a neural network provided by the embodiment of the present application includes a convolutional layer (Convolutional Layer), a Leaky ReLU activation function layer and a pooling layer (pool); the convolutional layers are connected in sequence (concat), and a Leaky ReLU activation function layer is connected after each convolutional layer; in addition, a maximum pooling layer (max-pool) is connected after every two convolutional layers, and every two convolutional layers are connected using a bypass connection (shortcut), that is, a residual mechanism. Multi-layer processing can enable the network to extract high enough dimensional features to generate a feature map.

[0066] As can be seen from the above, the introduction of Leaky ReLU greatly reduces the adverse effects of non-negligible negative inputs on network training in this task, and retains the good performance of ReLU to a certain extent; the introduction of the residual mechanism reduces the problem of gradient vanishing and inability to train when the network is too deep. Both make network training more efficient and stable, and improve network performance.

[0067] The RPN architecture is used to generate candidate regions. The RPN quickly generates candidate regions by using multiple predefined anchor boxes on the feature map. These anchor boxes cover different sizes and scales to ensure that targets of various sizes can be captured. Using the feature map extracted by the pre-trained convolutional neural network, the RPN predicts an object score and bounding box offset for each anchor box to determine whether it contains a moving object and the location of the moving object. The loss function of the RPN includes object loss and bounding box regression loss, and the network parameters are optimized by back propagation. The generated candidate regions are further screened through post-processing steps such as non-maximum suppression (NMS) to reduce overlap and improve detection accuracy. The key advantage of the RPN is that it can share convolutional features with subsequent target detection networks, reducing the amount of computation and speeding up detection. This approach enables the RPN to perform well in processing a large number of candidate regions, providing an effective solution for real-time target detection tasks.

[0068] The network also implements the classification layer and regression layer in two parts, integrating them during prediction. The classification layer classifies moving objects in the candidate area, determining their category, and uses a convolutional neural network to extract features and identify the objects. The regression layer, on the other hand, locates the detected moving objects and determines their position within the image. The regression layer outputs information such as the moving object's center coordinates, width, and height. The network performs trajectory post-processing on the obtained moving objects and their trajectory information, correlating detection boxes representing the same moving object across different image frames to obtain trajectory information representing the same moving object.

[0069] See also Figure 3 , a flowchart of a background sampling model based on an input video image provided by an embodiment of the present invention. This process reconstructs the background by modeling a video frame sequence as a low-dimensional manifold and using an autoencoder to learn this manifold. It also predicts background noise to calculate pixel-level thresholds for each frame, thereby achieving more accurate foreground segmentation. This model does not rely on temporal or motion information and performs well when processing videos captured by a fixed camera. In addition, a robust loss function and a strategy to prevent overfitting are introduced, further enhancing the model's generalization and practicality.

[0070] See also Figure 4, a network structure based on a background sampling model of an input video image provided by an embodiment of the present invention, the structure includes two main networks, namely a background sampling network and a background noise elimination network. The background sampling network uses the background image in the video as clean data and the moving foreground object as noise data to construct a loss function and perform training. The network is divided into two parts: encoding and decoding. The encoding stage uses a neural network to extract feature information of the foreground and background of the image. The decoding stage outputs only background prediction information based on the difference in feature information. The background noise elimination network adopts the idea of ​​a residual network to better adapt to larger dynamic background differences. The encoding stage uses a neural network to extract feature information from the obtained background prediction information and the original video frame and performs self-training. The decoding stage replaces the background prediction image feature information in different hidden layers according to the feature information of the original video frame, and finally obtains the background image.

[0071] See also Figure 5 This is a visualization of a decoded image quality evaluation indicator provided by an embodiment of the present invention. Using a controlled variable method, it is compared with current mainstream video codecs. As can be seen, given the same encoding memory size, the decoded image quality indicator achieved by the present invention is the highest. This demonstrates that efficient codec technology based on fixed-camera video analysis can be well-suited for video compression.

[0072] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A video encoding and decoding method, characterized in that: include: The video data frames captured by a fixed camera are used as input data; The input data is fed into a moving object trajectory analysis model to obtain moving objects and their trajectory information in the video data; Sending the input data into a background sampling model to obtain a background image; The moving object and its trajectory information as well as the background image are used as the codec content of the codec; The step of inputting the input data into a moving target trajectory analysis model to obtain moving objects and their trajectory information in the video data includes: Obtain the detection frame of the moving object in each frame: Send the image to the multi-target detection task composed of a multi-layer neural network to obtain the candidate region of each moving object. After sending the candidate region to the classification layer and regression layer, the detection frame data of the moving object is obtained, including the category, horizontal and vertical coordinates, and length and width; Associating the detection frames of the moving object in each frame of the image: calculating the confidence of the detection frames of the moving object in each frame of the image, associating the detection frames whose confidence is greater than or equal to a first preset threshold, and obtaining the trajectory of the same moving object in different frames; Sampling and merging different trajectory information representing the same moving object: Iteratively sample and filter the obtained trajectory information, and merge the motion trajectory pairs representing the same moving object in the video; The trajectory information obtained by iterative sampling and filtering is merged to represent the motion trajectory pairs of the same moving object in the video, including: If the motion trajectory of the same moving object is identified as multiple segments of motion trajectory information representing different object identification IDs, different motion trajectory pairs are associated, and a Beta distribution is initialized for each motion trajectory pair. Using the idea of ​​iterative sampling, Beta sampling is performed on the motion trajectory pairs, and the trajectory pair with the smallest value is selected. The target recognition neural network model is used to calculate the feature values ​​of different candidate boxes in the trajectory pair, and the normalized feature values ​​are used to update the Beta distribution parameters representing the trajectory pair. Finally, trajectory pairs with expected values ​​greater than or equal to a second preset threshold are selected for merging.

2. The method according to claim 1, characterized in that The background sampling model includes a background sampling network and a background noise elimination network; the background sampling network uses the background image in the video as clean data and the moving foreground object as noise data to construct a loss function and perform training. The background sampling network is divided into two parts: encoding and decoding. The encoding stage uses a neural network to extract feature information of the foreground and background of the image, and the decoding stage outputs only background prediction information based on the difference in feature information; the background noise elimination network is also divided into two parts: encoding and decoding. The encoding stage uses a neural network to extract feature information from the obtained background prediction information and the original video frame. The decoding stage replaces the feature information of the original video frame with the feature information of the background prediction information in different hidden layers to finally obtain the background image.

3. The method according to claim 1, characterized in that The format of the video data frame is a lossless data compression format.

4. The method according to claim 1, wherein The neural network includes a backbone network for extracting features from an image and generating a feature map; the backbone network includes a convolutional layer, a Leaky ReLU activation function layer, and a pooling layer; the convolutional layers are connected in sequence, and a Leaky ReLU activation function layer is connected after each convolutional layer; in addition, a maximum pooling layer is connected after every two convolutional layers, and every two convolutional layers are connected using a bypass connection; Then, the RPN architecture is used to generate candidate regions. The RPN quickly generates candidate regions by using multiple predefined anchor boxes on the feature map. For each anchor box, the RPN predicts an objectness score and bounding box offset to determine whether it contains a moving object and the location of the moving object. The loss function of RPN includes target loss and bounding box regression loss, and the network parameters are optimized through back propagation; the generated candidate regions are further screened through post-processing to reduce overlap and improve detection accuracy; the post-processing includes non-maximum suppression.

5. The method according to claim 1, wherein The neural network includes a backbone network composed of Darknet5 to extract image features, and a neck network composed of an SPP-PAN structure to further extract multi-scale features; the network's detection head is an anchor-free detection head, which realizes moving object detection by predicting category probabilities and bounding box offsets; the target detection task uses a composite loss function including classification loss, positioning loss and confidence loss, and adopts automatic learning rate adjustment, data enhancement and / or label smoothing techniques for training.

6. The method according to claim 1, characterized in that The step of calculating the confidence of the moving object detection frame of each frame image, associating the detection frames whose confidence is greater than or equal to a first preset threshold, and obtaining the trajectory of the same moving object in different frames includes: The movement of the same moving object will span multiple image frames. The similarity of the moving objects in adjacent image frames is calculated and the moving objects with a confidence level greater than or equal to a first preset threshold are associated to obtain the motion trajectory information of the moving object. The motion trajectory information includes the video frame ID, object identification ID, and the horizontal and vertical coordinates and length and width of the candidate box.

7. The method according to claim 1, characterized in that The codec content using the moving object and its trajectory information and the background image as the codec includes: The moving object and its trajectory information as well as the background image are used as the encoded storage content; The decoded image is obtained by combining the background image and the moving object of each frame image according to the trajectory information.

Images