Inter-frame coding method, device and medium based on semantic segmentation under Bayesian framework

Through Bayesian framework and semantic segmentation technology, combined with spatial attention mechanism and optical flow algorithm, quantitative parameters are dynamically regulated, and the computational complexity and real-time nature of semantic segmentation in video compression is solved, achieving higher video compression efficiency and vision quality.

CN118509592BActive Publication Date: 2024-11-12NINGBO KANGDA KAINENG MEDICAL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410963993.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-12
Estimated Expiration
2044-07-18

AI Technical Summary

Technical Problem

Semantic segmentation increases the computational complexity in video compression, and it is difficult to ensure real-time performance, affecting encoding efficiency and quality.

Method used

Using the inter-frame encoding method based on semantic segmentation under Bayesian framework, the ESPNet network is strengthened through the spatial attention mechanism, combined with the optical flow algorithm to track the target video feature point, dynamically regulate the quantization parameters based on the spatial and temporal semantic segmentation results, and reduce the calculation Complexity and improve coding efficiency.

Benefits of technology

It effectively reduces the bit rate of video encoding, ensures the visual quality of key areas, achieves higher video compression efficiency and real-time performance, and optimizes resource allocation by dynamically adjusting quantization parameters.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118509592B_ABST
    Figure CN118509592B_ABST
Patent Text Reader

Abstract

The present invention discloses an inter-frame coding method, device and medium based on semantic segmentation under a Bayesian framework, which relates to the field of image processing technology, and includes the steps of: performing semantic segmentation enhancement on an ESPNet network through a spatial attention mechanism, and performing spatial semantic segmentation on a current inter-frame image based on the enhanced ESPNet network; tracking target video feature points based on an optical flow algorithm, and performing temporal semantic segmentation of static background features and dynamic foreground features according to the tracking duration; based on the spatial semantic segmentation results and the temporal semantic segmentation results, performing category probability evaluation on each pixel point through a Bayesian framework to obtain a spatiotemporal semantic segmentation result; dynamically regulating quantization parameters according to the spatiotemporal semantic segmentation results; and encoding the current inter-frame image according to the adjusted quantization parameters. The present invention can achieve higher video compression efficiency while maintaining the same video quality by dynamically controlling quantization parameters based on a spatiotemporal semantic feature map.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an inter-frame coding method based on semantic segmentation under a Bayesian framework. Background Art

[0002] Video semantic segmentation is an important computer vision task that aims to assign each pixel in a video frame to a specific category. This technology has been widely used in autonomous driving, video surveillance, medical image analysis and other fields. Early video semantic segmentation relied on traditional methods based on manual features and image segmentation, which have limited performance when dealing with complex scenes and high-dimensional data. Currently, deep learning methods (such as convolutional neural networks, CNNs) dominate the field of video semantic segmentation. In particular, methods based on fully convolutional networks (FCNs) achieve higher accuracy and robustness through end-to-end training.

[0003] The application of semantic segmentation technology in VVC (Versatile Video Coding) is mainly focused on improving coding efficiency and video quality. VVC is a new generation of video coding standard, which aims to provide about 50% compression efficiency improvement over the existing HEVC (High Efficiency Video Coding) while maintaining the same visual quality. Through semantic segmentation technology, important objects and areas in video frames (such as faces, text, vehicles, etc.) can be identified, and more coding resources can be allocated to these areas, thereby improving the visual quality of these key areas. Semantic segmentation can help identify the background and foreground, so that different quantization parameters can be applied to different areas. Background areas can use higher compression rates, while foreground areas (such as moving objects) use lower compression rates to ensure visual quality. However, the application of semantic segmentation to video compression faces some challenges: First, after semantic segmentation and VVC coding are combined, the computational complexity increases, and further research on efficient implementation methods is needed. Second, ensuring the performance of real-time semantic segmentation and coding is a major challenge in practical applications. Summary of the invention

[0004] In order to reduce the computational complexity of semantic segmentation in VVC coding and balance coding efficiency and quality, the present invention proposes an inter-frame coding method based on semantic segmentation under a Bayesian framework, comprising the steps of:

[0005] S1: The ESPNet network is enhanced for semantic segmentation through the spatial attention mechanism, and the spatial semantic segmentation of the current inter-frame image is performed based on the enhanced ESPNet network;

[0006] S2: Track target video feature points based on the optical flow algorithm, and perform temporal semantic segmentation of static background features and dynamic foreground features according to the tracking duration;

[0007] S3: Based on the spatial semantic segmentation results and the temporal semantic segmentation results, the category probability of each pixel is evaluated through the Bayesian framework to obtain the spatiotemporal semantic segmentation results;

[0008] S4: Dynamically adjust the quantization parameters according to the spatiotemporal semantic segmentation results;

[0009] S5: Encode the current inter-frame image according to the adjusted quantization parameter.

[0010] Furthermore, in the step S1, the semantic segmentation enhancement of the ESPNet network through the spatial attention mechanism specifically includes:

[0011] S11: obtaining an initial feature map by performing multi-scale feature extraction on the current inter-frame image;

[0012] S12: Weighted processing of the initial feature map based on the spatial attention mechanism;

[0013] S13: Obtaining spatial features by fusing the weighted feature map and the current inter-frame image;

[0014] S14: Obtain the spatial semantic segmentation result through convolution operation based on the fused spatial feature map.

[0015] Furthermore, in the step S14, the spatial semantic segmentation result is obtained by the following formula:

[0016]

[0017] In the formula, The coordinates in the feature map are The characteristic points of is the number of categories, K is the number of feature points, For feature points Spatial semantic segmentation results.

[0018] Furthermore, in the step S2, the temporal semantic segmentation is determined by the following formula:

[0019]

[0020] Where t is the coordinate under the optical flow algorithm. The tracking duration of the feature points, For feature points The observed state, Indicates that the pixel belongs to the dynamic or static area attribute, s is the static feature, d is the dynamic foreground feature, and They are the mean and variance of the static background / dynamic foreground feature distribution described by Gaussian distribution, For feature points Temporal semantic segmentation results.

[0021] Furthermore, in the step S3, the spatiotemporal semantic segmentation result is obtained by the following formula:

[0022]

[0023] In the formula, The coordinates are The spatiotemporal semantic segmentation results of feature points, The coordinates are The temporal semantic segmentation results of feature points, The coordinates are The spatial semantic segmentation result of feature points, The coordinates are The spatiotemporal semantic feature values ​​of feature points after spatiotemporal semantic segmentation.

[0024] Furthermore, in the step S4, the adjustment of the quantization parameter is expressed as the following formula:

[0025]

[0026] In the formula, is the adjusted quantization parameter, is the baseline quantitative parameter, is the weight factor of dynamic foreground features and static background features, The coordinates are The spatiotemporal semantic feature values ​​of feature points after spatiotemporal semantic segmentation. is the average semantic feature value of all regions of the target frame image, is the adjustment value.

[0027] The present invention also includes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the inter-frame coding method based on semantic segmentation under the Bayesian framework are implemented.

[0028] Also included is a device for processing data, comprising:

[0029] a memory having a computer program stored thereon;

[0030] A processor is used to execute the computer program in the memory to implement the steps of the inter-frame coding method based on semantic segmentation under the Bayesian framework.

[0031] Compared with the prior art, the present invention has at least the following beneficial effects:

[0032] (1) The inter-frame coding method, device and medium based on semantic segmentation under a Bayesian framework described in the present invention effectively reduce the bit rate of video coding by dynamically controlling the quantization parameters based on the spatiotemporal semantic feature map, which means that higher video compression efficiency can be achieved while maintaining the same video quality;

[0033] (2) Using semantic segmentation technology, we can identify and distinguish the dynamic foreground and static background areas in the video and apply differentiated quantization parameters to different areas, thereby ensuring the visual quality of key areas in the video.

[0034] (3) Based on the spatiotemporal semantic characteristics of the video, the quantization parameters are dynamically adjusted. Lower quantization parameter values ​​are assigned to important areas to retain more details, while higher quantization parameters are used in non-important areas to reduce the encoding bit rate, thereby achieving efficient allocation of resources.

[0035] (4) Although semantic segmentation increases the computational burden, the computational complexity can be effectively controlled through the combination of the spatial attention mechanism and the optical flow algorithm, as well as the feature fusion under the Bayesian framework, ensuring the practicality and real-time performance of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a step diagram of the inter-frame coding method based on semantic segmentation under the Bayesian framework;

[0037] Figure 2 Schematic diagram of the principle of obtaining spatiotemporal semantic segmentation results. DETAILED DESCRIPTION

[0038] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments.

[0039] As a challenging task in the field of computer vision, video semantic segmentation requires the system to understand which object or region each pixel in the video belongs to, so as to achieve a detailed analysis of the video content. It first needs to preprocess the raw video data, which includes extracting key frames, adjusting the frame size to maintain consistency, and applying data enhancement techniques such as random flipping or rotation to improve the generalization ability of the model. Then use deep learning models such as convolutional neural networks (CNNs) to extract features of each frame. In addition, in order to capture the temporal dynamic information in the video, it may be necessary to introduce recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) to ensure that the model can understand the movement trajectory and changes of objects. Finally, each pixel is classified through a fully convolutional network (FCNs) or a similar architecture to determine its semantic category, such as people, cars, sky, etc.

[0040] It is not difficult to see that in the VVC coding standard, the quantization parameter (QP) plays a vital role, which directly affects the compression efficiency of the video and the quality of the decoded image. The VVC standard adopts a QP range of 0 to 63, where a lower QP value corresponds to a higher quality but lower compression ratio video stream, while a higher QP value means higher compression efficiency but may sacrifice some image details.

[0041] The QP control strategy is committed to finding the best balance between compression efficiency and image quality. In the actual encoding process, the QP value can be dynamically adjusted by analyzing the complexity of the video scene, the set target bit rate, and the characteristics of the video content to ensure that the video viewing experience is maintained as much as possible while meeting the bandwidth limit. In order to achieve the goal of reducing the encoding bit rate, the intelligent quantization parameter adjustment strategy is implemented to achieve a reasonable allocation of resources. Figure 1 As shown, the present invention proposes an inter-frame coding method based on semantic segmentation under a Bayesian framework, comprising the steps of:

[0042] S1: The ESPNet network is enhanced for semantic segmentation through the spatial attention mechanism, and the spatial semantic segmentation of the current inter-frame image is performed based on the enhanced ESPNet network;

[0043] S2: Track target video feature points based on the optical flow algorithm, and perform temporal semantic segmentation of static background features and dynamic foreground features according to the tracking duration;

[0044] S3: Based on the spatial semantic segmentation results and the temporal semantic segmentation results, the category probability of each pixel is evaluated through the Bayesian framework to obtain the spatiotemporal semantic segmentation results;

[0045] S4: Dynamically adjust the quantization parameters according to the spatiotemporal semantic segmentation results;

[0046] S5: Encode the current inter-frame image according to the adjusted quantization parameter.

[0047] ESPNet (Dilated Convolutional Pyramid) is known for its lightweight design and efficient multi-scale feature extraction in semantic segmentation networks, and can maintain high segmentation accuracy while reducing the amount of computation. In order to reduce the computational complexity while maintaining the efficient feature extraction and multi-scale fusion capabilities of the semantic segmentation network after VVC encoding is integrated into semantic segmentation. The present invention proposes the introduction of a spatial attention mechanism, which enables the network to dynamically adjust the attention to different spatial positions. The idea is mainly to generate a spatial attention map, identify the key areas in the image, and pay more attention to these areas, thereby improving the accuracy of image segmentation.

[0048] The infrastructure of ESPNet consists of several key components, including the initial convolutional layer, downsampling layer, multiple ESP modules, and the final convolutional layer. These components work together to achieve efficient feature extraction and multi-scale feature fusion, thereby providing high-precision semantic segmentation results while maintaining computational efficiency. In order to solve the problem of increased computational complexity caused by joining the semantic segmentation network, we add a spatial attention module after each ESP module to weight the feature map to achieve more accurate feature extraction and fusion. Feature fusion combines the weighted features with the original features to further enhance the network's ability to handle complex scenes. This improvement scheme not only maintains the efficient computation of ESPNet, but also significantly improves the effect of semantic segmentation by dynamically adjusting the focus on important areas.

[0049] The spatial attention mechanism dynamically adjusts the weight of each position in the feature map by generating a spatial attention map. The spatial attention mechanism is integrated into the ESP module to enhance the effect of multi-scale feature fusion. The specific steps are as follows:

[0050] S11: Obtain the initial feature map F by performing multi-scale feature extraction on the current inter-frame image F ESP ;

[0051] S12: Weighted processing of the initial feature map based on the spatial attention mechanism;

[0052] S13: The weighted feature map F is fused SA And the current inter-frame image F obtains the spatial feature F final =F+F SA ;

[0053] S14: Based on the fused spatial feature map F final The spatial semantic segmentation results are obtained through convolution operations.

[0054] Among them, the spatial semantic segmentation result can be regarded as the prior probability and obtained by the following formula:

[0055]

[0056] In the formula, The coordinates in the feature map are The characteristic points of is the number of categories (foreground, background, etc.), K is the number of feature points, For feature points Spatial semantic segmentation results.

[0057] After extracting the spatial semantic segmentation results of the current inter-frame image, the present invention also performs temporal semantic segmentation on the target video. To this end, the present invention tracks the feature points in the target video through an optical flow algorithm (such as Kanade-Lucas-Tomasi, KLT) and calculates the continuous tracking time of these feature points. Then, according to the duration (Time ofDuration, ToD) of the feature point, it is classified as a static feature s or a dynamic feature d. Based on the observed ToD data of the feature points, a probability distribution model (usually a Gaussian distribution model) is constructed to describe the ToD distribution of static features and dynamic features. Assume that the ToD distribution of the static feature is N (µ s , σ s 2 ), the ToD distribution of dynamic features is N (µ d , σ d 2 ),in and Represent the mean and variance of the distribution respectively. For a given feature point, whose ToD is t, the likelihood probability of it belonging to static feature and dynamic feature can be calculated respectively:

[0058]

[0059] Where t is the coordinate under the optical flow algorithm. The tracking duration of the feature points, For feature points The observed state, Indicates that the pixel belongs to the dynamic or static area attribute. For feature points Temporal semantic segmentation results.

[0060] Then, if Figure 2 As shown in the figure, based on the obtained spatial semantic segmentation results and temporal semantic segmentation results, the two are integrated through Bayesian theory to estimate the probability of the category of each pixel point and obtain the final spatiotemporal semantic segmentation result. The corresponding posterior probability is expressed as the following formula:

[0061]

[0062] In the formula, The coordinates are The spatiotemporal semantic segmentation results of feature points, The coordinates are The temporal semantic segmentation results of feature points, The coordinates are The spatial semantic segmentation result of feature points, The coordinates are The spatiotemporal semantic feature values ​​of feature points after spatiotemporal semantic segmentation.

[0063] Here, we use the mathematical tool of Bayesian theory, whose basic idea is to update the probability judgment of events based on existing information and new evidence. Prior probability represents the initial estimate of an event before there is new evidence (that is, the spatial feature information that can be obtained from the current inter-frame image), while likelihood probability represents the probability of an event occurring under certain known conditions (that is, the temporal feature information that can be obtained from the first few frames that have been encoded). Through Bayesian theorem, the posterior probability after the emergence of new evidence can be calculated, thereby more accurately evaluating the possibility of an event.

[0064] After obtaining the spatiotemporal semantic feature values ​​after spatiotemporal semantic segmentation, we can assign different quantization parameters (QP) to different meaning categories. That is, areas with high importance are assigned lower QP values ​​to retain more details, while areas with low importance can be encoded with higher QP values ​​to reduce the bit rate. The adjustment formula is expressed as follows:

[0065]

[0066] In the formula, is the adjusted quantization parameter, is the baseline quantitative parameter, is the weight factor of dynamic foreground features and static background features, is the average semantic feature value of all regions of the target frame image, is the adjustment value. According to the preset video quality threshold, for the dynamic foreground area , for the static background area .

[0067] Based on the adjusted quantization parameter, the current inter-frame image can be encoded, thereby ensuring better encoding quality and retaining more effective information while ensuring encoding efficiency.

[0068] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the inter-frame coding method based on semantic segmentation under the Bayesian framework are implemented.

[0069] Also included is a device for processing data, comprising:

[0070] a memory having a computer program stored thereon;

[0071] A processor is used to execute the computer program in the memory to implement the steps of the inter-frame coding method based on semantic segmentation under the Bayesian framework.

[0072] In summary, the inter-frame coding method, device and medium based on semantic segmentation under a Bayesian framework described in the present invention effectively reduce the bit rate of video encoding through dynamic control of quantization parameters based on spatiotemporal semantic feature maps, which means that higher video compression efficiency can be achieved while maintaining the same video quality.

[0073] Using semantic segmentation technology, the dynamic foreground and static background areas in the video are identified and distinguished, and different quantization parameters are applied to different areas, thus ensuring the visual quality of key areas in the video;

[0074] Based on the spatiotemporal semantic characteristics of the video, the quantization parameters are dynamically adjusted. Lower quantization parameter values ​​are assigned to important areas to retain more details, while higher quantization parameters are used in non-important areas to reduce the encoding bit rate, thereby achieving effective allocation of resources.

[0075] Although semantic segmentation increases the computational burden, the computational complexity can be effectively controlled through the combined use of the spatial attention mechanism and the optical flow algorithm, as well as the feature fusion under the Bayesian framework, ensuring the practicality and real-time performance of the method.

[0076] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0077] In addition, in the present invention, descriptions such as "first", "second", "one", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0078] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0079] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A Bayesian inter-frame coding method based on semantic segmentation. It is characterized in that Includes steps: S1: The ESPNet network is enhanced for semantic segmentation through the spatial attention mechanism, and the spatial semantic segmentation of the current inter-frame image is performed based on the enhanced ESPNet network; S2: Track target video feature points based on the optical flow algorithm, and perform temporal semantic segmentation of static background features and dynamic foreground features according to the tracking duration; S3: Based on the spatial semantic segmentation results and the temporal semantic segmentation results, the category probability of each pixel is evaluated through the Bayesian framework to obtain the spatiotemporal semantic segmentation results; S4: Dynamically adjust the quantization parameters according to the spatiotemporal semantic segmentation results; S5: Encode the current inter-frame image according to the adjusted quantization parameter; In the S1 step, the semantic segmentation enhancement of the ESPNet network through the spatial attention mechanism specifically includes: S11: obtaining an initial feature map by performing multi-scale feature extraction on the current inter-frame image; S12: Weighted processing of the initial feature map based on the spatial attention mechanism; S13: Obtaining spatial features by fusing the weighted feature map and the current inter-frame image; S14: Obtain the spatial semantic segmentation result through convolution operation based on the fused spatial feature map; In step S4, the adjustment of the quantization parameter is expressed as the following formula: In the formula, is the adjusted quantization parameter, is the baseline quantization parameter, is the weight factor of dynamic foreground features and static background features, The coordinates are The spatiotemporal semantic feature values ​​of feature points after spatiotemporal semantic segmentation. is the average semantic feature value of all regions of the target frame image, is the adjustment value.

2. The inter-frame coding method based on semantic segmentation in a Bayesian framework as claimed in claim 1, It is characterized in that In the step S14, the spatial semantic segmentation result is obtained by the following formula: In the formula, The coordinates in the feature map are The characteristic points of is the number of categories, K is the number of feature points, For feature points Spatial semantic segmentation results.

3. The inter-frame coding method based on semantic segmentation in a Bayesian framework as claimed in claim 1, It is characterized in that In the step S2, the temporal semantic segmentation is determined by the following formula: Where t is the coordinate under the optical flow algorithm. The tracking duration of the feature points, For feature points The observed state, Indicates that the pixel belongs to the dynamic or static area attribute, s is the static feature, d is the dynamic foreground feature, and They are the mean and variance of the static background / dynamic foreground feature distribution described by Gaussian distribution, For feature points Temporal semantic segmentation results.

4. The inter-frame coding method based on semantic segmentation in a Bayesian framework as claimed in claim 1, It is characterized in that In the S3 step, the spatiotemporal semantic segmentation result is obtained by the following formula: In the formula, The coordinates are The spatiotemporal semantic segmentation results of feature points, The coordinates are The temporal semantic segmentation results of feature points, The coordinates are The spatial semantic segmentation results of feature points, The coordinates are The spatiotemporal semantic feature values ​​of feature points after spatiotemporal semantic segmentation.

5. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the steps of the encoding method according to any one of claims 1 to 4 are implemented.

6. A device for processing data, It is characterized in that include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the encoding method according to any one of claims 1 to 4.