Ultra-high-definition video stream adaptive coding method based on deep learning visual saliency

Abstract

The invention discloses an ultra-high-definition video stream adaptive coding method based on deep learning visual saliency, and the method comprises the steps: carrying out the five-scale Gaussian filtering processing and image pyramid construction of a video frame, and combining Sobel gradient, Laplacian edge and local binary pattern feature extraction to generate a multi-scale feature map; a pre-training saliency detection network is adopted, and a smooth saliency thermodynamic diagram is generated through processing of a feature adaptation layer, a residual encoder, a self-attention mechanism and a transposed convolution decoder; dividing the video frame into a high region, a middle region and a low region according to the saliency thermodynamic diagram, and establishing a regionalization coding parameter table; performing differentiated prediction modes, motion estimation and quantization strategies on different salient regions; and organizing coded data according to an H.265 / HEVC standard, and embedding the saliency thermodynamic diagram into supplementary enhancement information for transmission. According to the method, the important region concerned by the user can be intelligently identified, a differentiated coding strategy based on content semantics is realized, and the coding efficiency is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video coding and digital media processing, and particularly relates to a super high-definition video stream adaptive coding method based on deep learning visual saliency. BACKGROUND

[0002] With the rapid development of display technology and the continuous improvement of user demand for visual experience, super high-definition video content has become the mainstream trend of the digital media industry. Super high-definition video formats such as 4K and 8K provide an extreme visual experience, but also bring huge data transmission and storage challenges. Super high-definition video data is huge, and the data volume of a single 4K video image can reach tens of megabytes. A complete super high-definition video file often requires hundreds of GB or even TB level of storage space. Under the condition of limited network bandwidth, how to efficiently transmit and encode super high-definition video streams has become a key technical bottleneck restricting its widespread application.

[0003] Traditional video coding methods are mainly based on inter-frame prediction and transform coding technologies, which eliminate temporal and spatial redundancies to achieve data compression. However, these methods face the problem of insufficient coding efficiency when dealing with super high-definition videos, often requiring a sacrifice in video quality to meet transmission bandwidth requirements, making it difficult to find the best balance point between compression rate and visual quality. Especially in complex scenes, traditional coding methods cannot identify the semantic importance of video content and use the same coding strategy for all regions, leading to over-compression of important information and waste of resources for secondary information.

[0004] There are some existing technologies for improving super high-definition video transmission, such as the super high-definition video stream transmission method disclosed in CN202211481886.7. This method generates a sequence of difference images by reorganizing the differences between adjacent frames, and dynamically switches the encoding and decoding mode according to the performance load parameters to optimize transmission efficiency. However, this method still has obvious technical defects. First, this method only reorganizes images based on inter-frame differences, lacks a deep understanding of the semantic importance of video content, and cannot distinguish between areas of real user interest and secondary background areas in the picture. Second, the encoding and decoding strategy of this method mainly relies on system performance parameters such as CPU occupancy, memory occupancy, and other hardware indicators, ignoring the visual characteristics of the video content itself, resulting in unreasonable allocation of coding resources. Third, the interlaced scanning difference comparison method used in this method is too simple and can only capture pixel-level change information, and cannot effectively identify salient regions with semantic meaning. The effect is limited when dealing with complex scenes containing important visual elements such as people, text, and moving targets.

[0005] In addition, most of the existing adaptive coding techniques are based on traditional image processing algorithms, such as edge detection, texture analysis and other bottom-up feature extraction methods. Although these methods can identify local features of the image to some extent, they lack in-depth modeling of human visual attention mechanisms and cannot accurately predict the user's focus of attention. The human visual system has the characteristic of selective attention, which automatically focuses attention on the most salient and important areas in the scene, while traditional coding methods cannot simulate this complex cognitive process.

[0006] With the rapid development of deep learning technology, neural network-based visual saliency detection methods have made significant progress in the field of computer vision. These methods can learn the rules of human visual attention through large-scale data training and accurately identify salient regions in images. However, existing video coding techniques have not fully utilized this advanced technology and still remain at the stage of traditional pixel-level feature-based coding optimization, missing the opportunity to achieve intelligent coding through deep learning saliency detection.

[0007] Therefore, there is an urgent need for a super high-definition video adaptive coding method that combines deep learning visual saliency detection technology, which can intelligently identify important areas in video content and implement differentiated coding strategies based on content semantics, significantly improving coding efficiency while ensuring the quality of important visual information, providing technical support for efficient transmission and storage of super high-definition video. SUMMARY

[0008] Therefore, there is an urgent need for a super high-definition video adaptive coding method that combines deep learning visual saliency detection technology, which can intelligently identify important areas in video content and implement differentiated coding strategies based on content semantics, significantly improving coding efficiency while ensuring the quality of important visual information, providing technical support for efficient transmission and storage of super high-definition video.

[0009] To achieve the above purpose, the present application provides a super high-definition video stream adaptive coding method based on deep learning visual saliency, which includes the following steps: S1: Through five-scale Gaussian filtering and image pyramid construction of super high-definition video frames, combined with Sobel gradient, Laplacian edge and local binary pattern feature extraction, a multi-scale feature map is generated; S2: Using a pre-trained saliency detection network, through the cascade processing of feature adaptation layer, residual encoder, self-attention mechanism and transposed convolution decoder, the multi-scale feature map is converted into a saliency heat map, and a smoothed saliency heat map is obtained by using Gaussian filtering smoothing processing; S3: adaptively divide the video frame into high, medium and low three levels of saliency region according to the smoothed saliency heat map, and establish a regional coding parameter table through connected domain analysis and average saliency value calculation; S4: based on the regional coding parameter table, different prediction modes, motion estimation and quantization strategies are performed on different saliency regions, combined with the standard video coding process to realize the rate-distortion optimization of saliency perception, and output the compressed video data stream; S5: organize the compressed video data stream according to the H.265 / HEVC standard format, record the coding configuration information through the sequence parameter set and the image parameter set, and embed the saliency heat map in the supplementary enhancement information after compression and transmit it with the bit stream.

[0010] As a further improved method of the application: Optionally, in the S1 step, the multi-scale feature map is generated by performing Gaussian filtering and image pyramid construction on the ultra-high definition video frame at five scales, and combining Sobel gradient, Laplacian edge and local binary pattern for feature extraction, including: obtaining a current video frame in YUV420 format in a color space in an ultra-high definition video stream ; performing Gaussian filtering on the Y component of the video frame at five different scales; performing down-sampling processing on the filtering result of each scale, and the down-sampling factors are 1, 2, 4, 8 and 16 respectively, to obtain five image pyramid layers with different resolutions as follows: 、 、 、 、 ; calculating gradient features of each layer, wherein the gradient features are calculated by Sobel operator to calculate the horizontal gradient and the vertical gradient of each resolution image in the image pyramid layer, so as to calculate the corresponding gradient amplitude; calculating the second derivative features of each layer, wherein the second derivative features are calculated by Laplacian operator to perform edge detection on each resolution image in the image pyramid layer; extracting local binary pattern features, wherein the local binary pattern features calculate the LBP histogram for each 8x8 pixel block of each resolution image in the image pyramid layer; performing the same multi-scale Gaussian filtering on the U and V chroma components of the video frame as the Y component, and down-sampling to the same resolution as the corresponding layer of the image pyramid layer to obtain five U component image pyramid layers with different resolutions and five V component image pyramid layers with different resolutions; Constructing a multi-scale feature descriptor, which extracts gradient features, second derivative features, local binary pattern features, U components and V components for each spatial location at each resolution; Z-score standardization is performed on the multi-scale feature descriptor, the mean and standard deviation of each feature dimension are calculated, and the feature values are converted to a zero-mean unit-variance distribution; The normalized multi-scale feature descriptors at different resolutions are up-sampled to the same resolution by bilinear interpolation, concatenated by feature dimensions, and smoothed by 3x3 mean filtering to obtain a multi-scale feature map.

[0011] Optionally, in the S2 step, a pre-trained saliency detection network is used, and the multi-scale feature map is converted into a saliency heat map through the cascade processing of a feature adaptation layer, a residual encoder, a self-attention mechanism and a transposed convolution decoder, and a smoothed saliency heat map is obtained using Gaussian filtering, including: The multi-scale feature map is input into a pre-trained saliency detection neural network; the saliency detection neural network includes a feature adaptation layer, a residual encoder, a self-attention layer and a transposed convolution decoder; The feature adaptation layer uses three 3x3 convolution layers to map the multi-scale feature map to 256 dimensions; The encoder uses four residual block structures to process the output of the feature adaptation layer, and each residual block contains two 3x3 convolution layers and a skip connection; The self-attention mechanism layer generates query matrix Q, key matrix K and value matrix V through three 1x1 convolution layers and calculates the self-attention weight matrix: wherein, represents the dimension of the key vector; represents the matrix transpose; is a normalized exponential function; The transposed convolution decoder part uses four transposed convolution layers to gradually up-sample the self-attention weight matrix; The output is limited to the range of 0 to 1 through a sigmoid activation function to obtain a pixel-level saliency heat map; The saliency heat map is smoothed by Gaussian filtering to obtain a smoothed saliency heat map.

[0012] This step can automatically learn and locate the key target area in the image through the saliency detection mechanism, focusing computing resources and attention on the most valuable features, thereby significantly improving the efficiency and accuracy of subsequent processing steps. This intelligent region selection capability enables the system to better focus on truly important visual information and avoid being disturbed by irrelevant background information.

[0013] The symmetric design of the encoder and the decoder in this step combines the self-attention mechanism to realize the deep fusion and accurate reconstruction of multi-level features. The encoder gradually extracts the deep semantic features of the image through four residual block structures. The jump connection in each residual block effectively alleviates the gradient disappearance problem in the training of deep networks, ensuring the integrity and stability of feature extraction. The self-attention mechanism calculates the query, key, and value matrices, enabling the network to establish long-distance dependencies between different regions of the image and capture global context information, which is of great significance for understanding the spatial relationships and semantic associations between targets in complex scenes. The decoder gradually restores the abstract high-level features to the original image resolution through the transpose convolution layer, recovering spatial details while maintaining semantic information, ensuring that the saliency heat map can be accurately located at the pixel level.

[0014] Optionally, in the S3 step, the video frame is adaptively divided into high, medium, and low three-level saliency regions according to the smoothed saliency heat map, and through connected component analysis and average saliency value calculation, a regional coding parameter table is established, including: Based on the smoothed saliency heat map, the video frame is divided into regions of different saliency levels, and the high saliency threshold is set to 0.7, the low saliency threshold is set to 0.3, and the region division rule is: when the pixel saliency value is ≥0.7, it is classified as a high saliency region; when 0.3≤pixel saliency value<0.7, it is classified as a medium saliency region; when the pixel saliency value is <0.3, it is classified as a low saliency region. The connected component analysis is performed on each region of different saliency levels, and the 8-connected mode is used to identify continuous regions, and scattered regions with an area less than 100 pixels are filtered out. The quantization parameters of each region of different saliency levels are calculated, and the region quantization parameter calculation formula is: wherein, represents the quantization parameter of the current calculated saliency level region, represents the quantization parameter adjustment range, represents the average saliency value of the current calculated saliency level region; represents the basic quantization parameter; The regional coding parameter table is constructed, which records the boundary coordinates, quantization parameters, and average saliency values of each region.

[0015] ​​The step realizes intelligent allocation and optimized utilization of video coding resources through adaptive region division and differentiated coding parameter setting. Traditional video coding methods usually use uniform coding parameters for the entire video frame, which cannot effectively distinguish the importance difference of different regions in the image, resulting in unreasonable allocation of coding resources. The step divides the video frame into high, medium and low three levels of saliency regions through saliency heat map, which can accurately identify the important region and the relatively secondary background region that the audience pays most attention to.

[0016] The step identifies truly continuous saliency regions through eight-connected region analysis and filters out scattered regions with an area less than one hundred pixels, ensuring the coherence and rationality of the region division result. This processing method makes each saliency region have sufficient spatial continuity, which is convenient for subsequent coding processing, and at the same time avoids the problem of coding efficiency decline caused by excessive subdivision.

[0017] Optionally, the S4 step performs differentiated prediction mode, motion estimation and quantization strategy for different saliency regions based on the regionalized coding parameters, combines the standard video coding process to realize saliency-aware rate-distortion optimization, and outputs the compressed video data stream, including: According to the quantization parameter in the regionalized coding parameter table Differential coding processing is performed on each region, fine quantization processing is performed on high saliency regions, standard quantization processing is performed on medium saliency regions, and rough quantization processing is performed on low saliency regions; Based on the saliency level of the region, a prediction mode selection strategy is performed, the intra-frame prediction mode is preferentially selected for high saliency regions, including vertical prediction, horizontal prediction, DC prediction and planar prediction mode, and the prediction direction with the smallest prediction error is selected by calculating the prediction error of each mode to obtain the best coding effect; The intra-frame and inter-frame prediction modes are comprehensively evaluated for the medium saliency region, the prediction errors of intra-frame prediction and motion compensation prediction are calculated respectively, and the prediction mode with smaller error is selected for subsequent coding processing; For low saliency regions, a simplified coding strategy is used to directly copy the pixel values of the reference frame at the same position using the skip mode, and motion estimation is only enabled to find a more matched reference position when the copying error exceeds the set threshold; According to the importance of the region, a hierarchical motion estimation processing is performed, the high saliency region is searched within ±32 pixels with a search accuracy of 1 / 4 pixel, the medium saliency region is searched within ±16 pixels with a search accuracy of 1 / 2 pixel, and the low saliency region is searched within ±8 pixels with a search accuracy limited to integer pixels; After the prediction mode selection and motion estimation are completed, the difference between the actual pixel value and the predicted pixel value is subjected to 8x8 discrete cosine transform, and the residual information in the spatial domain is converted into transform coefficient representation in the frequency domain; Quantize the transform coefficients by using the corresponding quantization parameter of each region; rearrange the quantized transform coefficients in Z-scan order, and then perform entropy coding compression by using run-length coding and Huffman coding to further reduce data redundancy, and finally output the compressed video data stream.

[0018] Optionally, the compressed video data stream is organized in the format of H.265 / HEVC standard in the S5 step, the encoding configuration information is recorded by sequence parameter set and image parameter set, and the saliency heat map is compressed and embedded in the supplemental enhancement information and transmitted together with the bit stream, including: The compressed video data stream is organized in the format of H.265 / HEVC standard, including sequence parameter set, image parameter set and network abstraction layer unit; The video resolution, frame rate and color format are recorded in the sequence parameter set; The quantization parameter and entropy coding type are recorded in the image parameter set; The compressed video data stream is encapsulated into a network abstraction layer unit, and a start code 0x000001 is added for separation; The saliency heat map is down-sampled to a resolution of 240x135, compressed by JPEG and embedded in the supplemental enhancement information for transmission; The compressed video data stream is byte-aligned, and an H.265 format video bit stream file containing saliency perception coding information is output.

[0019] Compared with the prior art, the present application has at least the following beneficial effects: The present application realizes intelligent optimization of video coding by deep learning visual saliency detection technology, and has significant technical advantages compared with traditional coding methods. The present application uses a pre-trained saliency detection neural network to analyze video frames, which can accurately simulate the human visual attention mechanism and identify important areas that users are really interested in, such as human faces, text information, moving targets and other content with semantic meaning. This saliency detection method based on deep learning has stronger semantic understanding ability and higher detection accuracy than traditional edge detection, texture analysis and other bottom feature extraction techniques, and can accurately distinguish foreground objects and background areas in complex scenes, providing reliable guidance information for subsequent differentiated coding strategies, and fundamentally solving the technical problem that traditional methods cannot understand the semantic importance of video content.

[0020] The application innovatively proposes a three-level region division strategy based on saliency levels and a differential quantization parameter allocation mechanism, and realizes intelligent allocation and optimized utilization of coding resources. By dividing a video frame into three saliency level regions of high, medium and low, and allocating different quantization parameters, prediction modes and motion estimation accuracies according to the region importance, the application can preferentially allocate limited coding bit resources to important regions of user's most concern, and ensure high-quality preservation of key visual information.

[0021] The application has good standard compatibility and practicability, and can be seamlessly integrated into existing video coding standards and transmission systems. The application strictly follows the H.265 / HEVC coding standard, and the generated video bit stream is fully compatible with the standard decoder, without any modification to the existing playback equipment and software. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A flowchart of a super high definition video stream adaptive coding method based on deep learning visual saliency for an embodiment of the application is shown in the figure. Figure 2 A schematic diagram of an original video frame is shown in the figure. Figure 3 A schematic diagram of a multi-scale feature map is shown in the figure. Figure 4 A schematic diagram of a saliency heat map is shown in the figure. Figure 5 An example of a smoothed saliency heat map is shown in the figure. DETAILED DESCRIPTION

[0023] The application will be further described below with reference to the accompanying drawings, but the application is not limited in any way by the drawings, and any transformation or replacement based on the teaching of the application is within the protection scope of the application.

[0024] Embodiment 1: A super high definition video stream adaptive coding method based on deep learning visual saliency, as shown in the figure, includes the following steps: Figure 1 S1: Through five-scale Gaussian filtering processing and image pyramid construction on the super high definition video frame, combined with Sobel gradient, Laplace edge and local binary pattern for feature extraction, a multi-scale feature map is generated: Obtain a current video frame in YUV420 format in a color space in a super high definition video stream , as shown in the figure. Perform five different scale Gaussian filtering processing on the Y component of the video frame Figure 2 , as shown in the figure. , as shown in the figure. , as shown in the figure. , , , , , the corresponding filter kernel sizes are 3x3, 5x5, 9x9, 17x17, 33x33, and the processing results are shown in FIG. 4; Figure 3 Down-sampling processing is performed on the filtering results of each scale, and the down-sampling factors are 1, 2, 4, 8, and 16, respectively, to obtain five image pyramid layers with different resolutions: , , , , For 8K and above resolution videos, to avoid memory overflow, an adaptive down-sampling strategy can be used. When the input video resolution exceeds 7680x4320, the down-sampling factor is adjusted to 2, 4, 8, 16, and 32; The gradient features of each layer are calculated, which are calculated by Sobel operator to calculate the horizontal gradient and the vertical gradient of each resolution image in the image pyramid layer, so as to calculate the corresponding gradient amplitude. In this embodiment, the horizontal Sobel kernel is [-1, 0, 1; -2, 0, 2; -1, 0, 1], and the vertical Sobel kernel is [-1, -2, -1; 0, 0, 0; 1, 2, 1]; The second derivative features of each layer are calculated, which are calculated by Laplace operator to detect edges of each resolution image in the image pyramid layer. In this embodiment, the Laplace kernel is [0, -1, 0; -1, 4, -1; 0, -1, 0]; Local binary pattern features are extracted, which calculate LBP histograms for each 8x8 pixel block of each resolution image in the image pyramid layer. In this embodiment, an 8-neighbor LBP operator is used, and the radius is set to 1 pixel; The U and V chroma components of the video frame are subjected to the same multi-scale Gaussian filtering processing as the Y component, and are down-sampled to the same resolution as the corresponding layer of the image pyramid layer, to obtain five U component image pyramid layers with different resolutions and five V component image pyramid layers with different resolutions; Multi-scale feature descriptors are constructed, which are the gradient features, second derivative features, local binary pattern features, U components, and V components extracted for each spatial position of each resolution; The multi-scale feature descriptors are subjected to Z-score standardization processing, the mean and standard deviation of each feature are calculated, and the feature values are converted to zero-mean unit variance distribution; ​The multi-scale feature descriptors of different resolutions after standardization are uniformly up-sampled to the same resolution by bilinear interpolation, spliced according to the feature dimension, and smoothed by 3*3 mean filtering to obtain a multi-scale feature map.

[0025] S2: A pre-trained saliency detection network is used to convert the multi-scale feature map into a saliency heat map through a cascade of a feature adaptation layer, a residual encoder, a self-attention mechanism and a transposed convolution decoder, and a smoothed saliency heat map is obtained by using Gaussian filtering smoothing: The multi-scale feature map is input into a pre-trained saliency detection neural network, which is trained based on 10,000 labeled saliency images in this embodiment. The training data includes 5,000 images from the SALICON dataset, 1,003 images from the MIT1003 dataset, 850 images from the PASCAL-S dataset and 3,147 labeled super high-definition video frame data collected by the present application. The saliency detection neural network includes a feature adaptation layer, a residual encoder, a self-attention layer and a transposed convolution decoder. The feature adaptation layer uses three 3*3 convolution layers to map the multi-scale feature map to 256 dimensions. In this embodiment, the first convolution layer has an input channel number of 60 and an output channel number of 128; the second convolution layer has an input and output channel number of 128 and 192, respectively; and the third convolution layer has an input and output channel number of 192 and 256, respectively. The residual encoder uses four residual block structures to process the output of the feature adaptation layer. Each residual block includes two 3*3 convolution layers and a skip connection. In this embodiment, the first residual block has an input and output channel number of 256 and 512, respectively; the second and third residual blocks have an input and output channel number of 512; and the fourth residual block has an input and output channel number of 512 and 256, respectively. The self-attention mechanism layer generates a query matrix Q, a key matrix K and a value matrix V through three 1*1 convolution layers, respectively, and calculates a self-attention weight matrix: wherein, represents the dimension of the key vector, which is set to 256 in this embodiment; represents matrix transposition; is a normalization exponential function. To solve the problem of memory explosion caused by super high-resolution feature maps, a linear self-attention mechanism can be used instead of the standard self-attention, and an ELU activation function is used. ​The transposed convolutional decoder uses four transposed convolutional layers to progressively upsample the self-attention weight matrix. In this embodiment, the first transposed convolutional kernel is 4×4 with a stride of 2 and padding of 1, and has 256 input channels and 128 output channels. The subsequent three transposed convolutional layers reduce the number of channels to 64, 32 and 1 respectively. By using the sigmoid activation function to restrict the output to the range of 0 to 1, a pixel-level saliency heatmap is obtained, such as... Figure 4 As shown; The saliency heatmap is smoothed using Gaussian filtering. In this embodiment, the filter kernel size is 5×5 and the standard deviation is 1.5, resulting in a smoothed saliency heatmap, as shown below. Figure 5 As shown.

[0026] S3: Based on the smoothed saliency heatmap, the video frames are adaptively divided into high, medium, and low saliency regions. Through connected component analysis and average saliency value calculation, a regional coding parameter table is established. Based on the smoothed saliency heatmap, video frames are divided into regions with different saliency levels, and a high saliency threshold is set. The significance threshold is 0.7. The significance value is 0.3, and the region division rules are as follows: when the pixel significance value is ≥0.7, it is classified as a high significance region; when 0.3≤pixel significance value<0.7, it is classified as a medium significance region; when the pixel significance value<0.3, it is classified as a low significance region. Connectivity analysis was performed on regions with different saliency levels. 8-connectivity was used to identify continuous regions and filter out scattered regions with an area of ​​less than 100 pixels. Calculate the quantization parameters for regions with different significance levels, including the basic quantization parameters. The value is set to 12, and the formula for calculating the regional quantization parameter is as follows: in, Quantization parameters representing the significance level region, This indicates that the quantization parameter adjustment range is set to 20. This represents the average significance value of the significance level region; for extremely low bitrate coding requirements, a non-linear quantization parameter mapping strategy can be adopted, using an exponential function. ,in It is a natural constant; A regional coding parameter table is constructed, which records the boundary coordinates, quantization parameters, and average significance value of each region.

[0027] S4: Based on the regional coding parameters, different prediction modes, motion estimation and quantization strategies are performed on different saliency regions, combined with the standard video coding process to realize saliency-aware rate-distortion optimization, and output compressed video data stream: According to the quantization parameter in the regional coding parameter table Each region is processed differently, with fine quantization for high saliency regions to preserve more image details, standard quantization for medium saliency regions, and coarse quantization for low saliency regions to achieve high compression ratio; Based on the saliency level of each region, the prediction mode selection strategy is executed, and for high saliency regions, the intra prediction mode is selected, including vertical prediction, horizontal prediction, DC prediction and planar prediction mode, and the prediction direction with the smallest prediction error is selected by calculating the prediction error of each mode to obtain the best coding effect; For medium saliency regions, a comprehensive evaluation of intra and inter prediction modes is performed, and the prediction errors of intra prediction and motion compensation prediction are calculated respectively, and the prediction mode with smaller error is selected for subsequent encoding processing; For low saliency regions, a simplified encoding strategy is used, which directly copies the pixel values of the reference frame at the same position using skip mode, and only when the copying error exceeds the set threshold, motion estimation is enabled to find a more matching reference position; According to the importance of the region, hierarchical motion estimation processing is performed, high saliency regions are searched within ±32 pixels with a search accuracy of 1 / 4 pixel, medium saliency regions are searched within ±16 pixels with a search accuracy of 1 / 2 pixel, and low saliency regions are searched within ±8 pixels with a search accuracy limited to integer pixels; After the prediction mode selection and motion estimation are completed, the difference between the actual pixel value and the predicted pixel value is subjected to 8x8 discrete cosine transform, and the residual information in the spatial domain is converted into transform coefficient representation in the frequency domain; The transform coefficients are quantized using the corresponding quantization parameters of each region, and the quantized transform coefficients are rearranged in Z-shaped scanning order, and then entropy encoded and compressed using run-length encoding and Huffman encoding to further reduce data redundancy, and finally output the compressed video data stream.

[0028] S5: Organize the compressed video data stream according to the H.265 / HEVC standard format, record the encoding configuration information through the sequence parameter set and the image parameter set, and embed the saliency heat map after compression into the supplemental enhancement information and transmit it with the bit stream: Organize the compressed video data stream according to the H.265 / HEVC standard format, including sequence parameter set, image parameter set and network abstraction layer unit; Record the video resolution, frame rate and color format in the sequence parameter set; Quantization parameters and entropy coding types are recorded in the image parameter set; The compressed video data stream is encapsulated into network abstraction layer units, and a start code 0x000001 is added for separation; The saliency heat map is down-sampled to a resolution of 240x135, compressed using JPEG, and embedded in the supplemental enhancement information for transmission; The compressed video data stream is byte-aligned, and an H.265 format video bitstream file containing saliency perception coding information is output.

[0029] It should be noted that the above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments. Moreover, the terms "include", "contain" or any other variant thereof in this paper are intended to cover non-exclusive inclusion, so that the process, device, article or method including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, device, article or method. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, device, article or method including the element.

[0030] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk) as described above, and includes a number of instructions for making a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0031] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An adaptive coding method for ultra-high-definition video streams based on deep learning visual saliency, characterized in that, Includes the following steps: S1: By performing Gaussian filtering on ultra-high-definition video frames at five scales and constructing image pyramids, feature extraction is performed by combining Sobel gradient, Laplacian edge and local binary mode to generate multi-scale feature maps. S2: A pre-trained saliency detection network is used. Through the cascaded processing of feature adaptation layer, residual encoder, self-attention mechanism and transposed convolutional decoder, the multi-scale feature map is converted into a saliency heatmap, and Gaussian filtering is used to smooth the saliency heatmap. S3: Based on the smoothed saliency heatmap, the video frames are adaptively divided into high, medium and low saliency regions. A regional coding parameter table is established through connected component analysis and average saliency value calculation. S4: Based on the regional coding parameter table, perform differentiated prediction modes, motion estimation and quantization strategies for different saliency regions, and combine with the standard video coding process to achieve saliency-aware rate-distortion optimization, and output compressed video data stream; S5: Organizes the compressed video data stream according to the H.265 / HEVC standard format, records the encoding configuration information through sequence parameter set and image parameter set, and embeds the saliency heatmap into the supplementary enhancement information after compression and transmits it along with the bit stream.

2. The adaptive coding method for ultra-high-definition video streams based on deep learning visual saliency as described in claim 1, characterized in that, Step S1 includes: Retrieve the current video frame in YUV420 color space from an ultra-high-definition video stream. ; For video frames The Y component is processed by Gaussian filtering at five different scales; The filtering results at each scale were downsampled with downsampling factors of 1, 2, 4, 8, and 16, resulting in five image pyramid layers at different resolutions: , , , , ; The gradient features of each layer are calculated using the Sobel operator to compute the horizontal gradient of each resolution image in the image pyramid layers. and vertical gradient This allows for the calculation of the corresponding gradient magnitude; The second derivative features of each layer are calculated, and the second derivative features are used to perform edge detection on each resolution image in the image pyramid layer using the Laplacian operator; Local binary pattern features are extracted, and the local binary pattern features are used to calculate the LBP histogram for each 8×8 pixel block of each resolution image in the image pyramid layer; For video frames The U and V chromaticity components are subjected to the same multi-scale Gaussian filtering as the Y component, and downsampled to the same resolution as the corresponding layer of the image pyramid, resulting in five U component image pyramid layers and five V component image pyramid layers with different resolutions. Construct a multi-scale feature descriptor, wherein the multi-scale feature descriptor is the gradient feature, second derivative feature, local binary pattern feature, U component and V component extracted for each spatial location at each resolution; The multi-scale feature descriptors are Z-score standardized to calculate the mean and standard deviation of each feature dimension, and the feature values ​​are converted into a zero-mean unit variance distribution. The multi-scale feature descriptors, after being standardized at different resolutions, are uniformly upsampled to the same resolution through bilinear interpolation, then concatenated according to the feature dimensions and smoothed by 3×3 mean filtering to obtain a multi-scale feature map.

3. The adaptive coding method for ultra-high-definition video streams based on deep learning visual saliency as described in claim 2, characterized in that, Step S2 includes: Multi-scale feature maps are input into a pre-trained saliency detection neural network; the saliency detection neural network includes a feature adaptation layer, a residual encoder, a self-attention layer, and a transposed convolutional decoder; The feature adaptation layer uses three 3×3 convolutional layers to map the multi-scale feature map to 256 dimensions; The transposed convolutional encoder uses a four-residual-block structure to process the feature adaptation layer output. Each residual block contains two 3×3 convolutional layers and one skip connection. The self-attention machine layer passes the encoder output through three 1×1 convolutional layers to generate a query matrix Q, a key matrix K, and a value matrix V, and calculates the self-attention weight matrix. in, Indicates the dimension of the key vector; Indicates matrix transpose; It is a normalized exponential function; The transposed convolutional decoder uses four transposed convolutional layers to progressively upsample the self-attention weight matrix. By using the sigmoid activation function to restrict the output to the range of 0 to 1, a pixel-level saliency heatmap is obtained. Gaussian filtering is applied to the saliency heatmap to obtain a smoothed saliency heatmap.

4. The adaptive coding method for ultra-high-definition video streams based on deep learning visual saliency as described in claim 3, characterized in that, Step S3 includes: Based on the smoothed saliency heatmap, video frames are divided into regions with different saliency levels, and a high saliency threshold is set. The significance threshold is 0.

7. The significance value is 0.3, and the region division rules are as follows: when the pixel significance value is ≥0.7, it is classified as a high significance region; when 0.3≤pixel significance value<0.7, it is classified as a medium significance region; when the pixel significance value<0.3, it is classified as a low significance region. Connectivity analysis is performed on regions with different saliency levels. 8-connectivity is used to identify continuous regions, while scattered regions with an area of ​​less than 100 pixels are filtered out, and a regional coding parameter table is constructed.

5. The adaptive coding method for ultra-high-definition video streams based on deep learning visual saliency as described in claim 4, characterized in that, The construction of the regionalized coding parameter table includes: Calculate the quantization parameters for regions at different significance levels. The formula for calculating the region quantization parameters is as follows: in, Quantization parameters representing the significance level region, Indicates the range of quantization parameter adjustment. The average significance value of the region representing the significance level; Indicates the basic quantization parameters; A regional coding parameter table is constructed, which records the boundary coordinates, quantization parameters, and average significance value of each region.

6. The adaptive coding method for ultra-high-definition video streams based on deep learning visual saliency as described in claim 4, characterized in that, Step S4 includes: Based on the quantization parameters of the saliency level regions in the regionalization coding parameter table. Differential coding is performed on each region, fine quantization is performed on highly significant regions, standard quantization is performed on moderately significant regions, and coarse quantization is performed on low significant regions. A prediction mode selection strategy is implemented based on the regional saliency level. Intra-frame prediction modes, including vertical prediction, horizontal prediction, DC prediction, and planar prediction modes, are selected for highly saliency regions. The prediction direction with the smallest error is selected by calculating the prediction residual of each mode to obtain the best coding effect. A comprehensive evaluation of intra-frame and inter-frame prediction modes is performed for the medium saliency region. The prediction errors of intra-frame prediction and motion compensation prediction are calculated separately, and the prediction mode with smaller error is selected for subsequent coding processing. For low saliency regions, a simplified coding strategy is adopted, using a skip mode to directly copy the reference frame pixel values ​​at the same location, and only when the copying error exceeds a set threshold is motion estimation enabled to find a more matching reference location. Hierarchical motion estimation is performed based on the importance of the region. High saliency regions are fully searched within ±32 pixels with a search accuracy of 1 / 4 pixel. Medium saliency regions are quickly searched within ±16 pixels with a search accuracy of 1 / 2 pixel. Low saliency regions are coarsely searched within ±8 pixels with a search accuracy limited to whole pixels. After the prediction mode selection and motion estimation are completed, the difference between the actual pixel value and the predicted pixel value is subjected to an 8×8 discrete cosine transform to convert the spatial residual information into the frequency domain transform coefficient representation. The transform coefficients are quantized using the quantization parameters corresponding to each region. The quantized transform coefficients are then rearranged in a zigzag scanning order, and entropy coding compression is performed using run-length encoding and Huffman coding to further reduce data redundancy. Finally, a compressed video data stream is output.

7. The adaptive coding method for ultra-high-definition video streams based on deep learning visual saliency as described in claim 6, characterized in that, Step S5 includes: The compressed video data stream is organized according to the H.265 / HEVC standard format, including sequence parameter sets, image parameter sets, and network abstraction layer units; Record video resolution, frame rate, and color format in the sequence parameter set; Record quantization parameters and entropy coding type in the image parameter set; The compressed video data stream is encapsulated into network abstraction layer units, and a start code 0x000001 is added for separation. The saliency heatmap was downsampled to 240×135 resolution, compressed using JPEG, and embedded in supplementary enhancement information for transmission. The compressed video data stream is byte-aligned and output as an H.265 format video bitstream file containing saliency-aware coding information.

Images